Sparse, Efficient and Explainable Data Attribution with DualXDA
Authors: Galip Ümit Yolcu
Moritz Weckbecker
Thomas Wiegand
Wojciech Samek
Sebastian Lapuschkin
Venue: Accepted by TMLR ()

Abstract: Data Attribution (DA) is an emerging approach in the field of eXplainable Artificial Intelligence (XAI), aiming to identify influential training datapoints which determine model outputs. It seeks to provide transparency about the model and individual predictions, e.g. for model debugging, identifying data-related causes of suboptimal performance. However, existing DA approaches suffer from prohibitively high computational costs and memory demands when applied to even medium-scale datasets and models, forcing practitioners to resort to approximations that may fail to capture the true inference process of the underlying model. Additionally, current attribution methods exhibit low sparsity, resulting in non-negligible attribution scores across a high number of training examples, hindering the discovery of decisive patterns in the data. In this work, we introduce DualXDA, a framework for sparse, efficient and explainable DA, comprised of two interlinked approaches, Dual Data Attribution (DualDA) and eXplainable Data Attribution (XDA): With DualDA, we propose a novel approach for efficient and effective DA, leveraging Support Vector Machine theory to provide fast and naturally sparse data attributions for AI predictions. In extensive quantitative analyses, we demonstrate that DualDA achieves high attribution quality, excels at solving a series of evaluated downstream tasks, while at the same time improving explanation time by a factor of up to 4,100,000 x compared to the original Influence Functions method, and up to 11,000 x compared to the method's most efficient approximation from literature to date. We further introduce XDA, a method for enhancing Data Attribution with capabilities from feature attribution methods to explain why training samples are relevant for the prediction of a test sample in terms of impactful features, which we showcase and verify qualitatively in detail. Taken together, our contributions in DualXDA ultimately point towards a future of eXplainable AI applied at unprecedented scale, enabling transparent, efficient and novel analysis of even the largest neural architectures -- such as Large Language Models -- and fostering a new generation of interpretable and accountable AI systems. The implementation of our methods, as well as the full experimental protocol, is available on github.

Published in Transactions on Machine Learning Research (12/2025)
Sparse, Efficient and Explainable Data Attribution with DualXDA
Galip Umit Yolcu*
Fraunhofer Heinrich Hertz Institute
Moritz Weekbecker'
Fraunhofer Heinrich Hertz Institute
Thomas Wiegand
Fraunhofer Heinrich Hertz Institute
Technische Universität Berlin
BIFOLD Berlin Institute, for the Foundations of Learning and Data
Wojeiech Samek!
Fraunhofer Heinrich Hertz Institute
Technische Universität Berlin
BIFOLD Berlin Institute for the Foundations of Learning and Data
Sebastian Lapuschkin
sebastion. wawpiiioow.ppuschhinicn Shiisshhhhkkhihhifeee frandhofrruunoerr.r.. de
Frawnhofer Heinrich Hertz Institute Technological University Dublin
Reviewed on OpenReview: https: //
// openreview. net/ forum? id= off.1N888...
Abstract
Data Attribution (DA) is an emerging approach in the field of eXplainable Artificial Intelli- gence (XAI), aiming to identify influential training datapoints which determine model outputs. It seeks to provide transparency about the model and individual predictions, e.g. for model debugging, identifying data-related causes of suboptimal performance. However, existing DA approaches suffer from prohibitively high computational costs and memory demands when applied to even medium-scale datasets and models, forcing practitioners to resort to approximations that may fail to capture the true inference process of the underlying model. Additionally, current attribution methods exhibit low sparsity, resulting in non-negligible attribution scores across high number of training examples, hindering the discovery of decisive patterns in the data. In this work, introduce DualXDA, framework for sparse, efficient and explainable DA, comprised of two interlinked approaches, Dual Data Attribution (DualDA) and eXplainable Data Attribution (XDA): With DualDA, we propose a novel approach for efficient and effective DA, leveraging Support Vector Machine theory to provide fast and naturally sparse data attributions for Al predictions. In extensive quantitative analyses, demonstrate that DualDA achieves high attribution quality, excels solving series of evaluated downstream tasks, while at the same time improving explanation time by factor of up to 4,100,000x compared to the original Influence Functions method, and up to 11,000x compared to the method's most elficient approximation from literature to date. We further introduce XDA, a method for enhancing Data Attribution with capabilities from feature attribution methods to explain why training samples are relevant for the prediction of a test sample in terms of impactful features, which we showcase and verify qualitatively detail. Taken ter,ther, our contributions in DualXDA ultimately point towards a future of eXplainable Al applied at unprecedented scale, enabling transparent, efficient and novel analysis of even the largest neural architectures such as Large Language Models and
"Equal contribution by authors. i Corresppnndig authors.

---

Published in Transactions on Machine Learning Research (12/2025)
fostering a new generation of interpretable and accountable AI systems. The implementation of our methods, as well as the full experimental protocol, is available on gi githut
1 Introduction
Despite the remarkable achievements of deep learning approaches, these methods remain black-boxes due to the opacity of their information processing procedures. Explainable AI (XAI) (Samek et al.) 2019) has emerged to address the need to elucidate the inference processes of these models. Specifically, global XAT aims to provide insights into overall model characteristics, while local XAI explains outputs for individual test samples. Early approaches have been centered on feature attribution (Bach et al.] 2015 Ribeiro et al. 2016), assigning an attribution score to each input feature, which indicates which part of the input is more or most influential in determining the model output. Later approaches provide different explanation paradigms, such as concept-based (Kim et al. 2018 Achtibat et al. 2023) and counterfactual explanations (Guidotti 2024).
Feature- approaches put a strong emphasis on model inputs, parameters and representations but ignore the training data that the model has observed during optimization. A relatively recent lens of interpretability, Data Attribution (DA) attributes the model outputs to samples of the training dataset, producing attribution scores for each (training) datapoint. This a complementary approach to feature attribution, allowing to rank the training data with respect to their effect in determining the model parameters (global DA) or specific outputs (local DA).
Data quality is integral in order to obtain robust model performance: Biased sampling @Tapuschkin et al. 2019h, poor data quality management (Lin et al.) (022) or adversarial attacks (Shafahi et al.) 2018 Zhu et al. 2019), can drive the model to suboptimal solutions. Therefore, in addition to understanding and debugging the decision processes of the model, DA methods have found applications identifying and solving data-related problems, such as detecting mislabeled samples (Koh & Liang] 2017 Yeh et al. 2018 Pruthi
et al. 2020), designing (Fang et al.] 2020) and counteracting backdoor attacks (Hammoodeeh & Lowd 2022).
Furthermore these attribution methods have demonstrated effectiveness in dataset distillation (Liu et al.
2021 Naujoks et al. 2024), i.c. finding small subsets of training data which can be used to optimize on without having to sacrifice model performance. This is an important step towards the creation of efficient training loops for deep learning models, which usually require large amounts of data and energy for large
scale training in order to be effective predictors (Thompoon et al. 2020
In practice, the utility of most DA methods is limited by two key factors: First, statc-of-the-an methods depend on the estimation of inverse Hessian vector products, which is computationally expensive. While recent work has improved the efficiency of several approaches to DA with the intent to make them feasible for applications on larger language models such as BART or BERT (Guo et al. 2021 Ladhak et al. 2022). these methods remain challenging to scale and are often impractical under strict computational constraints, limiting their accessibility in real-world applications. Some methods can be adjusted to reduce computational load by resorting to lower quality estimations of attributions. Still, the application of these methods in time-critical scenarios, where attributions are required in the order of seconds or less following inference, is exceedingly difficult due to the tightly imposed temporal constraints contrasting their computational complexity.
Secondly, many existing DA methods produce dense attribution scores, i.e., they assign non-negligible importance to a large portion of the training dataset. In contrast, sparse attributions, where only small subset of candidate components receive substantial credit, can lead to explanations of lower complexity, which have been shown to enhance human interpretability (Colin et al., 2022). The connection between sparsity and interpretabilit has been well-established in feature altribution literature, where sparse explanations mitigate cognitive load and improve the users' ability to understand model behavior (Chalasani et al. 2020 Warnecke et al. 2020 Bhatt et al. 2021). This desideratum receives empirical support from Ilyas et al. (Iyas et al.) 2022), who demonstrate that ground-truth data attributions subset of training samples exerting influence on the model output predicted on individual test points. To enforce the effect of attribution sparsity, an additional (artificial) sparsification step is sometimes applied to non-sparse methods (Park et al. 2023). This process discards a large portion of the attributions, which in turn,
inherently sparse, with only small
httpss//ghhhhab.ccmmgmmmyyy///// DualXDA
2

---

Published in Transactions on Machine Learning Research (12/2025)
may potentially result in attributions that do not reflect the actual prediction patterns of the model being explained. Regarding explanations of black-box models, the property of closely representing the inference of a model it aims to explain is referred to as "faithfulness" in XAI literature (Hedström et al. 2023). Therefore, sparse attributions that are also faithful to the model are highly preferred in DA.
To address both problems, we turn to established machine learning algorithms, where the influence of individual training data points has already been studied in detail. Concretely, we build upon the concept of Support Vector Machines (SVMs) (Cortes & Vapnik, 1995). SVMs perform linear classification by optimizing weight vectors in terms of a linear combination of small subset of the training data. This allows the derivation of an alternative formulation of the prediction function as a sum of independent contributions relating to each training sample, by solving the associated dual problem (Crammer & Singer 2001). Kernel SVMs, in turn, learn linear classifiers on nonlinear representations of training samples, which are then used to perform inference on test data. Therefore, given a neural network predictor and assuming the penultimate layer thereof as a mapping into a reproducing kernel Hilbert space, we build a SVM-based surrogate model replacing the final layer of the network, yielding an interpretable and faithful surrogate for the whole model.
Finally, we address another limitation of DA methods: the attribution values themselves opaque, providing no insight into why certain training datapoints are deemed important for given prediction. To this end, we combine data attribution with feature attribution to identify relevant features that explain the attribution values in both training and test input domains simultaneously. In our paper, we make the following contributions:
We introduce DualDA: a novel DA method which employs multiclass kernel SVMs (Crammer & Singer 2001 as surrogates for deep learning models. Based on their dual problem, we derive sparse and high-quality explanations, both locally and globally, in a time- and memory-efiiciet manner.
We evaluate DualDA in terms of quality and sparsity of explanations, and runtime as well as memory efficioncy. In total, we compare nine different methods on seven quantitative metrics across three different datasets and models, the most extensive quantilative evaluation of prominent DA methods so far. We find that DualDA performs at state-of-the-an thrtr-rrt while providing an up to 4,100,000x speed- up compared to the least efficient competing method and up to 11,000 compared with the SoTA implementations of Inftuence Functions. These efficiency gains meaningfully expand the practical usability of data attribution, including scenarios where existing methods are too computationally demanding.
We prove that DualDA attribution scores agree within the multiclass kernelized SVM setting, almost all datapoints, with attributions produced by both the gradient-based Influence Functions framework (Koh & Liang] 2017) (with minor modification) and the kernel surrogate-basee based Representer Points framework (Yeh et al. 2018
We also introduce XDA: method that leverages the popular Layer-wise Relevance Propagation (LRP) (Bach et al., 2015 of-the-art feature attribution (FA) method and eXplains attribution values assigned to samples by mapping them back through the model to the (input) feature space. This yields insights as to why a certain training datapoint is important by identifying relevant features such as pixels or tokens both in the training and test input domains simultaneously. To our knowledge, XDA is the first method to enable seamless integration of feature attribution and data altribution, offering joint explanations that connect model behaviour across feature and data provenance dimensions.
Montavon et al.; 2019 Samek et al., 2021 Achtibat et al.) 2024) a state-
We demonstrate how our DualXDA framework can be used to explain model decisions with both data attribution values and feature attribution explanations in case studies across three different datasets, substantiating the abilities of DualDA and XDA to explain counterfactuals via an ablation study.
In summary, we propose DualDA as an efficient, sparse and effective alternative to existing Data Attribution methods, Building on this foundation, we present XDA as a bridge to feature attribution, enabling - for the first time- - joint, interpretable explanations that connect influential training samples to meaningful input.
3

---

Published in Transactions on Machine Learning Research (12/2025)
features. Together, these components form the DualXDA framework, providing novel opportunities for deeper, more actionable analyses into the inner workings of modern neural networks.
2 Related Work
Let assume supervised learning regime with given training data D - vi) i E [N]), where 24 E Rd are samples from the imput domain and y; E [K] are the corresponding class labels. A neural network ((; Rd + R with trained parameter takes a sample from the input domain and emits logits for all classes, which can be turned into class probabilities or a class prediction. In the following, we consider networks with final fully-connected layer: In this case, the network parameters consist of weight matrix W xx, and remaining parameters i that the network function can be split into feature extractor f(:0): IR + Rdd consisting of the neural network from the imput up to the final layer and a final affine layer g(: W)= RRd, + R
(+:6)) = f); 1(:0,W) W)
To incorporate a bias term into this formulation, the extracted feature vectors can be extended by appending a constant value of 1.
2.1 What is Data Attribution?
The goal of Data Attribution (DA) is to identify training data that have been influential in determining the prediction model, or the model itself.
Definition 2.1. Global DA methods assign each pair (z,, e) of training sample 2; E D and class c E [K] a real-valued attribution which indicates the relevance or influence of the training datapoint on the fit of the model ((:; ()- Local DA methods produce function Te : IR [N] + E for each class c. The inputs this function are test sample and the index for training point. The value (z( indicates the relevance influence of the training datapoint
the model output (xx; 0)c-
DA has historically been framed the problem of approximating the ground-trute effects of model retraining under altered training data compositions (Koh & Liang, 2017 Park et al.) 2023). Recent work (Wei et al. 2024) advocates for a different standpoint, aiming to estimate the changes induced by fine-tuning an already trained model with an altered training set instead. Other authors do not claim for their attribution values to approximate a ground truth, instead they are interested in the utility of their attribution values fulfilling certain downstream tasks. In all of these formulations, the resulting attributions are interpreted such that positive attribution means the training sample contains evidence for the model decision while negative attribution is indicative of evidence against the decision being explained.
While there are many different settings that DA has been studied in, we consider post-hoc attribution setup, where we are given the training dataset, knowledge about training hyperparameters, and model parameters collected at different epochs a single training run.
2.2 Data Attribution Methods
Existing DA methods can be broadly categorized into four different groups, in terms of how they produce attribution scores, as listed below. Detailed explanations and formulas for all methods can be found in Appendix D.1
Sisela approaches compute the similarity between the training and test samples, building on the intuition that the model inferences should be informed of training samples that produce similar latent representations (Pozeshkpour et al. 2021). or through other measures of similarity (Singla et al.] 2023 However, these attributions are class- unspecific as the measured similarity is independent of the predicted class.
Gradient-based approaches estimate the effect of retraining on a new training set where sample has been removed from the training corpus. In this framework, positive attribution indicates a decrease in loss, which indicates more confidence for the associated class. To this end, these methods employ Taylor
4

---

Published in Transactions on Machine Learning Research (12/2025)
Original Model
Fit Sparse Kernel Surrogate
Test Sample
Extractor
Features
Linear Classifier
o (stme))
((aa)
Ftest
Ag
[a)
f(r)
DualDA: Global Data Attribution
Ki
Training Dataset
d(f(sw)))
[(a))
Σ (x((s)) ((Fmm)
Σ {T()))
5: train
DualDA: Local Data Attribution
Local Attribution
Global Attribution
Aaffss) Rirmal
Opponents Proponents
Aullsa Movel
((
0
276
4
2
Hone Howe
Ti(Peen))
Auf (ax,)
T
X
XDA: Concurrent Attribution Maps
Test Attiibution Map
A
(xz)) (aras)
O
Relevances
Total Relevance in heatmap
Train Attrlbution Mapp
Figure 1: DualDA efficiently identifies training samples which are influential for both the overall model fit. (global attribution) as well as for the prediction for specific test samples (local attribution). O Our method assumes models with a nonlinear feature extractor f followed by a fully-connected layer as the classification head g. DualDA substitutes the final layer of the original model with lincar SVM. The resulting weight vector w then be expressed linear combination of the final layer latent. embeddings of training samples. Note that binary classification case is visualized for the sake of simplicity and legibility, whereas DualDA employs multiclass SVM. ① The global attribution of each training datapoint is quantified by its corresponding scalar coefficient X; in the linear decomposition of w. @ Moreover, since w represented as a combination of training feature embeddings, we can decompose the output of the surrogate model for a given test point into a sum of contributions from each training sample. This local attribution (i.c. the contribution of a training point the prediction for specific test point) is given by the inner product of the feature embeddings of the training and samples, scaled by the global influence cocfficient of the training sample. ① To trace these inffuences back to the input space, approach employs Layer-wise Relevance Propagation (LRP) DualDA attributions. The method propagates the attributions from the surrogate model's output, through the feature extractor, down to the input pixels for both training and test samples. The result is a pair attribution heatmaps one for each training test. pair highlighting input regions that contributed positively or nogatively to the model's inference.
XDA
5

---

Published in Transactions on Machine Learning Research (12/2025)
approximations, which involve gradients of the training and test sample. The earliest method, Influence Functions (Koh & Liang 2017), calculates a first-order approximation using the parameters of the final trained model. As this involves calculations of the inverse Hessian, which are infeasible to compute in limited time for even modestly-sized models, approximations using the LiSSA algorithm (Agarwal et al. 2017], the Arnoldi method (Schioppa el al., 2022) or the (Figenvaluu--corrcclee)) Kronecker-Factored Approximate Curvature method (EK-FAC) (Grosse et al. 2023) are commonly used. Furthermore, the method TRAK (Park et al. 2023) applied to a single model can be seen as a version of Influence Functions with a general Gauss-Newton approximation of the Hessian. To include the behaviour of the model over the entire training process, the method TracIn (Pruthi et al.] 2020) collects the dot product between the gradients at different epochs and accumulates them in a weighted manner in relation to the corresponding step sizes. GradDot and GradCos (Charpial et al. 2019) are simplifications of TracIn which only consider the (normalised) dot product at the final epoch.
Surrogate-based approaches propose exchanging parts of the network for similarly behaving surrogates which are inherently interpretable in terms of the importance of training data points for predictions. The Representer Points (Yeh et al. 2018) method retrains only the final fully-connected layer of the model using a modified loss with an added E2-penalty. This enables the application of the Representer Theorem to decompose the networks output into the sum of kernel evaluations involving the test point and the individual training points.
Retraining-basee based approaches sample counterfactual models by retraining the model on various altered training sets and then use these samples to estimate the model output for an arbitrary training subset. Consequently, these methods are generally regarded baselines and are very demanding in terms of computational resources (Park et al. 2023
3 Dual eXplainable Data Attribution
In this section, we introduce our two novel methods: DualDA, which efficiently produces sparse and faithful data attribution, and XDA, which explains the attribution values from DualDA by highlighting relevant features in the input space responsible for the attribution between pairs of training samples and the test input. Combined, DualDA and XDA constitute the DualXDA framework.
DualDA is a surrogate-basod DA method, which replaces the final linear layer of the model with linear multiclass SVM. We selected SVMs surrogates for four reasons: (1) they closely approxinate the behavior of the original model, (2) are inherently interpretable in DA sense in their dual form, (3) yield sparse attributions, (4) and enable fast training and inference, as further outlined below.
(1) Prior research shows that gradient descent various models has an implicit bias towards margin maximizing
behaviour (Gunasekar et al. 20088 Soudry et al. 2018: Gidel et al. 2019: Ji & Telgarsky 2019 Tarzanagh let al.) 2023) which the defining feature of SVMs. In particular, recent studies demonstrate convergence of neural networks to kernelized SVMs under certain conditions in binary and multiclass classification
scenarios (Lyu & Li 2019 Ravi et al. 2024 This provides the theoretical motivation to assume that an
SVM can serve as a faithful surrogate for the original final layer of neural network predictor. We validate
this assumption empirically in Section 4.5
(2) The learned weights an SVM in feature space can be exactly expressed as a weighted sum of representa- tions of the training datapoints by solving the corresponding dual problem. This provides a natural approach to global DA, and in extension to attributing the neural network outputs for individual test points.
(3) When working with large datasets containing millions of samples, attribution methods that yield similar or indistinguishable attributions across large number of data points become difficult to interpret effec- tively (Twata & Yoshikawa, 2021). SVMs exhibit an implicil bias towards sparsity in their attribution values as, under constrained conditions, only a limited number of support vectors contribute in determining the model's docision boundary. While this sparsity can enhance interpretability by focusing on the most inffuential samples, excessive sparsity may hinder the expressiveness of the resulting explanations. To mitigate this, the degree of sparsity can be controlled through the choice of the regularization hyperparameter of the underlying
6

---

Published in Transactions on Machine Learning Research (12/2025)
SVM, allowing balance between conciseness and representational richness in the attributions, as shown in detail in Appendix F.3
(4) The training of a linear SVM has worst-case time complexity between O(dN) and O(dN2) Huus et al, 2006), where is the size of the attribution (e.g. training) dataset and is the size of the input space of the SVM, i.e. the size of the penultimate layer. This allows efficient applicability on large datasets and models. Additionally, computing attributions for single pair of train and test samples requires only an inner product. The computational complexity per local attribution score is therefore only O(d). This enables near-instantnneous attribution calculation for new test samples after caching hidden layer activations of the attribution dataset and SVM weights.
To explain the attribution values of DualDA, XDA maps calculate attribution values back through the model to imput space, identifying relevant imput features (e.g., pixels in images or tokens in text) for the model inferences. XDA simultancously creates relevance mappings for both training and test samples, producing corresponding relevance healmaps specific to each pair of samples that discover the relationship between them. This approach renders our DualXDA framework inherently explainable by elucidating the interaction between training and test samples. Moreover, it enhances feature attribution analysis, i.e. the process of tracing model behaviour to input features, by decomposing feature explanations into the constituent effects of individual training samples.
A further extension of our approach including explanations using human understandable concepts to fully unify explanations on the data-level, feature-level and concept-level ( (inspired by (Achtibat et al. 2023 ) is
given in Appendix H
3.1 DualDA: Using Support Vector Machines for Sparse and Efficient Data Attribution
This subsection presents a step-by-step derivation of DualDA attributions, with a graphical overview provided in Figure [ DualDA uses the soft-margin multiclass SVM formulation introduced by Crammer and Singer (Crammer & Singer 2001) as surrogate model. The contribution of each training sample to the output of this surrogate model can be determined in straightforward way, which examine in detail below.
We start by fitting a weight matrix W, consisting of one weight vector w. per row for every class predicted by the original model, where each weight vector may optionally incorporate a scalar bias parameter (see Steps I to 2 in Figure []. We learn these parameters to classify data points in the kernel space induced by the feature extractor (=;() (cf. Step @ in Figure 1]. This achieved by solving the (primal) optimization problem
minw s.t.
1| + CEM
ViE [N], VC [K]
W.T [ 0) w. fz.;; 0) + dey, 2 1 C
Here, the hyperparameter C' >>0 0 regulates the sparsity of the solution: A high value for C' will produce solution where W depends only on few training points, whereas a low value for C creates solution to which many datapoints contribute. The surrogate model output is then given by W Wf(Eresti 0) Rs (2tess 0). This programme aims to ensure that for a vector /; = f(x;; f) the class score wy corresponding to the true class ys is at least greater by 1 than the score for all other classes c. If this difference is smaller than one, it will contribute to the loss. Therefore, this problem is oquivalent to lz-rogularized empirical risk minimization with a multiclass Hinge lossP] Due to the formulation of SVMs through margin maximization, the same problem can be seen hyperparameter C' determines how hard the margins are, i.e. how strongly each datapoint contributes to the final loss: A higher value increases the penalty for classification errors, encouraging the model to reduce training mistakes at the risk of overfitting.
margin maximization problem for multiclass classifier. In this formulation, the
As per standard SVM theory, we can formulate the dual problem. The full mathematical formulation of the dual problem is excessive in length and not casily interpretable, we therefore refer the interested reader
2This formulation differs from training multiple SVMs it common for multickss classification problems, since here, information from different. classification heads shared through the constraints.
7

---

Published in Transactions on Machine Learning Research (12/2025)
Appendix A where the detailed derivation is given. However, through solving the dual problem, we obtain dual variables Oie for each training vector Si and each class c. These relate to the optimal weight vectors we through the formula
N
Wc =
diafi:
(2)
where the coefficients Xie defined as
(C aty.)
yo =
Ale
(3)
Osic
else
express how much each training feature vector Si contributes to the surrogate fit and can therefore be interpreted as global attribution scores (see Step ② in Figure I. Through this formula, we can further identify the individual contributions of each fú to the predicted class score of test vector festt for some class
T We frest
Σ Aiefi frest (C-aaiy fuest
Σ f. frest
(4)
=0
Given this motivation, DualDA defines the attribution value of a training sample 24, given the prediction of a lest sample Ftest and class e, as
TeD (Irest, 1) = dieSi feest
(5)
where DD constitutes the abbreviation of DualDA. This local attribution is depicted in Step ! in Figure I We refer the reader to Appendix [ for the full derivation. By this definition, DualDA fulfills an approximate conservation property, meaning that the attributions over all training samples sum up to the output of the model, i.e.
((Ieat)e FC we frest = ΣTe (Itest, 1).
DD
(6)
Relating DualDA to other approaches While we introduce DualDA with a focus on SVM theory, it can be interpreted within the framework of the Representer Points method as an lg-regularized linear surrogate fitted on top of the extracted features in the penultimate layer. The methods differ in the loss used to train the surrogate. Whereas Representer Points keeps the original loss and simply adds an additional lr-regularization, DualDA attributions can be derived by using multiclass Hinge loss in the formula for Representer Points in Equation [D6 This derivation of DualDA is detailed in Appendix
Furthermore, we can align our surrogate-based approach with the framework of gradient- based approaches. Applying a generalized version of Influence Functions outlined in Corollary 1 of (Naujoks al.) 2024), we can investigate how the optimally derived W in Equation (1 changes, under an infinitesimal downweighting of the contribution of datapoint/ fs and how this in turn affects the class score wj. fiest. This influence value aligns with the attribution calculated by DualDA, for datapoints that do not. lie exactly on the margin. We report this result in Theorem [.1 and refer the reader to Appendix
the proof.
Theorem 3.1. Let W denote the solution to the SVM optimization problem in Equation 1 Denote by W the parameters of an SVM trained on the same dataset and a optimization criterion modified w.r.t. to the i-th training sample:
minw s.l.
211 &
Vj [N], VeE [K]
(²,;());))))) wo [(r;; 0) + Sey; 2 Ejj
The contribution of the i-th training sample is down-weighted by a factor ef s. Then, for any test sample x the infinitesimal change in W f fax 8) is equal the DualDA attributions, if the i-th training sample does not lie eraclly on the margin of the SVM, i.e. arg max (1 -5eyy w J3 w. f(]] is unique. In this case, the following equality holds:
DD c
O(W - W x
(x,
(8)
de
8

---

Published in Transactions on Machine Learning Research (12/2025)
3.2 XDA: Mapping Data Attribution to the Feature Space
While DualDA constitutes scalable and practically applicable solution for DA, we argue that DA alone is not sufficiently informative on its own: DA informs about which training datapoints are influential for a prodiction, yet not why they are influential: A single datapoint may be relevant for various reasons, c.g., due to background patterns, parts of objects, or other semantic features. DA, however, does not provide any information to distinguish between these sources of information and their influence, resulting in potentially ambiguous explanations. To resolve this lack of clarity, we combine data attribution with feature attribution, leveraging the structure of our DualDA attributions with the introduction of XDA in order to obtain additional degrees of detail in our combined DualXDA explanations. This integrated perspective allows for deeper, more interpretable explanations of model behavior. While in principle, any FA approach could be levied to further explain the attributions of DualDA we choose to integrate the principles of Layer-wise Relevance Propaga tion (LRP) (Bach et al.) 2015) to achieve XDA because like DualDA, it leverages the model's activations on its hidden layers. IRP is a feature attribution technique based on the decomposition of a neural network's s prediction by backward propagating relevance scores layer-by-layer from the output to the input, following a relevance conservation principle that maintains and preserves the total amount of relevance throughout the decomposition process. As such, LRP is effectively striving towards the same goal as DualDA: the conservative attribution of model outputs to units of interpretation. However, the crucial difference between the two approaches lies in the orthogonal selection of the components largeted for interpretation, i.e., individual test features for LRP and training samples in their totality for DualDA. In the image domain, explanations derived from LRP are commonly represented as a heatmap over the corresponding input.
XDA combines the approaches of DualDA and LRP to further explain the data attribution scores T. (Freest i) w.r.t. the interaction of both Frest and the influential training datapoint z;. An overview of the XDA relevance computation process is presented in Step ① of Figure || Specifically, the DualDA surrogate model naturally produces the data attributions as an intermediate signal, each of which is given by a weighted dot product of features computed by the feature extractor part of the network. XDA focuses on this branch of the surrogate model, whoreby the output of this branch is isolated and explained via LRP with initial relevance determinod depending on class e explained by DualDA as
R(rrett 1) = =die Feesa.f.
The algorithm then backpropagates this quantity from layer L of the DualDA surrogate model backwards through the feature extractor towards the input sample Ztest as well as the training sample x; in two separate backward passes. This procedure results a pair of feature attributions R (Ireat: i)d and Rarain (Frest: 1) which together explain how in terms of related features the influential support vector x; particular is affecting the model in the prediction of Stess As such, XDA shares conceptual similarities with BiLRP (Eberle et al. 2022) and CRP (Achtibat et al. 2023). but differs by explaining the feature kernel of surrogate SVM (rather than deep similarity model, as in BiLRP) and by splitting propagation pathways across support vectors instead of latent concepts (as in CRP). Overall, XDA leverages the conservative properties of DualDA and LRP to achieve data attribution as well as feature attribution, combining the benefits of the two orthogonal approaches. The conservative property of XDA further implies that the generated feature attribution heatmaps are also conservative whereby the sum of XDA test feature attribution maps recovers the IRP feature attribution map, i.e. LRP explanation of Frest for class c at layer I is approximately equal to ΣN Reeett (Zrests i)/-
4 Evaluating Data Attribution
To test the validity of DualDA, we compare the method against other state-of-the-art attribution methods introduced in Section 2 and defined in detail in Appendix D.1] Details about implementations are given in Section 4.3 We analyze three different vision models of varying sizes, each trained on a different dataset: a simple Convolutional Neural Network with 3 convolutional and 3 fully connected layers with RcLU non-linearities Irained on the MNIST dataset (LeCun el al., 2010), ResNet-18 (He el al. 2016 trained
Note that while we can derive such explanations in all layers 1 I., we will focus on imput-layer explanations in pixel space with in this paper.
9

---

Published in Transactions on Machine Learning Research (12/2025)
CIFAR-10 (Kiihevsky & Hinton, 2009 and ResNet-50 (He et al,, 2016) trained on the Animals with Attributes 2 (AwA2) dataset (Xian et al. 2018) (training details can be found in Appendix D.2. For each dataset and model, we calculate the attribution score of all training datapoints for the predicted class for 2,000 randomly selected test points, using the attribution methods. We evaluate the quality of the derived attributions on seven different metries outlined in Section 4.1] two of which are novel contributions. For detailed explanations of evaluation metrics, including formulas, we refer the reader to Appendix E In our study, we rely on quanda (Bareeva et al., 2024), a toolkit designed for the systematic evaluation of DA methods. The results of our experiments are presented in Section 4.2 We further present the resource consumption of each attribution method in Section 4.3 and compare the sparsity of the corresponding explanations in Section 4.4 In Section 4.5 we analyze how close the surrogate model matches the original model for our DualDA meth as well as the Representer points method. All experiments have been executed on a single NVIDIA A100 Tensor Core-GPU.
4.1 Evaluation Metrics
Two simple samity checks are proposed by Hanawa et al. (Hanawa et al. 2021]: The Identical Class Test relics on the assumption that the most positively important training points for a class decision should be of the same class as the model prediction. We therefore measure for all lest points the proportion of the highest ranking training data points according the DA methods outputs belonging to the same class as predicted for the given test sample. The Identical Subclass Test further asscsses whether attributions can distinguish between different subpopulations within the same class. Following (Hanawa et al. 2021 we therefore artificially group dissimilar classes together into a superclass. Given a predictor trained on this new classification setting, we check whether the DA methods can still attribute influence to training samples from the same true subclass, rather than just the broader grouped class set, as the test sample.
Furthermore, two metrics are derived from common downstreem tasks of DA: Mislabeling Detection (Koh & Liang 2017 Yeh al.] 2018 Pruthi al.) 2020), which aims to discover incorrectly labeled training samples, and Shortcut Detection (Koh & Liang 2017 Hammoudeh & Lowd 2022), which tries to identify misleading associations in training data that allow models to achieve accuracy through dataset-specific biases rather than generalizable relationships.
Finally, relraining-based metrics test the counterfactual validity of training data attributions by retraining models on subsets of the training data and measuring whether the attributions align with observed changes in model behaviour. The Lincar Datamodcling Score (LDS) (Park et al., 2023) tests whether attribution scores can predict changes in model outputs when retrained on random subsets of data. This is achieved by measuring the rank correlation between predicted attribution scores and actual model outputs. Coreset Selection evaluates whether attributions can identify the most valuable training samples by retraining models only on the highest-attributed data points and measuring how well performance is maintained.
Adversarial Data Pruming tests the opposite of coreset selection by removing the most. highly attributed training samples and measuring how much performance degrades. Unlike Coreset Selection, which evaluates whether attributions can identify a representative training subset, Adversarial Data Pruning assesses whether attributions correctly identify samples that contain unique information.
For all metrics except Coreset Selection, where superior performance is indicated by a small loss, higher scores indicate higher atlribution quality.
4.2 Results
Figure _ provides comprehensive summary of our experimental findings. For each method and dataset, it displays the average rank across all evaluation metrics plotted against the total runtime required to attribute 2,000 test images. This analysis demonstrates that DualDA exhibits competitive performance relative to other methods: it achieves the fourth-best. average rank on MNIST, second-best on CIFAR-10, and the best average rank on AwA2. Crucially, DualDA invariably requires the least computational time among all evaluated methods, which includes both the cache time required for generating explanations (such as caching hidden activations) as well as the explanation time for all 2,000 test samples. Compared to the most efficient version of the widely-employed Influence Functions framework, DualDA is 175 times faster on MNIST, 383
10

---

Published in Transactions on Machine Learning Research (12/2025)
DualDA
Influence Functions
DualDA C 10
DualDA C= 10
DualDA C 10
LISSA
Arnoldi
EK-FAC
Gradient Products
TracIn
GradDot
GradCos
TRAK
Points
Representer
*
Average Rank Better ranked)
{ Better ranked)
Average Rank Better ranked)
Average Rank
96
16
10
10
10"
10
10'
10a
10a
10"
10
Total Runtime for 2,000 test images (s)
Total Runtime for 2,000 test images (s) (e- More efficient)
Total Runtime for 2,000 test images (s) (4- More efficient)
(+- More efficient)
MNIST
CIFAR-10
AwA2
Figure 2: Average rank across all metrics plotted against the total runtime for calculating the attribution for 2,000 test images. DualDA demonstrates competitive performance with drastically reduced computational time requirements across datasets. The figure showcases the results for nine different methods, averaged over seven different metrics, on three diffcrent datasets and models.
times faster on CIFAR-10, and 520 times faster on AwA2. For explanation time only, this speedup increases to 4,000x on MNIST, 11,000x on CIFAR-10, and 8,000 on AwA2.
DualDA, with appropriately selected hyperparameter C for each metric, ranks among the top three performing methods for 6 metrics on MNIST, all 9 metrics on CIFAR-10, and 8 metrics on AwA2. Furthermore, it achieves optimal performance on 5 out of 9 metries on AwA2.
For more detailed discussion, present the scores of all metrics and methods for the AwA2 dataset in Figure 3 (results for the MNIST dataset and for the CIFAR-10 dataset can be found in Figure F.1 and Figure in Appendix F.1:: On the Identical Class test, DualDA achieves a perfect result, while most methods except Representer Points, Arnoldi and Tracin achieve a score below 0.2. On the Identical Subclass test, no mothod achieves a score above 0.8, but DualDA with low sparsity performs comparatively well and achieves the second-best score. On Mislabeling Detection, DualDA demonstrates excellent performance. Here, most methods perform reasonably well, except for Representer Points. We do show results for LiSSA, which can not be computed in fcasible time, and GradCos, which docs return inconclusive attributions in this setting by design (c.f. Figure 3 and Appendix E On Shortcut Detection, DualDA achieves perfect score while the conceptually related Representer Points only attains a score of 0.2 and all other methods fail almost completely. Notably, despite not being explicitly designed to estimate outputs of counterfactually trained models, DualDA also performs strongly on the retraining- based metrics. On LDS, it achieves the third-highest rank. On Coreset Selection, it is the best performing method both 10% and the combined average. At the 10% data pruning level, DualDA does not rank among the top three performing methods. Nevertheless, the performance differences between all evaluated approaches remain relatively modest, highlighting the inherent difficulty of eliminating unique information when pruning only a small fraction of the dataset. When considering the weighted average across all pruning levels, DualDA demonstrates competitive performance, securing the second-hipe highest score and substantially outperforming the majority of alternative methods.
From the experiments, we can also see how the choice the sparsity hyperparameter C affects the performance of DualDA on various metrics. This particularly pronounced on Shorteut Detection, where a small sparsity is preferable, likely because it is preferable have many training samples included in the support vectors find all spuriously correlated samples. Conversely for Coreset Selection, higher sparsity yields markedly better
11

---

Published in Transactions on Machine Learning Research (12/2025)
0.03
0.8
Metric score
0.02
Dua DA
DualDA
0.4
C' 10
0.01
DuaiDA
C 10-3
DualDA C'= 10
0.00
0.00
Identical class Lest (1)
Identical subclass test (1)
Mislabeling detection (1)
Shortcut detection (f)
LDS (T)
Iufluence Functions
LISSA
-0.01
Arnoldi
EK-FAC
Gradiect Producta
Metric 2001e
2.5
Tracin
GradDat
6
GradCos
TRAK
Regresenter
2-
Points
Coreeet selection 10% (4)
Coreset selection Ang (4)
Data Pruning 10% (1)
Data Pruning Ang (f)
Figure 3: Evaluation results on the AwA2 dataset. The rank for DualDA with the best-performing hyperpa- rameter C is denoted over the corresponding bar. Note that Mislabeling Detection requires calculating the self-inftuence of the entire train set (see Appendix Ep. For LiSSA, caleulating self-attributions for all training points would take roughly year of runtime and is therefore computationally infeasible. As GradCos is defined as the cosine of the angle between the Lest and training sample's fea vectors, the self-attribution for GradCos is equal to regardless of the sample.
results, plausibly because a representative subsample of the data is selected in the sparsification process. We extensively analyze the effect of the sparsity hyperparameter over wide numorical range in Appendix F.3 Based these results, recommend default. setting of C 10-3 for image classification tasks with convolutional neural networks, which corresponds to the primary experimental configuration considered in this work.
4.3 Resource Consumption
Computation time and memory requirements are the main concerns for scaling DA approaches to big models and datasets. Early approaches such as LiSSA (Koh & Liang 2017 require O(Np) time to attribute a new test point, where p denotes the number of model parameters. This is indeed much slower than the explanation time of DualDA, which requires only (dd time, as d is the size of the penultinate layer of the network and therefore d p (for the ResNet-50 model, have d = 2,048 and p 25,557.033 (032.. Other approaches such Arnoldi (Schioppa et al. 2022), EK-FAC (Grosse et al. 2023 and TRAK (Park et al. 2023 provide efficient approximations, which come at the cost of precomputing and and creating an information cache. In Figure 4 we report caching limes, cache sizes and local explanations times (excluding caching times) for evaluations on 2,000 datapoints per dataset. The total runtime on a single GPU per method and dataset, which is the sum of caching and explanation time, is used as the z-coordinate each plot of Figure 2 For TRAK, Representer Points and EK-FAC, we rely on official code releases of the original publications. For LiSSA, we the implementation from different publication [ae et al. 2022 it is written in the PyTorch (Paskke)
12

---

Published in Transactions on Machine Learning Research (12/2025)
1w
4h
MNIST
15m
4m
1m
15s
DuaIDA
1s
DualDA C 10
0.2s
Caching time
Explanation time for 2000 samples
MB
10 MB
100 MB
1 GB
10GB
DuaiDA 10-3
4w
DuaIDA C 10-1
1d
Inflence Functione
4h
CIFAR:31 10
LISSA
15m
Arnoldi EK-FAC
4m
1m
15s
Gredient Products
1s
GradDot
Caching
Explanation samples
20GB
GradCos
1w
Representer
Points
4h
15m
4m
1m
15s
15
0.25
Caching time
Explanation time for 2000 samples
MB
10 MB
100 MB
1 GB
10GB
Time
Cache
Figure 4: Caching times and cumulative explanation times over 2,000 test samples for all methods, mapped onto a logarithmic scale, as well as sizes precomputed cache information per method. Note that LiSSA does not require any caching and thus has no caching time or cache size. As calculations are made on GPU, the sparsity level of DualDA has only very minor impact on the runtime.
13

---

Published in Transactions on Machine Learning Research (12/2025)
et al. 2019) framework. Employing the same framework universally guarantees a just comparison between all methods. We have implemented GradCos, GradDot and TracIn ourselves, using the random projection trick explained in (Pruthi et al., 2020). For Arnoldi, we relied on the captum (Kokhlikyan et al. 2020 implementation. To implement DualDA in Python, we have modified - the multiclass SVM implementation from the Python package scikit-learn (Pedregosa el al. 2011].
Cache Size Tracin, GradDot and GradCos save the gradients of the fully trained model, and Tracin additionally saves gradients at different training epochs. To make cache requirements manageable, these methods further require storing a random projection matrix mapping model gradients to subspace of manageable dimensionality D. This method can bring about memory-rclated challenges with larger models, as it requires storing & random vector of size D for each model parameter. The Arnoldi and EK-FAC implementations of Influence Functions as well as TRAK also require a large cache. All of these methods rely on the introduction of random sampling and projections to ameliorate cache space and runtime requirements at the cost of approximation inaccuracios. Note that for TRAK, we use the GPU-level C++ implementation provided by the authors. Reported results concerning TRAK do not include any cache for random projection matrices, and the reported runtimes are obtained using setup with maximal efficiency.
In comparison, Representer Points only saves the extracted latent features of the training data and the surrogate weights. DualDA further reduces memory requirements by storing only the extracted latent features for training samples that serve as support vectors. This substantially reduces cache size at higher sparsity levels: compared to the most cache-intensive method, TracIn, DualDA reduces the required cache by a factor of 1,750 on MNIST, 636 on CIFAR-10, and 370 on AwA2.
Caching Time TracIn requires the computation of gradients for model chockpoints, whereas Arnoldi, TRAK and EK-FAC require the compulation of Hessian approximations and in some cases, projected model gradients. These computations take considerable time, compared to the surrogate based methods DualDA and Representer Points. Caching times for competing methods range from 5 times to 735 times longer than the kernel surrogate approaches.
Explanation Time To a substantial degree the slowest method at inference is LiSSA. Even though we have only included the final fully-connected layer in the Hessian and gradient computations, LiSSA takes 35 minutes for the explanation of single test point of the AwA2 dataset, requiring almost 7 weeks for explaining the entire test set of 2,000 datapoints. This is the minimal runtime that is achieved by following the authors' suggestion (Koh & Liang] 2017] that the total number of iterations dedicated to the estimation of the inverso- -essiann--veooo--ordree of Equation (D.I) be greater than the training dataset sizc. Arnoldi, TracIn and GradCos are moderately slow, requiring more than one second per test point on the Aw datasot. DualDA is the fastest method overall across all datasets, only requiring between 0.2ms and 4ms per test sample. Compared to the slowest method, LISSA, DualDA is 2.4 x 10 times faster on MNIST, 4.1 x 108 times faster on CIFAR-10, and 5.4 x 10a times faster AwA2. Compared to the most efficient Influence Functions method, DualDA provides a speedup of 4,000 - 11 x,000xx, depending on the dataset. When generating explanations using GPU, higher sparsity does not significantly decrease explanation time since the required matrix multiplication is already effectively parallelized and other parts of the code dominate the runtime. In our experiments, we observed, however, that when generating explanations on CPU, high sparsity significantly improves the computation time for explanations.
4.4 Sparsity of Attributions
We analyze the distribution of attribution scores over the training set for various DA methods, exemplified on the AwA2 dataset. Comparable patterns emerge across all datasets examined in this study. From a practical standpoint, sparsity can be quantified by determining the number of training points required to account for given percentage of total attribution for specific test sample. In Figure 5 we evaluate the sparsily characteristics of each method by analyzing how the cunulative sum of absolute attributions grows as a function of the fraction of most influential datapoints included. Training samples are incorporated in descending order of absolute attribution value. Methods exhibiting sparse attribution patterns demonstrate steep initial increases in this cumulative curve, while non-sparse methods show more gradual growth. Our 4Code will be made available upon acceptance.
14

---

Published in Transactions on Machine Learning Research (12/2025)
DuaIDA
Ratio absolate cumulative aftribution
DualDA C 10-5
DualDA C 10
DualDA
C= 10-2
0.6
Inftumee Punetions
LISSA
Aroold
EK-FAC
0.4
Gaulimm Products
Tracin
0.2
Gradioot
Grad
TRAK
Represencer
0%
0.5%
1%
1.5%
2%
2.5%
3%
3.5%
4%
4.5%
5%
Top x% of training samples (sorted by absolute attribution)
Figure Analysis of cumulative distribution of positive and negative attributions for various DA methods the AwA2 dataset. The x-axis represents the top x% of training samples when sorted by their absolute attribution scores, while the y-axis shows what fraction of the total absolute attribution is contained within those top samples. The cumulative curves are obtained by computing individual curves for each test sample and subsequently averaging across the complete test set.
findings reveal that the sparsity hyperparameter of DualDA effectively controls the sparsity of the resulting explanations. With C 10-1, the method reaches full attribution saturation when selecting only 0.2% of the training samples descendingly ranked by absolute attribution. EK-FAC exhibits faster accumulation than Arnoldi, TRAK, GradDot, and Representer Points. Although these latter methods initially surpass DualDA configured with low sparsity (C'= 10-5), their accumulation rate decreases more rapidly, enabling DualDA with low sparsity to still reach 90% cunulative absolute allribution quicker. TracIn, GradCos, and LiSSA demonstrate minimal sparsity, distributing similar absolute attribution values across numerous training samples.
4.5 Faithfulness of Surrogates
When using a surrogate model to obtain explanations, we must ensure that the surrogate in question closely represents the original model. To test this empirically, we observe three model similarity metrics for the surrogate models Representer Points and DualDA across all three datasets: For the original and the surrogate model, record the cosine similarity between their last. layer weight matrix (which which change in the model), the average cosine similarity of the output logits, and the multiclass Matthews correlation coelficient (MCC) of the predictions. All three metrics range from -1 to +1, where higher scores indicate greater similarity. The monochrome bars in Figure present the results for DualDA and Representer Points on the AwA2 dataset, results for the remaining two datasets are shown in Figure F.3 in Appendix F.2 The overall faithfulness of the surrogates the original model are high across methods, metrics and datasets. The similarity scores are overall comparable for DualDA and Representer Points. Interestingly, an increase the hyperparameter C can either lead to an increase or decrease in the scores, depending on the similarity metrics and dataset. However, most metrics seem to be highest at C = 10-3, indicating that there exists optimum which maximizes the similarity of the surrogale to the original model.
the only weights
Artificially Sparsifying Representer Points Following the assumption that sparse attributions are preferable due to their lower complexity {Colin al.) 2202),, is only natural to decide take non-sparse attribution method and subsequently sparsify its results. As Representer Points method similar in spirit DualDA, achieving promising results in attribution quality, test the feasibility of this approach by post-hoc
15

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
0.8
Metric scores (+)
DualDA
Representer
Points
0.6
10-5
C 10
0.4
C = 10-1
0.2
Unsparsified
0.0
Weight Cosine Similarity
Logit Cosine Similarity
Prediction MCC
Figure 6: We test surrogate DA faithfulness metrics for DualDA with sparsity hyperparameter C' e {10-5,10 10-3, 10-1) and Representer Points on the AwA2 dataset. The results are shown the monochrome bars. In order to assess whether post-hoc artificially sparsified Representer Points surrogates will lead to faithful explainers, we induce the same sparsity levels they naturally emerge for DualDA with the corresponding sparsity parameter C, and present the corresponding surrogate faithfulness scores via striped bars.
and artificially sparsifying the Representer Points surrogate model. We first sort the absolute coefficients [Atel for all classes e and datapoints 2,- Afterwards, we set all cocfficients except for those with highest absoluto value to zero. The number of non-zero coefficients for each class set to the number of support vectors defining a trained DualDA surrogate for C' E {10-1,10 10-3, 10100-5 $3 (an overview over the mumber of support vectors for cach class and datasct be found in Figure F.7 in Appendix F
The striped bars in Figure 0 present the results for these artificially sparsified Representer Points surrogales for the AwA2 dataset and in Figure F.3 in Appendix F.2 for the remaining datasets. For better readability, the striped bars are placed next to the bar corresponding to the DualDA setting at the same sparsity level, i.c we maintain the same number of support vectors per class for Representer Points as in the corresponding DualDA configuration. The results indicate that the quality of the Representer Points surrogate decreases rapidly as the sparsification intensifies. The decrease in faithfulness metrics is less severe in CIFAR- 10, however for the other datasets, surrogate faithfulness. This indicates that in order to achieve sparse data attribution, it does not suffice to choose attribution method and post-hoc sparsify artificially for optimal results it is required to choose a method that creates inherently sparse attributions.
that already the first sparsity level corresponding to C = 10-5 has low
5 XDA Experiments
XDA provides unprecedented insight into the interaction between training and test samples by simultaneously highlighting relevant features in both input domains, thereby illuminating the similaritics and dissimilarities that drive their relationship. This approach transforms previously opaque attribution values into interpretable explanations, offers more nuanced understanding of model predictions, and enables feature-level ablation of training test samples to analyze and modify training dynamics. To demonstrate XDA's capabilities, we apply the approach to small convolutional network trained on the MNIST datasct, and VGG-16 network trained on AwA2 and the ImageNet dataset (Deng el al. 2009) respectively. Within the LRP framework, different formulas, or "rules", exist to distribute the relevance from the output back to the input space. For our experiments, we use the recommended rule composite EpsilonPlusFlat, as implemented in the zennit toolkit (Anders et al. (021), which applies the flat rule for any linear first layer, the z-plus rule for all other convolutional layers and the epsilon rule for all other fully connected layers. For VGG-16, additionally
16

---

Published in Transactions on Machine Learning Research (12/2025)
apply the flat rule for the first 10 layers for better legibility of the corresponding relevance heatmaps. An explicit description of the corresponding formulas is given in (Montavon el. al. 2019
5.1 Qualitative Case Study: Gaining Insight with XDA
Using an exemplary sample from, the AwA2 dataset, step-by-step guide the reader through the inter- pretation of XDA explanation with the help of Figure 7 ① The trained model classifies this image a wolf. The corresponding LRP heatmap shows which parts of the image were relevant for the decision: Red areas support the model's decision (the ears, the fur, and parts of the background), while the model considers blue areas (the eyes and the smout) contradictory to its classification. Surprisingly, the animal's characteristic facial features are considered as opposing the classification as a wolf. While LRP and other feature attribution methods can show us where informative features are picked up by the model for the classification, they cannot explain why the model has identified these arcas based on the training data. We can answer this question by the characteristic of XDA of splitting the single LRP explanation into individual, training- laaniingddta-speeiffi attributions (which, when summed up, equate to the LRP explanation again). ② The right part of the middle row depicts the strongest proponents for the classification of the test sample as class wolf, together with their label and corresponding DualDA attribution value. All shown most influcntial training images depict wolves in a grassy or lealy habilat. @ The XDA maps of the lest sample altribute the corresponding DualDA attribution for each training sample onto the input test image. Note that all of these heatmaps unique: For the first two training samples, where snouts of the animals are only hardly or not visible, the snout area of the test sample is negatively only barely positively relevant according to the attribution. This is different for the third and fourth sample, where the eyes and snout are clearly visible, and the corresponding part of the test sample receives high positive attribution scores. Note that for all pairs of test and training image, the image areas depicting the wolves' cars and fur are attributed with positive relevance. 4 The XDA heatmaps computed in the input domain do not only connect DualDA attributions to the test sample fcatures, but also to the features of the influential training samples. We observe that for the third and fourth most positively influential training samples, the snout and eyes are attributed most strongly, indicating high responsibility of those features for the sample's influence on the prediction. 5 The center row, from the midpoint to the left, depicts the strongest opponents against the classification of the test sample as a wolf, identified by the strong negative attribution scores received from DualDA, Note that all samples depict instances of class "fox". " c The XDA test healmaps indicate that the eyes and snout are responsible for the high negative attributions for these training samples. We recognize from the shown training samples of class "fox", representing influential training data subscts where the depicted animals all share similar facial features as the wolf in the test image, that the model must have associated this characteristic canine facial structure more to class "fox" than to class "wolf", which in turn influences the currently analyzed prediction. ① In the XDA heatmaps of the training samples, again find that the corresponding eyes and noses are most relevant for the negative attribution. Altogether, XDA explains the a priori unanticipated result from the LRP heatmap in I which causes the facial area of the wolf to be negatively relevant for the classification as a wolf: While there exist some other wolves in the training set with a similar muzzle, which the model correctly uses to detect these training samples as relevant for the classification of the test sample, the training corpus also contains multiple other canines such as foxes, making the snout an unreliable indicator for wolves in the AwA2 dataset.
Additional examples of DualXDA explanations for samples from the MNIST, AwA2, and Imagenet dataset. are presented in Appendix G
5.2 Ablation Study
To ascertain the capability of XDA to explain counterfactual effects, we additionally conduct a controlled ablation study on the MNIST datasct, the results of which are depicted in Figure g. Figure Sal shows the original lest sample logether with its five most relevant training samples and the corresponding XDA heatmaps for the test sample. For the first three training samples with the highest relevance, the upper segment of the test sample is positively influential for their relevance. On the other hand, for the last two training samples, which are attributed with lower relevance and do not have a similar are, the upper segment holds much less relevance. First, we perturb the test sample by removing the base of the digit "2" and keeping only the upper
17

---

Published in Transactions on Machine Learning Research (12/2025)
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
Locel
Label
Libst
Libal not
Test
XDA (ours)
Figure 7: Overview figure showcasing the use of DualXDA on the AwA2 dataset. ! Test sample classified as wolf and corresponding LRP heatmap. @ Strongest proponents with corresponding label and DualDA attribution (attribution reduces in the direction of the arrows). 3 XDA test heatmaps relevance proponents. ① XDA training heatmaps relevance of proponents. ① Strongest opponents with corresponding label and DualDA attribution. ④ XDA test heatmaps relevance of opponents. 7 XDA training heatmaps relevance of opponents.
loop of the test sample. We now recompute DualDA attributions for this perturbed sample and present them with the corresponding XDA heatmaps in Figure 8b yet keep observing the training samples as previously identified via DualDA for the unperturbed test point. We can observe that for the two rightmost training samples the attribution scores are reducod by approximately 70%, which is proportionally higher than the reduction in altribution scores on the three highest altributed training samples at approximately 60% on average. Furthermore, the ordering of the training samples by attribution is not changed by the ablation.
In second variant of our experiment, we intervene by removing the upper from the test sample and calculate the DualDA attribution for the same five training samples again, as depicted in Figure Sc Now, the relevance of the first three training samples reduces approximately by 67% on average. In contrast, the relevance for the last two training samples decrcases by only approximately 51% and the attribution scores remain at higher levels than in Figure 8b where the base of the numeral was removed. The training samples which previously have received lower attribution scores are now the second and third most attributed sample respectively. After the ablation, the attribution maps of all five training points are similar, indicating that the increased relevance of the first three training points before ablation be explained with the presence of the upper are in the test sample. This is further supported by observations on the XDA heat for the modified test sample, which now appear more similar across different training samples; any extent of positive attribution associated with the are has vanished.
6 Limitations & Future Work
The current formulation of DualXDA employs a multiclass SVM surrogate to achieve Data Attribution, which restricts the demonstrated applications to classification settings. However, the framework does not rely on this restriction, and alternative sparse kernel- models (c.g., Support Vector Regression (Drucker let al. 1996), One- Class SVMs (Schōlkopf et al., 200T), Incremental SVMs (Cauwenberghs & Poggio 20000) or RVMs (Tipping, 2001)) can be used to extend DualXDA regression, outlier detection, active learning, or provide probabilistic formulations.
Furthermore, the evaluation of DualXDA in the present study is limited to image classification models of moderate scale, and does not yet include a comprehensive quantitative assessment on contemporary large-scale or multimodal foundation models beyond the exploratory analyses provided in Appendix | The method's reliance on penultimate-layer representations suggests compatibility with other network families, including transformers. Our preliminary observations Appendix suggesst that DualXDA may produce meaningful attributions for large transformer models in specific scenarios, although systematic evaluation across model types, tasks, and data modalities remains to be conducted in future work.
18

---

Published in Transactions on Machine Learning Research (12/2025)
XDA Train
POSITIVELY relevant training samples
Test sample
2
12.21. Label
Releoroc Label:
Relwarce:
Label:
Latel
XDA Test
(a) DualDA attributions and XDA heatmaps for the unperturbed test sample. 'The three most attributed training points (left) share similar top are as the test sample. In the corresponding XDA heatmaps, this are receives a large quantity of positive attribution scores. The other two training samples lack the arc, accumulating attribution at their base.
XDA Train
Test sample
Lazel
Label
Lazel
Labd: 2
Label 2
XDA Test
(b) When removing the base of the test sample and attributing the perturbed image that only consists of the top arc, attributions are reduced, but the ordering induced by the data attribution scores remains the same. The XDA training sample heatmaps show weakened attribution at the digit base. All positive relevance has vanished from the bottom the test heatmaps.
XDA Train
Test sample
4.5 Lase:
2 Label:
Lasel
Label: 2
Lsbel:
XDA Test
(c) When removing the upper half of the test sample and only leaving the base, the attribution of the three most attributed training points significantly reduced. Conversely, attribution for the remaining two training points is leas affected. They now become the second and third highest attributed training samples respectively.
Figure 8: Ablation study of XDA explanations for the mumeral "2" from the MNIST datasct. We take a test sample and the five most highly positively attributed training samples and evaluate how the altribution and the corresponding XDA heatmaps change when removing different parts of the digit.
19

---

Published in Transactions on Machine Learning Research (12/2025)
The evaluation of XDA currently remains qualitative in nature. While the visualized attribution produced by XDA provide insight into feature-level interactions between training and test samples, quantitative evaluation metrics for such explanations are not yet established. Existing metries for attribution map assessment have been developed for featuro-attribution scttings (Hedström et al., 2023) and do not directly apply to feature level attributions revealing interactions between training and test datapoints (instead of explaining classifier outputs), making the development of suitable evaluation measures an important direction for future work.
We note that the training of surrogate SVM models can introduce computational overhead for large training sets, although practical efficiency can be improved using early stopping or GPU-accelerated SVM implementations.
Finally, prior research suggests that combining Data Attribution methods over an ensemble of models may improve altribution efficacy. Considering the computational efficiency of DualXDA, this approach provides another potential avenue for extending DualXDA.
7 Discussion & Conclusion
In this work, we introduce DualXDA, a framework that enables efficient and sparse data attributions, coupled with feature-level explanations which clarify how specific training samples influence the prediction for a given test point. DualXDA consists of two parts: First, DualDA is efficient surrogate- DA method for explaining neural networks using the penultimate hidden representations to define a kernel map to fit a multiclass SVM. This strategy naturally provides global as well as local attributions as an approximate decomposition of the model output. Additionally, DualDA produces sparse attributions by virtue of the associated optimization process, which is beneficial for reducing the explanations' complexity and volume. Sccond, XDA combines data attribution with fcature attribution by mapping data attribution values back to the train and Lest. input domain simultaneously, to identify relevant. corresponding features and explain previously opaque data attribution results. Together, our framework overcomes the computational inefficiency of popular DA approaches and allows the user to gain more fine-grained insights and novel analytical capabilities.
Morcover, we thoroughly evaluate our DualXDA framework using both qualitative analysis and quantitative metrics to assess its performance. First, we present the results of an extensive quantitative evaluation which compares eight trate-of-the-art DA methods across seven evaluation metrics, building on existing evaluation approaches from literature, three datasets and deep neural network architectures. We further report the memory and runtime requirements of all evaluated methods. To the best of our knowlodge, this is the most exlensive quantitative evaluation of DA methods lo date. Our results strongly suggest that surrogale-basel DA methods can help reduce the computational requirements over several orders of magnitude, without sacrificing attribution quality. Our proposed mothod DualDA in particular outperforms statc-of-the art methods for numerous evaluation metrics while executing up to six orders of magnitude faster than competing approaches. DualDA also exhibits the lowest memory consumption among all methods that use caching. These improvements above the previous hate-of-theeart can be significant in the context of applications under strict constraints on computational resources.
Additionally, we investigate the sparsity of the resulting attributions. DualDA achieves its remarkable performance while producing sparse attributions, which results in insightful attributions that are more easy to interpret than from competing methods. The resulting reduction in cognitive load is crucial to analyze and understand model behaviour modern applications with a vast number of training samples. For use cases in which less sparse attributions may be desirable, we show how to effectively tune the sparsity using the hyperparameter C. Our experiments show that the hyperparameter C. is vital for the effectiveness of the attributions features for downstream tasks, and can be chosen carefully by the user depending on the application at hand. As DualDA has only a single hyperparameter, it is more straight to apply and more user-friendly than competing methodologies.
Finally, we employ XDA to gain deeper insight into the DualDA attributions across three different datasets. Through qualitative analysis, we demonstrate how XDA explains muances in model predictions and confounding factors in training data, while showing robustness in input ablation studies. To our knowledge, this is the first work to propose methodology to explain the interaction of training and test datapoints in the inference
20

---

Published in Transactions on Machine Learning Research (12/2025)
context of trained model from both perspectives, feature level. By unifying the attributions in two orthogonal approaches in XAI, feature altribution and data attribution, we obtain a method that leverages the benefits of both approaches, constituting a novel explanation paradigm.
Acknowledgements
We would like to express our gratitude to Mclina Zeeb for her work in the creation of Figure 1 as well as guidance on the visual representation of the results. This work was supported by the European Union's Horizon Europe research and innovation programme (EU Horizon Europe) as grants [ACHILLES (101189689), TEMA (101093003)), the Fraunhofer Internal Programs as grant PREPARE (40-08394), the Federal Ministry of Education and Research (BMBF) as grant BIFOLD (01IS18025A, 011S180371I) and the German Research Foundation (DFG) research unit DeSBi [KI-FOR 5363) (459422098).
21

---

Published in Transactions on Machine Learning Research (12/2025)
References
Reduan Achtibat, Maximilian Dreyer, Πlona Eisenbraun, Scbastian Bosse, Thomas Wiegand, Wojeiech Samek, and Sebastian Lapuschkin. From attribution maps to lurnan-uuneestarddabe explanations through concept relevance propagation. Nature Machine Intelligence, 5(9):1006-108)) 2023.
Reduan Achtibat, Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Aakriti Jain, Thomas Wiegand, Sebastian Lapuschkin, and Wojciech Samek. AttaLRP: Attention-Aware Layer- Wise Relevance Propagation for Transformers. In Proceodings of the 41st International Conference on Machine Learning, volume 235, pp. 135-168 PMLR, 2024.
Naman Agarwal, Brian Bullins, and Elad Hazan. Second- Stochastic Optimization for Machine Learning in Linear Time. Journal of Machine Learning Research, 18(116):1 40, 2017.
Christopher J. Anders, David Neumann, Wojciech Samek, Klaus- Albur-Robert Müller, and Sebastian Lapuschkin. Software for datasel-wide xai: From local explanations to global insights with Zennit, CoRelAy, and ViRelAy. CoRR, 0bbs/2106.13000 2021.
Scbastian Bach, Alexander Binder, Grégoire Montavon, Frederick Klauschen, Klaus-Robert Müller, and Wojeiech Samek. On pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. PLoS ONE, :011),0013 2015.
Juhan Bae, Nathan Ng, Alston Lo, Marzyeh Ghassemi, and Roger B Grosse. If Influence Functions are the answer, then what is the question? Advances in Neural Information Processing Systems, 35:17953-17908 2022.
Dilyara Bareeva, Galip Umit Yolcu, Anna Hedström, Niklas Schmolenski, Thomas Wiegand, Wojciech Samek, and Sebastian Lapuschkin. quanda: An Interpretability Toolkit for Training Data Attribution Evaluation and Beyond, 2024.
Umang Bhatt, Adrian Weller, and José M. F. Moura. Evaluating and aggregating feature-based model explanations. In Proceedings of the Twenty-Ninth International Joint Conference Artificial Intelligence, LJCAI20, 2021. ISBN 9780999241165.
Gert Cauwenberghs and Tomaso Poggio. Incremental and decremental support vector machine learning. Advances in Neural Information Processing Systems, 13, 2000.
Prasad Chalasani, Jicfeng Chen, Amrita Roy Chowdhury, Xi Wu, and Somesh Jha. Concise Explanations of Neural Networks using Adversarial Training. In Hal Daumé III and Aarti Singh (eds.), Proceedings of the 37th International Conference Machine Learning, volume 119 of Proceedings of Machine Learning Rescarch, pp. 1383-1391. PMLR, 07 2020.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input Similarity from the Neural Network Perspective. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Wei-Lin Chiang, Mu-Chu Lee, and Chih-Jen Lin. Parallel dual coordinate descent method for large-scale linear classification in multi-core environments. In Proceedings of the 22Nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, pp. 1485-1494, 2016.
Julien Colin, Thomas Fel, Rémi Cadène, and Thomas Serre. What cannot predict, I do not understand: A human- centered evaluation framework for explainability methods. Advances in neural information processing systoms, 222222 2022.
Corinna Cortes and Vladimir Vapnik. Support-vector networks. Machine learning, 20:273 297, 1995.
Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernel- based vector machines. Journal of machine learning research, 2(Dec):265 2922, 2001.
22

---

Published in Transactions on Machine Learning Research (12/2025)
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hicrarchical image database. In 2009 IEFE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009.
Maximilian Dreyer, Erblina Purelku, Johanna Vielhaben, Wojeiech Samek, and Sebastian Lapuschkin. PURE:
Turning polysemantic neurons into pure features by identifying relevant circuits. In Proceedings of the IEEE/CV Conference on Computer Vision and Pattern Recognition, pp. 8212 8217, 2024.
Harris Drucker, Christopher Burges, Linda Kaufman, Alex Smola, and Vladimir Vapnik. Support vector regression machines. Advances in neural information processing systems, 9, 1996.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The Llama Herd of Models. arXiv e-prints, pp. arXiv-24000 2024.
Oliver Eberle, Jochen Büttner, Florian Kräutli, Klaus-Robert Müller, Matteo Valleriani, and Grégoire Montavon. Building and Interpreting Deep Similarity Models. IEEE Transactions on Pattern Analysis and Machine Intelligence, 44(3):1149-144 1161, 2022.
Minghong Fang, Neil Zhenqiang Gong, and Jia Liu. Influence Function based data poisoning attacks to top-n recommender systems. In Proceedings of The Web Conference 2020, pp. 3019 3025, 2020.
Gauthier Gidel, Francis Bach, and Simon Lacoste-Julien. Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networks. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d' d'Alché-But F. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, Evan Hubinger, Kamilé Lukošiuté, Karina Nguyen, Nicholas Joseph, Sam McCandlish, Jared Kaplan, and Samuel R. Bowman. Studying Large Language Model Gencralization with Influence Functions, 2023.
Riccardo Guidotti. Counterfactual explanations and how to find them: literature review and benchmarking. Data Mining and Knowledge Discovery, 38(5))2770-2824 2024.
Suriya Gunasckar, Jason D Lee, Daniel Soudry, and Nati Srebro. Implicit Bias of Gradient Descent on Lincar Convolutional Networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
Chengeheng Guo, Bo Zhao, and Yanbing Bai. Deepcore: A comprehensive library for coreset selection deep learning. In International Conference on Database and Expert Systems Applications, pp. 181- 195. Springer, 2022.
Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fast Scalable Influence Functions for Efficient Model Interpretation and Debugging, 2021.
Zayd Hammoudeh and Daniel Lowd. Identifying a Training-Set Attack's Target Using Renormalized Influence Estimation. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, CCS '22, pp. 1367-1381, New York, NY, USA, 2022. Association for Computing Machinery.
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of Himilarity-based Explanations. In International Conference on Learning Representations, 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Anna Hedström, Leander Weber, Daniel Krakowczyk, Dilyara Bareeva, Franz Motzkus, Wojciech Samek, Sobastian Lapuschkin, and Marina M-C Höhne. Quantus: An explainable AI toolkit for responsible evaluation of neural network explanations and beyond. Journal of Machine Learning Research, 24(34):
1-11, 2023.
23

---

Published in Transactions on Machine Learning Research (12/2025)
Don Hush, Patrick Kelly, Clint Scovel, Ingo Steinwart, and Bernhard Schölkopf. QP Algorithms with Guaranteed Accuracy and Run Time for Support Vector Machines. Journal of Machine Learning Research, 7(5), 2006.
Andrew Nyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry. Datamodels: Understanding Predictions with Data and Data with Predictions. In Kamalika Chaudhuri, Stefanie Jegelka, Le Song, Csaba Szepesvari, Gang Niu, and Sivan Sabato (eds.), Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pp. 9525 9587.
PMLR, 07 2022.
Tomoharu Twata and Yuya Yoshikawa. Training Deep Models to be Explained with Fewer Examples, 2021.
Ziwei Ji and Matus Telgarsky. Gradient descent aligns the layers of deep linear networks. In 7th International Conference Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenRe- view.net, 2019.
Ayrton San Joaquin, Bin Wang, Zhengyuan Liu, Nicholas Asher, Brian Lim, Philippe Muller, and Nancy F Chen. In2core: Leveraging influence functions for coreset selection in instruction finetuning of large language models, 2024.
Selvarak Sathiya Keerthi, Sellamanickam Sundararajan, Kai-Wei Chang, Cho-Jui Hsieh, and Chih-Jen Lin. A sequential dual method for large scale multi-class linear SVMs. In Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 408 416, 2008
Been Kim, Martin Wattenberg, Justin Gilmer, Carrie Cai, James Wexler, Fernanda Viegas, et al. Inter- pretability beyond feature attribution: Quantitative Testing with Concept Activation Vectors (TCAV). In International conference on machine learning, pp. 2668-26677 PMLR, 2018
Pang Wei Koh and Percy Liang. Understanding Black-box Predictions via Influence Functions. In Doina Precup and Yee Whye Teh (eds.), Proceedings of the 34th International Conference on Machine Learning, volume 70 of Procoedings of Machine Learning Resoarch, pp. 1885-1894. PMLR, 08 2017.
Narine Kokhlikyan, Vivek Miglani, Miguel Martin, Edward Wang, Bilal Alsallakh, Jonathan Reynolds, Alexander Mclníkov, Natalia Kliushkina, Carlos Araya, Siqi Yan, and Orion Reblitz-iicharson.. Captum: A unified and generic model interpretability library for PyTorch, 2020.
Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images, 2009.
Harold W. Kuhn and Albert W. Tucker, Nonlinear Programming, In Proceedings of 2nd Berkeley Symposium. Berkeley: University of California. Press, pp. 481-492, 1951.
Faisal Ladhak, Esin Durmus, and Tatsunori Hashimoto. Contrastive Error Attribution for Finetuned Language Models, 2022.
Sebastian Lapuschkin, Stephan Wäldchen, Alexander Binder, Grégoire Montavon, Wojciech Samek, and Müller. Unmasking Clever Hans predictors and assessing what machines really learn. Nature Communications, 10(1):1096, 2019.
Yann LeCun, Corinna Cortes, and CJ Burges. MNIST handwritten digit database. ATT Labs [Olinee] www.n.com cow.unc.o///dbbbbb /ee////ddb/ii/eee mnnsst, 2, 2010.
Available:
Jinkun Lin, Anqi Zhang, Mathias Lécuyer, Jinyang Li, Aurojit Panda, and Siddhartha Sen. Measuring the effect of training data on deep learning predictions via randomized experiments. In International Conference on Machine Learning, pp. 13468 13504. PMLR, 2022.
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui Hc. Influence Sclection for Active Learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9274-997999 7774,928,,,11 10 2021.
Kaifeng Lyu and Jian Li. Gradient Descent Maximizes the Margin of Homogeneous Neural Networks, 2019.
24

---

Published in Transactions on Machine Learning Research (12/2025)
Grégoire Montavon, Alexander Binder, Sebastian Lapuschkin, Wojeiech Samek, and Klaus-Robent Müller. Tayer-wise Relevance Propagation: An overview. In Explainable AI: Interpreting, Explaining and Visualizing deep learning, volume 11700, pp. 193 209. Springer, 2019.
Jonas R. Naujoks, Aleksander Krasowski, Moritz Weckbecker, Thomas Wiegand, Sebastian Lapuschkin, Wojeiech Samck, and René P. Klausen. PINNftuence: Influence Functions for Physics-Informat Neural Networks, 2024.
Sung Min Park, Kristian Georgiev, Andrew Nyas, Guillaune Leclere, and Aleksander Madry. TRAK: attributing model behavior at scale. In Proceedings of the 40th International Conference on Machine Learning, (3. JMLR.org, 2023.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In Н. Wallach, H. Larochelle, A. Beygelzimer, F. d' A'cch- Buc, E. Fox, and R. Garnett (eds.), Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019.
Fabian Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Griscl, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12:2825 2830, 2011.
Pouya Pezeshkpour, Sarthak Jain, Byron C. Wallace, and Sameer Singh. An Empirical Comparison of Instance Attribution Methods for NLP, 2021.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating Training Data Influence by Tracing Gradient Descont. In H. Larocholle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (cds.), Advances in Neural Information Processing Systems, volume 33, pp. 19920-19930. Curran Associates, Inc., 2020.
Hrithik Ravi, Clay Scott, Daniel Soudry, and Yutong Wang. The implicit bias of gradient descent on separable multiclass data. Advances in Neural Information Processing Systems, 37:81324-81323 2024.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. "Why Should Trust You?": Explaining the Predictions of Any Classifier, In Proceedings of the 22nd ACM SIGKDD International Conference Knowledge Discovery and Data Mining, KDD '16, pp. 1135-11444 New York, NY, USA, 2016. Association for Computing Machinery. ISBN 9781450342322.
Wojeiech Samek, Grégoire Montavon, Andrea Vedaldi, Lars Kai Hansen, and Robert. Müller (eds.). Explainable Al: interpreting, explaining and visualizing deep learning, volume 11700 of LNCS. Springer Nature, 2019.
Wojeicch Samck, Grégoire Montavon, Scbastian Lapuschkin, Christopher Anders, and Klaus-Robert Müller. Explaining deep neural networks and beyond: A review of methods and applications. Proceedings of the IEEE, 109(3):245 2021.
Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up Influence Functions. Proccedings of the AAAT Conference on Artificial Intelligence, 36(8):8179-85 198888697,6,,,, 2022.
Bernhard Schölkopf, John C Platt, John Shawe-Taylee Alex J Smola, and Robert C Williamson. Estimating the support of a high-dimensional distribution. Neural computation, 13(7):1443-144 2001.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. Poison frogs! Targeted clean-label poisoning attacks on neural networks. Advances in Neural Information Processing Systems, 31, 2018.
Vasu Singla, Pedro Sandoval- -Sguur,, Micah Goldblum, Jonas Geiping, and Tom Goldstein. A Simple and Efficientt Baseline for Data Attribution on Images, 2023.
25

---

Published in Transactions on Machine Learning Research (12/2025)
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasckar, and Nathan Srebro. The Implicit Bias of Gradient Descent on Separable Data. Journal of Machine Learning Research, 19(70):1-57, 2018.
K. S. Srikantan. Testing for the Single Outlier in Regression Model. Sankhya: The Indian Journal of Statistics, Series A (1961-2002), 23(3):251-28) 260, 1961.
Davoud Ataee Tarzanagh, Yingcong Li, Christos Thrampoulidis, and Samet Oymak. Transformers as Support Vector Machines, 2023.
Neil Thompson, Kristjan Greenewald, Keeheon Lee, and Gabriel Manso. The Computational Limits of Deep Learning, 07 2020.
Michael E. Tipping. Sparse Bayesian learning and the relevance vector machine. Journal of Machine Learning Rescarch, ((Jun):211-11
2001.
Alexander Warnecke, Daniel Arp, Christian Wressnegger, and Konrad Rieck. Evaluating Explanation Methods for Deep Learning in Security. In 2020 IEEE European Symposium on Security and Privacy (EuroS&P), pp. 158 174, 2020.
Dennis Wei, Inkit Padhi, Soumya Ghosh, Amit Dhurandhar, Karthikeyan Natesan Ramamurthy, and Maria Chang. Final-Model-Onff Only Data Attribution with a Unifying View Gradient- Methods, 2024.
Yongqin Xian, Christoph H Lampert, Bernt Schiele, and Zeynep Akata. Zero-shot learning comprehensive evaluation of the good, the bad and the ugly. IEEE transactions on pattern analysis and machine intelligence, 41(9):2251 2265, 2018.
Chih-Kuan Ych, Joon Kim, Ian En-Hsu Yen, and Pradcep K Ravikumar. Representer Point Selection for Explaining Deep Neural Networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa-Bianchi and R. Garnett (eds.), Advances in Neural Information Procossing Systems, volume 31. Curran Associates, Inc., 2018.
Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text classification. Advances in Neural Information Processing Systems, 28, 2015.
Chen Zhu, W Ronny Huang, Hengduo Li, Gavin Taylor, Christoph Studer, and Tom Goldstein. Transferable clean-label poisoning attacks on deep neural nets. In International conference on machine learning, pp. 7614-7623. PMLR, 2019.
26

---

Published in Transactions on Machine Learning Research (12/2025)
A Derivation of DualDA using the SVM framework
Given the soft-margin optimization problem of (Crammer & Singer 2001
minw
W 18 + C CE Ei
s.t.
Vi [N], VC E [K]
(A.1)
(w (b) - (w: li) + dey, 2 &
where dey. =1 = y:) denotes the Kroenecker delta function, we derive the Lagrangian
K
N
1 2
C(W,E.D,a)
1w:11
C
aio [[w6 wyr) fs + Sew - -6l
(A.2)
To enforce the stationarity condition variables and set them to 0:
Kuhn & Tucker
1951
take derivatives with respect to the primal
Oc
K
E [N]:
C Oie =0
Vi [M]
Σ
aisc
C
(A.3)
gg,
7L
uk
aakfi
Σ
Oie f==0
(A.4)
0wk
=C by
A.3
N
= 4k E
[K]
wk =
Σ
(C8 auxff
(A.5)
1=1
From Equations A.3 and A.5
we derive:
K
Vi E [
Σ
Oiec
ViE (N,C Cigs = Σ Oiic
(A.6)
Citl #C
N (C88gee-a ateffi
VC E [K] we
[c )f Σ
aieh
(A.7)
1=1
tigie
Combining these two results, we have:
W
Σ ask)
Σ aseft
(A.8)
As such, the output of the surrogate model on input x is given by:
Vc [K] (<x;; 0, W). w. f(x;
E ands? f(;
Σ aief. aieh f(z; 0)
(A.9)
Thus, the output of the network for any class a linear combination of the inner products of the test point with training points. This motivates our choice in Equation
To solve this problem, we first derive the dual problem by plugging Equation (A.8 in the Lagrangian and
maximize it subject to the dual feasibility and complementary slackness conditions Kuhn & Tucker 1951)
max 2 E f3) E ai.c) aj.c)]
MN Cigs
S.1.
(A.10)
ΣK Oiee C and Vc E [K], aie 0.
27

---

Published in Transactions on Machine Learning Research (12/2025)
We refer the reader to (Crammer & Singer 2001 for the derivations.
After solving this problem, we obtain a nonnegative real matrix A E X consisting of the dual variables dijer which constitute the multiplier of each training datapoint in the final decision. Each row i of A corresponds to a training datapoint fs. Each entry in the matrix contains a positive scalar are corresponding to class c.
Each datapoint æ; contributes negatively to decisions for classes other than its own class ys (classes y)), and the multiplier for this contribution is dic-
Each datapoint 2₁ contributes positively to decisions for its own classes (c = V:), and the multiplier for this contribution is the sum of the mealtipliers for other classes: Enty, Oik-
The variables aig, are the values that satisfies the Karush-Kult Kuhn-Tucker condition expressed Equation A.3 concerning positivity of slack variables.
Note that only subset of the training data will have positive attributions, since all the datapoints outside the maryin will have all zero values in the corresponding rows of A.
B Derivation of DualDA using the Representer Points Framework
To formulate the training of an SVM in the spirit of Representer Points Selection, we first reformulate the Multiclass SVM problem as an optimisation problem with the following loss function:
1
L(W: P.. vi)
W
Σ
max {1 [deg + w Si] CE[K]
(B.1)
W = argmin L vft
(B.2)
w
Consider data points that do not lie preciscly on the decision boundary between classes, such that the argmax operation in the hinge loss function yields unique class label. This is true for most datapoints, except few which are situated exactly on the margin. Denote in the following
c² = arg max {1 [Jog + 1. w ft]}
(B.3)
cE[K]
and
Be(s) = =w0
(B.4)
For class é', we have
if c' Vi c²
i 032 (f:)
C if c' = c* # y1
C max {1 15eg, + By, (fa) Be(fi))
(B.5)
cE[K]
0 if c' = yi c*
0 + yi, d' # c*
since an infinitesimal step docs not change the output of the argmax operator. Setting ((Be(f)) ye W) C ma [1 [6rg + Sy, (f:) - (.())) and replacing terms in Equation (D.6) we obtain the Representer
Points attribution using the Hinge loss:
Cf. fret
c = yi if c = c*
HingeRP
(Irest, i)
dl (l(4(z;;)) 0), v.) 04(2; 0)
f(
-CS, Jurst
(B.6)
0
if c Vi c² if c + Vi, c c#c c²
0
We now establish that these attributions coincide with those of DualDA up to positive scaling. This equivalence is proven in the next section via Influence Functions derivation of DualDA.
28

---

Published in Transactions on Machine Learning Research (12/2025)
C Derivation of DualDA using the Influence Functions Framework Proof of Theorem 3.1
To prove Theorem 3.1 we again rely on the formulation given by Equations [.1 and B.2p.. We seek quantify the effect of infinitesimally downweighting the loss contribution of the i-th training sample (fi, vs) by an infinitesimal e.
(f v/) (. =C max {1 [Bem, + wy we [,]]
(C.1)
cE[R]
W arg min L' ST. W
(C.2)
For given test point 2test with features fiest f fætesti 0) and an arbitrary class c, our objective is determine how the class score w. ftest responds to the infinitesimal downweighting of the i-th datapoint. Using Corollary 1 in (Naujoks et al. 2024), we derive
W Sreat Tot frest frest Vw (10g frest)
Hw V (c (Cmax cE[K] (1 [deg, + W - wo f1]) + O(e), (C.3)
where Hw is the Hessian of the loss function L, so Hw is equal to the identity matrix. As before, under the assumption that the maximum achieved by a unique class index, denote in the following
arg max {1 [day. +w cE[K]
& Ji a sa]
(C.4)
For classes and d, we have that
a 20 freet =f ftost if d
-10
(C.5)
else
and furthermore
fr
if y1 + C*
S if = c² # 91
V o max {1 [day, + Wi N sā]} = [E]
(C.6)
0 if c' = = c*
0
if 4 Wi d' c4C c*
So overall
T. T Stest - wit ftest
Vw (5 (amax (1
Σ V w (w) (w. Freat)
[5om + +w T - w. ril} +((E)
<E(K]
Cf frest
if yi
ch frest
if ifc c²
(C.7)
0 if yi c
0
if if Wi. c c*
We now prove that this is indeed the same the attributions calculated by DualDA, which are given by
DD Tc
(Irest 1) = Eepp. aieft fust -Oiefe fust
yt = else
(C.8)
To establish this result, we consider two distinct cases. For correctly identified training points, we have Yi = c By complementary slackness the SVM, we have for the dual variables a; for # yi and therefore Eosy, aic So
DD
(Itests i) ={ Ecfpr Chick Stest
ifc y1 = c²
(C.9)
Csaff T fust = 0
if c yi = c*
29

---

Published in Transactions on Machine Learning Research (12/2025)
For incorrectly identified training points, we have ys # c² then by complementary slackness dic 0 for # c² and since EeE(c] Olie C, it follows that Oic = C. Therefore
{
(C (00-ase)fe ase)s frest frest
ffes
if y/4 # if # yıs = c² if c + yı, # c
DD c
(Irest:))
aiefi fi frest OieJi fwst
-Cfi frest
(C.10)
D Technical Details of Experiments
D.1 Data Attribution Methods
As DualDA is already defined in Section 3 we now specify the other DA methods we compare our approoach to in this paper. In all experiments, we employ GPU implementations and follow the best practices as suggested by original authors of the papers or code bases, as explained in the individual sections. This includes the use of random projections and CUDA-level implementations to efficiently perform these projections for TRAK. For the caching phase of DualDA, the extraction of latent features is using GPU capabilities, while the training of the SVM uses CPU. The sequential nature of the underlying SVM optimization algorithm does not allow for a straightforward GPU parallelization of existing implementations (Keerthi et al. 2008 Chiang et al. 2016
D.1.1 Influence Functions
Influence Functions (TF) originates from the field of robust statistics, where was initially used for the analysis the effects of outliers on linear models (Srikantan) 1961; Hammoudeh & Lowd. 2022). Koh and Liang (Koh & Liang 2017) proposed using this method for local data attribution of deep neural networks. It approximates the effect of discarding training point using second-order Taylor approximation. However, this approximation is conditioned the current trained parameters of the network and hence includes information about the decision making strategies employed by the trained model. Data attribution by IF is given by
Te (z, (=)((((;))) " (o
(D.1)
and Ho Viltrai (0, (0,D) D) is the Hessian of the training loss Erain (0, D) E(E.weD ((((z;)) vi) where E(:,))
is a sample-wise loss function.
Computation of the inverse Hessian is computationally prohibitive for modern architectures with millions of parameters. For this reason, (Koh & Liang, 2017) propose to use the LiSSA algorithm (Agarwal et al.) 2017) which approximates the inverse Hessian vector product ((#(;; e) H, for each test point using an iterative procedure. While this mitigates the computational cost, does not entirely solve the problem: computing explanations for single test point can still take long time, possibly in the order of hours [Hammouchh & Lowd 2022). In our experiments, we have followed the authors' suggestions to determine hyperparameters which produce meaningful influence values while minimizing the required computation time. First, we have used only the final layer of deep neural networks to compute gradients for influence estimation with LiSSA. Secondly, we have set the total number of iterations dedicated to the estimation of the inverse Hessian vector product to the training dataset size in each experiment.
Another estimation of the inverse Hessian achicvod by using the Arnoldi iteration algorithm to approximato its largest eigenvalues and the corresponding eigenvectors. These can then be used for an approximative diagonalization of the Hessian, which is simple to invert. For Arnoldi explanations, we have used 128 as the number of random projections and 150 as the Arnoldi space dimensionality. We have used 10,000 randomly sampled datapoints from the training dataset to estimate the Hessian matrix.
Finally, the authors of Grosse et al. Grosse et al. 2023 propose to solve the EnverseHesssian-eeee product using an Eigenvalue-Coreecte
Cronecker-Factorie Approximate Curvature (EK-FAC), which derived from an approximation of the Fisher information of the network. As hyperparameters, we have used the default values given in the original code release.
30

---

Published in Transactions on Machine Learning Research (12/2025)
D.1.2 Tracin
TracIn circunvents the computational problem of calculating the inverse Hessian by estimating the elfect of discarding a training sample with a first-order Taylor approximations. However, instead of only considering the final trained model, it aggregates this approximation over the entire training procodure using training checkpoints and the optimised training parameters O, at each epoch e, weighted by the slep size f/--
T Tracin (c,
E epochs
Tee Volo(((2 T Vol(o(00 (())))))
(D.2)
Two more attribution methods can be viewed as simplified version of the former. GradDot functions like TracIn but only considers the final training epoch:
GradDot (z, Vol( Vo((((2;0))))
(D.3)
As small variation, GradCos uses the cosine similarity between gradients instead of the unnormalized inner product:
GradCos (r, i) cos (v) (ol(a((;;0),u)))
T
(D.4)
In the implementation of these mothods, we have employed random projections on gradients to be able to cache training gradients, following (Pruthi et al. 2020 This allowed the attributions to be achieved in a practical time scale. We have used 128 random projections.
D.1.3 TRAK
TRAK is originally motivated by averaging the attribution over multiple models trained on the same data to derive more stable attributions that are model-agnostic and instead only focus the data. However, it also be used as single-model estimator. In this case, it is similar to Influence Functions but uses projected version of the generalized Gauss-Newton approximation to the Hessian.
Let pi denote the prediction probability corresponding the ground truth label of data point zN Let further G (oi: be matrix with columns 9, Vo log 1-p: and Q be diagonal matrix with entries Q = - -p: Then the TRAK attribution for a multiclass classification task is given by:
TRAK (x, = (z; (Z T
(D.5)
Similar to TracIn, TRAK operates on random projections of gradients to achiove feasible runtimes. We use the official code release, following best practices including the GPU level implementation of random projections dranatically decrease runtimes and memory requirements. Following the original publication, we use 2,048 random projections.
D.1.4 Representer Points
Representer Points trains the final (fully-connected) layer until convergence with an added & weight decay regularization. The authors prove a representer theorem this setup, showing that the retrained last layer can be written as a lincar combination of final hidden features of training datapoints. Therefore, the model outpul correspouding class e of the new model can be written as the sum of the following individual contributions. Letting f(..-) denote sample-wise loss function, which accepts a test sample and target class, Representer Points is defined
RP (Irest, i) (E(((xx;;) 0).1:
f. fuet-
(D.6)
00(1; 0)
31

---

Published in Transactions on Machine Learning Research (12/2025)
D.2 Model Training
For all datasets use stochastic gradient descent training without any learning rate scheduling. Experiments are run on Rocky Linux 8.
For MNIST, we use a 6-layer convolutional neural network with 0.001 a learning rate of 0.001.
For CIFAR-10, we train ResNet-118 (He et al. 2016) with random cropping and flipping as data augmentations, with a learning rate of 0.0003. We further include weight decay term with coefficient of 0.01 as part of the loss term.
For A2, we have use a learning rate of 0.001 and augment the data with random horizontal flipping of the images as data augmentation for training a ResNet-50.
E Evaluation Metrics
For evaluating attribution quality, we employ a variety of metrics from the literature and three newly proposed metries, all of which will be introduced in this Section. These metrics are derived from an intuitive notion of data attribution or from emulating downstream applications that utilize data attribution.
Consider a test sample E Diest, and a target class to be attributed. This means that we use DA methods to attribute the corresponding model output to the training dataset. We denote by Th a permutation of training data indices sorted decrensingly by attribution Tx( as assigned by a DA method such that >> j = Telæ, Tf) S Telx, xf())). We let (1( denote the 7th element this list, where we start indexing at 1 (i.e. the first value correspouding to the highest attribution is (1)). Finally, we explicitly denote the Kroenecker delta function 80,6 ò(a, b), and we denote the prediction class for test sample x with c(x) arg max; (x;
E.1
Sanity Checks
Hanawa al. Hanawa et al. 2021 suggest sanity checks to determine the quality of DA methods.
Identical Class Test The authors first posit that reasonable explanation method should produce permutations x that satisfy the following:
Y(r v) E Drest, Vc E [K], (r)(1))
where Diest denotes the test dataset. plain language, the authors posit that the strongest proponents for the classification of a test sample as class e should be training samples of that same class. In practice, we consider the top five attributed samples for the class predicted by the model and report
Σ 5 d ((z))..goo))
(E.1)
(Destt|
Deen
Identical Subclass Test Hanawa al. further suggest that attributions should be able to detect different subpopulations in the same class (Hanawa et al.) 2021 To create this condition, they propose using arbitrary partitioning of classes, and group each set in the partition into super group. In practice, we have selected pairs of classes to group. These pairs of classes are chosen to be dissimilar to ensure that different. strategies are learned for samples belonging to different subclasses. After training a neural network on the modified dataset, a successful DA methods should attribute highly to training samples that belong to the same subclass as the original class the test sample.
To express this condition formally, following (Hanawa al. 2021 let s(-) be an oracle function that returns the true (sub)clase of the dalapoint given its input. Similar to the Identical Class Test, we report
Σ
Σ
8 (s(x), (g)0)
(E.2)
D test
N.yED DeusE
1=1
32

---

Published in Transactions on Machine Learning Research (12/2025)
E.2 Metrics Emulating Downstream Tasks
Many reamstream applications, which initially served as the motivation to several DA methods, can be used for evaluating the effectiveness of DA methods as estimators of model behaviour in counterfactual training sctups. We include two metrics that measure the effectiveness of DA mothods in downstream tasks.
Mislabeling Detection The following metric mimics the downstream task of identifying mislabeled samples in the training data (Koh & Liang] 2017; Yeh et al. [018 Pruthi al.) (020). Correctly labelled datapoints can be classified by model by relying similar training samples of the same class. In contrast, mislabeled samples diverge from the typical feature patterns of their labeled class and consequently require individual memorization rather than generalization. Therefore, mislabelled datapoints are highly influential on their own model prediction, allowing the user to leverage DA approaches to detect mislabeled datapoints. To simulate this scenario, we randomly change the labels of a subset of the training data and retrain the model. We then use the attribution method calculate each training sample's 's self-influence, Ty (z., To quantify the performance of the mislabeling detection by DA method, we traverse the training data set in descending order of self-influence and record the cumulative densily function of the percenlage of mislabeled samples have identified. We report the area under the curve of this cumulative density function, linearly transformed to score between O and 1, where 0 corresponds to the minimum achievable area, 1 represents the maximum achievable area, and random attribution by sampling from a uniform distribution yields an expected score of 0.5.
Shortcut Detection Another candidate downstream application for DA to identify the causes of shorteut learning and backdoor poisoning attacks. If we modify the training dataset to introduce artificial shorteut. to trigger a certain model prediction, DA methods should attribute the responsible training images highly whenever the shorteut feature is present in the test sample. Similarly to previous work (Koh & Liang) 2017 Hammoudeh & Lowd. 2022), we formulate a shortcut detection metric: determine a perturbation w(-) which takes a sample xy and adds a shortcut perturbation on it. Importantly, w applics a consistent and detectable perturbation to all chosen samples. We use a small box in the center, and a frame around the images in our experiments.
We select class c' as the shortcut class, and randomly sample a subset of datapoints from that class to apply W. By retraining the model this perturbed training data, the model learns connection between the artifact and the class c'. We now create particular dataset for evaluation by applying the perturbation to all samples and collecting all perturbed samples, which are not originally from class c' but who, after the perturbation is applied, are classified as class d' i.c. Deval = (w((x) y) (x,y) E D, y y d', arg max c(x)) = c'). This ensures that if a test point now classified as class d, it is primarily due the influence of the uebturbation and therefore, the perturbed training samples should be most influential for the classification as class c'. For all such test points, we report the Area Under the Reccil Curve (AUPRC) score for the detection of perturbed training samples, following (Hammoudeh & Lowd 2022
E.3 Retraining- Metrics
The counterfactual relevance of training data can be empirically assessed through model retraining under an identical training regime with modified training sets. As explained in Section 2.2 this approach is the motivation for methods based on Influence Functions (Koh & Liang 2017). The subsequent metrics evaluate the correspondence between observed changes in model behaviour and the attributions of the training data.
Lincar Datamodcling Score (LDS) LDS measures the effoctiveness of attributions as an estimator for the effect of leaving out training samples by retraining the model on general subsets of the training data (Park) et al. 2023 Given attributions Tx(x,,), and subset of training data indices S [N], we define
E(T S,2) (T(x,
iES
to be the overall attribution of the subset for the output c. We know want to measure how strongly this overall attribution is correlated with the model performance of new counterfactual model, which was trained
33

---

Published in Transactions on Machine Learning Research (12/2025)
only the subset S. To compute the LDS, randomly sample subsets {S1, S2, Sm.. We train independently on these subsets. Finally, we compute the average rank correlations
models { .. , )m) of actual outputs and predicted outputs over the test set:
Σ
((( 1 E [m]] ( 1 [m]})
(E.3)
Dtest
(z.
where :) denotes Spearman rank correlation.
Corseset Selection A downstream application of data attribution is dataset distillation: limiting the training dataset to fewer examples, while maintaining high test accuracy (Guo et al., 2022 Joaquin et al.) 2024). In contrast to LDS, we are not interested the model performance for randomly chosen subsets the training data, but instead for the highest attributed subsct. To mcasure this in metric, we need a stratogy to determine the global importance of a training point for the model performance achieve this, compute the average value of the absolute attributions of training point over the test set. We then restrict the training data to the most globally relevant datapoints and mcasure the test loss of a model retrained on the restricted training set. We report the test loss when retrained on 10% of the data as well as a weighted average
the entire test set. To
CS EpEp(H) ((S((()
EpEp
eCS where (eee(()) (p) is the Lest. loss when retrained only on the most relevant p% of the training data and we use
steps of 10%, ie. p= (10,...,.0) ......,9)), Additionally, report model accuracy variations across different training subsets in Appendix F.44
This is the only metric for which lower values indicate better performance.
Adversarial Data Pruning Inspired by previous work, which investigates the sensitivity of model predictions to ablation of the training data (llyas et al. 2022). we measure the test loss of model that was retrained on a train set which had its most influential datapoints removed. Similarly to Coresct Selection, we consider the highest altributed subset. But contrast, we aim to judge how much exclusive relevant. information this subset holds that would induce an increase in the loss when the subset is removed. We use the same strategy as in Coreset Selection determine the global importance of training samples. We report the test loss when retrained without the 10% most influential data as well weighted average
DP EtEP(H) (pei(p) EpEr
where (v) is the test loss when retrained without the most relevant p% of the training data and we again choose P = (10....,90} As for the Coreset Selection metric, we also report model accuracy across different training subsets in Appendix F.5
F Experiments: Additional Findings
F.1 Metric Scores for MNIST and CIFAR-10
Figures and
show the evaluation results for MNIST and CIFAR-10 datasots respectively.
F.2
Surrogate Faithfulness Scores
Figure F3 shows the surrogate faithfulness scores for DualDA and Representer Points, as well artificially sparsified Representer Points surrogate models.
F.3 DualDA: Effect of sparsity hyperparameter
We evaluate how different choices of the sparsity hyperparameter affect DualDA attribution performance across the evaluation metrics. We therefore evaluate all metrics by choosing C over expanded range from
34

---

Published in Transactions on Machine Learning Research (12/2025)
0.025
0.8
Meiric score
Dua DA
0.015
DuaiDA
C'=
0.4
0.610
DuaiDA
C' = 10-3
DuaIDA
0.005
10-
0.0
0.000
Influmnce Functions
Identical class (f)
Identical subclass test
Mislabeling detection (1)
Shortcut detection (t)
LDS (1)
-0.005
LISSA
Arnoldi
EK-FAC
0.5
0.25
Gradieci Products
Metric score
0.4
Tracin
GradDot
0.3
0.15
GradCos
0.2
0.10
TRAK
0.05
Representer Points
0.0
0.00
Corcsct selcetion
Coreset soloction Aug (1)
Data Pruning
Data Pruning Aug (f)
10%
10% (t)
Figure F.1: Evaluation results on the MNIST dataset. LiSSA and GradCos do not have a score for the mislabeling detection metric. Note that Mislabeling Detection requires calculating the self-influence of the entire train set (see Appendix E For LiSSA, calculating self-attributions for all training points is computationally infeasible. As GradCos is defined as the cosine of the angle between the test and training sample's feature vectors, the reib-tit for GradCos equal to 1 regardless of the sample.
35

---

Published in Transactions on Machine Learning Research (12/2025)
0.030
1.0
0.025
Metric score
0.020
0.6
Dua DA
0.015
DualLAA 10-5
0.010
DuaiDA
C' 10
0.2
0.005
DuaiDA 10-1
Identical class test (f)
Identical subclass (1)
Mislabeling detection (1)
Shortcut detection (t)
LDS
Inilummee Fonetions
-0.005
LISSA
-0.010
Arnoldi EK-FAC
0.5
Gradient Products
Metric score
0.6
Tracin
GradDot
GradCCss
0.4
TRAK
0.2
Representer Points
0.0
Coreset selection 10%
Coreset selection
Data Pruning 10% (1)
Data Pruning Ang (t)
Ang (L)
Figure F.2: Evaluation results on the CIFAR-10 datasct. LiSSA and GradCos do not have score for the mislabeling detection metric. Note that Mislabeling Detection requires calculating the self-influence of the entire train set (see Appendix FJ. For LiSSA, calculating self-attributions for all training points is computationally infeasible. As GradCos is defined as the cosine of the angle between the test and training sample's feature vectors, the self-attribution for GradCos equal 1 regardless of the sample.
36

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
0.8
Metric soores (+)
0.0
0.4
0.2
Weight Cosine Similarity
Logis Cosine Similarity
Prediction MCC
Weight Cosine Similarity
Logit Cosine Similarity
Prediction MCC
MNIST
CIFAR-10
1.0
Representer Points
DuaIDA
Metrio sootes (+)
10-3
0.0
10
0.4
C 10
Unsparsified
0.2
0.0
Weight Cosine Similarity
Logit Cosine Similarity
Prediction MCC
AwA2
Figure F.3: Surrogate similarity metrics for DualDA with sparsity hyperparameter C E {10 - 10 10 and Representer Points the MNIST, CIFAR-10, and AwA2 datasct.. Additionally, sparsified Representer Points artificially to the same sparsity levels as DualDA (in terms of the number of support vectors) with the corresponding sparsity parameter C' and present corresponding surrogate similarity metrics as well.
37

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
0.8
0.6
0.02
Mitric
0.01
0.2
0.00
Identical class test (1)
Identical subcloss Lest. (t)
Mislabeling detection (?)
Shortcut detection (1)
DS
DualDA C 10-
DuaIDA
C= 10-7
0.01
DualDA
C= 10-6
DuaiDA 10-a
0.02
8
DualDA
('= 10-1
0.12
DualDA 10'
Metric sc0te
0.10
0.08
4
0.06
0.04
0.00
Caresel selection 10% (4)
Cocesel selection
Date Pruning
Data Pruning Ang (1)
Ang (1)
10% (t)
Figure F.4: Evaluation results on the MNIST dataset for DualDA with different sparsity hyperparameters.
10-9 to 10 for DualDA. Figure F.4 shows the results for the MNIST dataset, Figure F.5 for CIFAR-10, and Figure F.G for the AwA2 dataset. DualDA performs best in the Identical Class Test, irrespective of the chosen hyperparameter. For the Identical Subclass Test, lower sparsity appcars preferable for better performance, boosting scores from 0.72 to 0.96 on the MNIST dataset and from low of 0.45 to maximum of 0.70 on AwA2. However, on CIFAR-10, scores remain largely unaffected by the choice of C. Mislabeling Detection is not affected by the sparsity level on MNIST and CIFAR-10, but on AwA2, higher sparsity allows the method to achieve near-perfect performance. Shortcut Detection benefits substantially from lower sparsity levels. For example, on MNTST, where DualDA with C E (10-5, 10-3 10-1) fails at the task as presented in Figure F.1 for C' 10-9, DualDA achieves perfect score. For the LDS metric, no clear pattern emerges: on MNIST, lower sparsity yields worse results, but CIFAR-10 and AwA2, it increases scores. Coreset Selection is improved by higher sparsity on all tasks, likely because the SVM selects more important training points support vectors. This selection effect is reduced when we decrease sparsity and therefore select more support vectors. The effect on Adversarial Data Pruning is also unclear: for MNIST and CIFAR-10, lower sparsity yields better results, but on AwA2, the opposite is true.
These findings reveal that optimal sparsity settings are both metric- and dataset-dependent. While some metrics show consistent patterns across datasets, others exhibit dataset-specifie behaviors that require hyperparameter selection for optimal performance.
Furthermore, we analyze how the sparsity hyperparameter affects the fit of the surrogate SVM. The top row of plots in Figure F.7 displays the training and test accuracies of DualDA surrogate models for CE E {10-6, 10-50 10-4 10 10 1002, 2 10-1, 100 10 1,00, 100} in comparison to the test accuracy of the original
model. The middle row shows the training time in black. The final row displays the number of support vectors per training class, where each coloured line represents one class. In general, the figure suggests that optimal choice for C always exists, satisfying the requirements for fast training to fit DualDA surrogate,
38

---

Published in Transactions on Machine Learning Research (12/2025)
0.014
1.6
0.012
0.008
0.6
Metric
0.006
0.004
DualDA C 10
0.2
0.002
DuaIDA C 10-7
0.000
DualDA C'= 10
Ldeenticall class lest. (1)
Adenntinaa subcloss Lesd. (t)
Mislabeling detesttion (?)
Slaoeticue detecction (1)
LDS
DualDA 10-a
DualDA
C 10-1
Metric 8001e
DualDA 10*
0.1
0.2
0.0
Corcent selection 10% (1)
Coreeet selection
Data Preming
Data Pruning
Any
10% (t)
Any (t)
Figure F.5: Evaluation results on the CIFAR-10 dataset for DualDA with different sparsity hyperparameters.
closely matching the original model, and resulting an adequate amount of support vectors to assure a faithful DualDA surrogate generating sparse and faithful data attributions. Our extended results further show that with optimal choice of hyperparameter C, DualDA can be used as highly effective method for solving several DA-related downstream tasks.
These analyses of the sparsity and accuracy of the surrogate SVM models, alongside corresponding evaluation results indicate that the choice of C' 10-3 yields a sparse surrogate while preserving strong fidelity to the original model and stable attribution performance networks. However, under alternative sctups, where the task, data modality and model architecture differ, the optimal hyperparameler might change. In such cases, independent hyperparameter optimization process may be necessary under these evaluation criteria. This process is considerable more tractable compared to competing approaches, due to the computational efficiency provided by DualDA.
the image classification task with convolutional neural
F.4 Accuracy Rates for Coreset Selection Metric
Accuracy rates for the Coreset. Selection Lask are presented Table F.T for the MNIST dataset, Table for the CIFAR-10 dataset, and in Table F. for the AwA2 dataset. We observe that increasing the sparsity hyperparameter enhances the performance of DualDA on this task, establishing it as the top-performing method across multiple sparsity levels on all three datasets.
F.5 Accuracy Rates for Adversarial Data Pruning Metric
Accuracy rates for the Adversarial Data Pruning task are presented in Table F.4 for the MNIST dataset, in Appendix F for the CIFAR-10 dataset, and Table F.6 for the AwA2 datasct. In contrast the Coresct Selection task, no consistent pattern emerges regarding the optimal sparsity level for DualDA: reduced sparsity produces superior results MNIST and CIFAR-10, whereas increased sparsity yields better performance
39

---

Published in Transactions on Machine Learning Research (12/2025)
0.025
1.0
0.021
0.615
0.6
Metric
0.010
DualDA C 10-a
0.005
0.2
DuaIDA
0.0
0.000
DualDA C= 10-3
Identical cless test (t)
Identical subclaes (t)
Mielaheling desection (1)
Shorteut desection
LDS (1)
DuaIDA (/= 10-x
2.0
DuaiDA
Metric score
2.5
1.5
DuaIDA C 10*
2.0
1.3
1.0
1.0
0.5
0.0
Coreset selection
Coreset selection Aug (-)
Date Pruning
Data Pruning
10% (4)
10% (t)
Ang (1)
Figure F.6: Evaluation results the AwA2 dataset for DualDA with different sparsity hyperparameters.
10%
20%
30%
40%
50%
60%
70%
80%
90%
DualDA C = 10-9
40.81%
56.88%
70.75%
77.49%
82.50%
87.21%
89.40%
38.289
98.66%
98.44%
DualDA C 10-7
43.20%
57.33%
69.64%
78.86%
83.89%
86.91%
89.73%
98.68%
98. 8005
DualDA C 10-5
83.23%
98.84%
98.78%
98.70%
98.79%
98.69%
98.58%
98.095
DualDA C 10
98.24%
98.41%
98.56%
98.74%
98.75%
98.68%
98.68%
98.89%
98.739 08.73%
DualDA G
98.31%
98.19%
98.38%
98.73%
98.93%
98.71%
98.76%
98.70%
98.83%
DualDA C 10
98.35%
98.41%
98.54%
98.56%
98.56%
98.64%%
98.75%
98.63%
98.5-1%
LiSSA
97.09%
98.18%
98.19%
98.71%
98.86%
98.70%
98.64%
98.89%
98.68%
Arnoldi
98.18%
98.84%
98.84%
08.65%
98.99%
98.80%
98.58%
98.76%
98.58%
EK-FAC
98.41%
98.61%
98.69%
98.78%
98.79%
98.64%
98.86%
98.74%
98.73%
Tracin
97.20%
98.53%
98.60%
98.9 98.90%
98.74%
98.89%
98.56%
98.79%
98.56%
GradDot
98.31%
98.75%
98.80%
98.65%
98.54%
98.709 70%
98.73%
98.59%
98.73%
GradCos
94.50%
96.41%
98.04%
88.40%
98.39%
98.39%
98.68%
98.81%
08.59%
TRAK
98.38%
98.86%
98.75%
98.83%
98.59%
98. 73%
98.84%
98.70%
08.75%
Representer Points
98.43%
98.76%
98.96%
98.86%
98.84%
98.59%
98.83%
98.81%
98.71%
Table F.1: Corcset Selection accuracy rates on MNIST dataset (model retrained on top x% of training data, sorted by attribution values of corresponding DA method). Higher accuracy rates indicate better DA performance on the task of the metric. Best values are highlighted in bold.
40

---

Published in Transactions on Machine Learning Research (12/2025)
MNIST
CIFAR-10
AwA2
100.0%
100.0%
100.0%
95.0%
95.0%
95.0%
90.0%
90.0%
90.09
Accuracy
Accuracy
Accuracy
85.0%
85.0%
85.0%
(originel)
Train acc. (surrogate)
Tast acc. (surrogate)
10
10
10"a
10a
10
10
10"
10a
10"
10
10"*
10-
400
1.4
350
300
1200
Train time 65)
1.3
250
Train time (s)
1000
Train time
1.2
200
800
400
1.0
200
(suroogate)
Number of support vectors per class
10000
Number of support vectors per class
2500
10000
2000
8000
Number of support
6000
6000
4000
4000
1000
2000
500
10-7
10-3
10-7
10
10
10-2
10-
107
10-5
30-3
10-1
10'
Figure F.7: Further analysis of the effect of the hyperparameter C' on the training of the SVM. First rows Accuracy rates on the train and test set are compared to the accuracy rate of the original model on the test set for all three datasets. The SVM performs well over a large range of hyperparametors, reaching accuracy rates on the test set in line with the performance of the original models. However, overtly strong regularisation substantially degrades performance on the CIFAR-10 and AwA2 datasets. Second row: The training time for the surrogate is displayed for all three datasets. Overall, the training time very short, but exceedingly strong regularisation leads to remarkable increase in training time. Third row: Each line plots the number of support vectors, i.e. samples that have non-zero contribution to the fit of the SVM, for one class of the train set. Stronger regularisation leads to a reduction in support vectors.
41

---

Published in Transactions on Machine Learning Research (12/2025)
10%
20%
30%
40%
50%
60%
70%
80%
90%
DualDA C = 10 22.71%
57 37.38%
68.79%
73.63%
77. 17.16%
78.33%
78.58%
79.75%
81.16%
DualDA C
24.10%
54.31%
66.96%
72.14%
75.20%
78.35%
81.25%
80.48%
81.73%
DualDA C = 10
28.60%
59.55%
71.43%
73.08%
79.04%
80.08%
79.93%
80.98%
81.28%
DualDA C
47.15%
69.35%
74.78%
77.98%
79.50%
77.16%
80.05%
80.48%
81.38%
DualDA C 10-1
55.60%
73.18%
77.31%
77.13%
79.86%
81.15%
81.24%
80.10%
80.44%
DualDA C 10
56.90%
70.74%
76.23%
78.46%
79.16%
80.38%
80.19%
78.24%
78.64%
LiSSA
68.71%
76.68%
79.71%
75.63%
80.40%
79.73%
81.13%
79.0%%
81.14%
Arnoldi
40.39%
50.61%
64.50%
75.69%
75.99%
77.53%
78.10%
81.16%
80.56%
EK-FAC
35. 35%79%
51.74%
62.26%
73.46%
74.91%
79.74%
80.14%
80.49%
50.98%
TracIn
28.18%
47.41%
72.23%
78.10%
78.76%
81.18%
80.10%
81.35%
80.9%%
GradDot
38.95%
51.83%
66.76%
75.23%
75.79%
76.63%
78.80%
79.80%
79.43%
GradCos
27.48%
57.81%
65.86%
71.35%
77.89%
77.45%
80.20%
80.28%
81.79%
TRAK
38.23%
51.97%
65.44%
72.78%
75.96%
77.61%
80.26%
81.75%
81.10%
Representer Points
45.89%
60.93%
71.59%
77.03%
79.51%
80.83%
80.26%
80.94%
79.71%
Table F.2: Coreset Selection accuracy rates on CIFAR-10 dataset (model retrained on top x% of training data, sorted by attribution values of corresponding DA method). Higher accuracy rates indicate better DA performance on the task of the metric. Best values are highlighted in bold.
10%
20%
30%
40%
50%
60%
70%
80%
90%
DualDA C = 10
57.01%
77 77.53%
80.87%
87 97.88%
89.44%
90.46%
92.48%
92.90%
93.38%
DualDA C
57.89%
74.54%
83.64%
88.08%
89.21%
90.21%
91.80%
92.88%
91.56%
DualDA C = 10
50.90%
78.39%
84.56%
89.18%
90.76%
90.81%
91.27%
91.40%
93.38%
DualDA C 10
85.18%
90.59%
90.59%
90.52%
91.89%
91.91%
93.43%
92.75%
93.38%
DualDA C 10
77.62%
91.47%
92.39%
91.75%
93.10%
91.87%
92.92%
92.41%
93.87%
DualDA C 10
87.09%
90.63%
90.35%
91.91%
90.72%
89.25%
92.26%
91.62%
92.68%
LiSSA
16.27%
33.93%
42.74%
44.79%
50.86%
61.67%
67.50%
79.00%
87 87.00%
Arnoldi
63.79%
89.99%
92.39%
93.05%
92.70%
92.46%
92.30%
93.01%
93.38%
EK-FAC
59.59%
84.04%
92.57%
91.91%
92.88%
92.28%
92.90%
93.38%
93.84%
TracIu
65.54%
86.08%
91.40%
92.26%
93.07%
93.32%
92.41%
93.53%
93.43%
GradDot
56.75%
83.71%
89.60%
90.98%
92.75%
93.40%
93.12%
92.79%
93.70 73%
GradCos
82.41%
84.57%
87.77%
89.42%
90.79%
92.74%
93.34%
92.57%
92.52%
TRAK
67.41%
89.80%
92.37%
91.67%
92.30%
92.83%
93.86%
93.53%
93.20%
Representer Points
72.29%
90.22%
91.86%
92.79%
92.79%
93.31%
92.98%
93.09%
93.93%
Table F.3: Coreset Selection accuracy rates on AwA2 datasel (model retrained on top x% of training data, sorted by attribution values of corresponding DA method). Higher accuracy rates indicate better DA performance on the task of the metric. Best values are highlighted in bold.
42

---

Published in Transactions on Machine Learning Research (12/2025)
10%
20%
30%
40%
50%
60%
70%
80%
90%
DualDA C
53.28%
83.51%
90.00%
97.23%
97.10%
98.28%
98.66%
98.69%
98.74%
DualDA C = 10
63.35%
81.85%
94.39%
96.39%
97.75%
98.60%
98.68%
98.79%
98.76%
DuaIDA C 10-5
97.15%
97.31%
97.59%
97.88%
97. 97.60%
97.20%
97.90%
97.96%
96 98.39%
DualDA C 10-3
97.74%
98.36%
98.41%%
98.49%
96 14.54%
98.39%
98.50%
98.49%
98.71%
DualDA C 10-1
97.94%
97.94%
98.38%
98.01%
98.71%%
98.48%
98.54%
98.73%
98.63%
DualDA C = 10'
97.74%
98.40%
98.06%
98.65%
98.53%
98.31%
98.70%
98.70%
98.63%
LISSA
98.18%
98.54%
98.56%
98.70%
98.66%
98.65%
98.50%
98.60%
98. 98.86%
Arnoldi
74.96%
80.03%
89.20%
92.95%
96.83%
97.41%
97.89%
98.24%
98. 48.46%
EK-FAC
74.76%
77.24%
86.80%
94.51%
96.98%
98.04%
98.29%
98.08%
98, 54%%
TracIn
78.38%
84.08%
86.48%
89.25 29%
90.96%
93.48%
96.35%
96.85%
97.88%
GradDot
72.66%
74.76%
86.89%
94.18%
97.08%
96.68%
97.78%
98.11%
98.65%
GradCos
93.93%
95.69%
98.28%
98.34%
98.46%
98.59%
98.76%
98.45%
98. 80%
TRAK
82.69%
81.34%
80.51%
94.04%
96.93%
97.60%
97.84%
98.08%
98.56%
Representer Points
76.83%
93.61%
95.80%
96.93%
96.96%
97.09%
97.74%
97.96%
98.59%
Table F.4: Adversarial Data Pruning accuracy rates MNIST dataset (model retrained bottom x% of training data, sorted by attribution values of corresponding DA method). Lower accuracy rates indicate better DA performance on the task of the metric. Best values are highlighted in bold.
10%
20%
30%
40%
50%
60%
70%
80%
90%
DualDA C = 10
30.84%
39.05%
55.66%
64.90%
67. 17.63%
68.43%
69.25%
71.13%
78.70%
DualDA C 10-7
29.40%
36.18%
54.19%
62.94%
66.98%
69.06%
68.78%
72.59%
78.39%
DualDA C = 10-6
64.96%
68.09%
69.68%
72.31%
71.99%
74.35%
76.35%
78.04%
79.39%
DualDA C 10
71.94%
70.41%
77.48%
77.46%
80.03%
81.56%
78.80%
81.75%
80.74%
DualDA C = 10
72.53%
78.48%
78.71%
80.18%
80.53%
80.61%
81.43%
81.05%
81.08%
DualDA C 10
70.69%
73.94%
78.68%
77.48%
79.71%
78.48%
81.34%
81.04%
81.89%
LiSSA
71.19%
75.44%
78.25%
77.94%
79.40%
80.11%
80.51%
81.20%
81.28%
Arnoldi
65.75%
70.56%
76.51%
77.74%
77.08%
78.78%
81.69%
80.44%
81.68%
EK-FAC
63.28%
69.44%
74.23%
73.88%
77.66%
78.50%
79.86%
80.95%
81.08%
TracIn
60.69%
62.93%
69.33%
71.99%
75.20%
77.09%
79.60%
79.74%
80.13%
GradDot
61.31%
66.90%
76.69%
77.71%
79.04%
78.58%
81.06%
80.43%
80.84%
GradCos
41.51%
57.80%
59.14%
63.89%
68.49%
69.90%
74.74%
78.98%
82.31%
TRAK
65.79%
72.56%
75.08%
79.10%
78.36%
78.20%
80.14%
82.71%
81.74%
Representer Points
67.93%
75.59%
77.36%
78.46%
77.08%
77.98%
80.55%
78.19%
82.14%
Table F.5:
Adversarial Data Pruning accuracy rates on CIFAR-10 dataset (model retrained on bottom x% of training data, sorted by attribution values of corresponding DA method). Lower accuracy rates indicate better DA performance on the task of the metric. Best values are highlighted in bold.
on AwA2. With optimized hyperparameters, DualDA achieves substantial performance gains: when models are retrained on only 10% of the data, accuracy is reduced to 53.28% compared to 72.66% achieved by the second-best method. On CIFAR-10, DualDA similarly attains the best performance the same data retention level, with accuracy of 29.40% versus 44.51% for the second-best approach. On AwA2, it reduces accuracy to 12.07%, performing competitively with the top method, LiSSA, which achieves 9.19%.
F.6 Details of Mislabeling Detection
In Figure F&
we present the curves produced by using different DA methods to detect mislabeled training samples as explained Appendix under the paragraph about the Mislabeling Detection metric.
F.7 Details of Sparsity Experiment
In Figure 5 we analyze the distribution of absolute attribution scores over the training set for various DA methods, exemplified on the Aw dataset. Since positive and negative attributions carry distinct semantic
43

---

Published in Transactions on Machine Learning Research (12/2025)
10%
20%
30%
40%
50%
60%
70%
80%
90%
DualDA C' 10-9
53.43%
70.60%
75.28%
77.95%
84.90%
67.66%
91.45%
93.29%
93. 93.69%
DualDA C' = 10-7
46.50%
71.40%
73.26%
79.48%
83.16%
89.01%
91.34%
92.33%
93.43%
DualDA C 10
65.59%
73.95%
80.63%
83.84%
86.35%
88.59%
89.10%
92.79%
92.52%
DualDA C' 10-3
13.90%
28.30%
42.00%
56.31%
67 47.46%
81.58%
91.67%
92.75%
93.38%
DualDA C 10
19.94%
42.61%
63.13%
81.69%
91.58%
91.14%
93.36%
91.36%
94.19%
DualDA C= 10'
12.07%
26.08%
36.98%
50.64%
61.83%
73.17%
83.95%
93.84%
92.81%
LiSSA
9.19%
20.38%
31.55%
40.22%
48.22%
62.53%
73.17%
80.59%
85.95%
Arnoldi
79.95%
89.64%
86.92%
91.82%
92.52%
92.20%
93.43%
93.76%
93.56%
EK-FAC
78.06%
88.28%
88.35%
91.56%
92.02%
93.20%
93.05%
93.64%
92.79%
TracIn
54.84%
69.59%
80.96%
82.59%
90.85%
90.19%
91. 91.36%
93.60%
93.42%
CradDot
84.48%
88.92%
88.08%
90.11%
91.45%
92.19%
92.83%
93.49%
92.98%
GradCos
73.64%
88.61%
88.33%
90.46%
91.60%
92.13%
93.10%
94.02%
93.27%
TRAK
85.99%
88. 79%
89. 18%
91.40%
90.15%
92.66%
93.32%
93.82%
93.23%
Representer Points
86.21%
90.08%
87.64%
90.04%
92.20%
92.06%
93.89%
93.67%
93.42%
Table F.6: Adversarial Data Pruning accuracy rates on AwA2 dataset (model retrained on bottom x% of training data, sorted by attribution values correspouding DA method). Lower accuracy rates indicate better DA performance on the task of the metric. Best values are highlighted in bold.
DualDA
Influence Functions
DualDA C = 10-5
DualDA C 10
DualDA
C' C== 10
LISSA
Arnoldi
EK-FAC
Gradient Produets
Representer Points
TracIn
GradDot
GradCos
TRAK
Detected Poisoned Samples
1.0
Ratio Detected Poisoned Samples
Ratio Detected Posooed Samples
0.8
8.8
0.8
0.6
8.6
0.6
0.4
0.1
0.2
0.2
502
0.2
0.4
0.6
0.8
1.0
0.2
0-4
DUG
0.8
1.0
0g0
0.2
0.6 0.8
10
Ratio of Controlled Images
Ratio of Contralled Images
Ratio Controlled Images
MNIST
CIFAR-10
AwA2
Figure F.8: Mislabeling detection curves for the three datasets. Dashed grey lincs indicate the best and worst possible curves for a DA method, depending on the amount of poisoned samples for each dataset. The score is the area under each curve which falls inside area inside the grey dashed curves, normalized be between and 1. Note that for the AwA2 dataset, the curves for TracIn and GradDot overlap, such that only the curve corresponding to GradDot is visible.
44

---

Published in Transactions on Machine Learning Research (12/2025)
interpretations, it is worthwhile to investigate the distribution of positive and negative attributions for different method. In this appendix, we provide a more detailed view into the altributions, taking into account. the distribution of negative and positive attributions. Initially, we examine the overall ratio of cumulative positive attribution versus negative attribution, displayed on the left side of the figure. Positive attributions show that a training point carries information corroborating the model prediction being explained, as such, they are expected to be concentrated on fewer datapoints from the corresponding class, compared to negative attributions which carry information about different kinds of evidence contradicting the prediction at hand. As expected, most explainers with the exception of GradDot and GradCos exhibit greater positive than negative cumulative attribution.
The proportion of negative to positive cumulative attribution varies significantly across methods. The LiSSA implementation of Influence Functions shows the lowest ratio at only 53.7%, while TRAK demonstrates the highest at 98.7%.
For DualDA, we observe that the sparsity hyperparameter C directly affects this ratio increasing values of C' (i.e., increasing sparsity) bring the positive cumulative relevance increasingly closer to the negative cunulative relevance.
On the right side of Figure $.9 we analyze the growth of the cumulative sum of both positive and negative attribution as a function of the fraction of most influential datapoints included, to assess the sparsity characteristics of attributions with distinct semantics. When comparing the development of the cumulative sum between positive and negative attribution, we observe that DualDA maintains consistent sparsity across both types. In contrast, Arnoldi and EK-FAC show increased sparsity for negative attribution, while Representer Points and GradDot demonstrate significantly reduced sparsity for negative attribution compared to positive attribution.
G Additional DualXDA Results
We present additional DualXDA explanations in Figures G.1 to G.6
H Unifying Concept-Level Explanations, Feature Attribution and Data Attribution
The formulation of DualDA can be combined with concept-level explanations to understand the impact that individual concepts - which may be present or absent in the training and test data - have on their combined DualDA attribution value. For a given concept k, assume through a concept-discovery method such linear probes we have obtained Concept Activation Vector (CAV) 6x {Kim et al., (0018). The larger the dot product between the hidden activations fiest and the vector Uk, the more likely it is that the concept is present in the test sample. We further normalize the CAV v attribution that is explained by concept. It can be quantified
éq Then, the amount of the DualDA lbella
DD (Irest 16) =
(
(C (C- diy.) (f. (T) pk) if y: c -Xic (J, (pess))) else
(H.1)
Furthermore, if given a set concepts XC of which the corresponding CAVs {ok E KC} form an orthogonal basis of the final hidden layer, then this concept decomposition fulfills a conservation property:
Te DD (Feest 1) = Σ Te DD (Irest / kk)
(H.2)
REA
An example for such basis is given by the final layer neurons, whose CAVs are the unit vectors of the Cartesian coordinate system. Furthermore, the concept-attribution values DD (Irest, i k) can be propagated back through the network, similarly to the per-support-vector attributions originating from DualDA via XDA. This will provide us for each concept with attribution maps for both training and test samples in which the attribution that is mediated through the concept is localized in the input space.
An illustrative application of this methodology to an ImageNet sample presented in Figure The attribution decomposition is performed with respect to concepts encoded by neurons in the penultimate
45

---

Published in Transactions on Machine Learning Research (12/2025)
1.0
1.0
Ratio of positive cumulatiie attribution
cumulative positive attributio (+)
0.8
DualDA
Ratio of
0.6
DualDA
C 10
DualDA 10
0.4
DuaIDA 10-1
0.2
10.2
Influeaace Functioon
L'SSA
0.0
Arnaldi
0.0
0.0
EK FAC
Ratio of cumulative negative attribution (-)
Rabio of negative cumulative latiee attributiion
Grufient Prodlucts
0.2
Tracin
Grad
GradCoo
0.4
0.4
TRAK
Representer
Points
0.8
0.8
1.0
0.5%
1%
1.5%
2.5%
3%
x% of training samples (sorted by attribution)
Figure F.9: Analysis of cumulative distribution of positive and negative attributions for various DA methods on the AwA2 dataset. The y-axis represents the cumulative sum of attributions over the dataset, ordered descendingly by attribution magnitude.
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
6
6
Test
XDA (ours)
Figure G.1: DualXDA explanations for the numeral "0" from the MNIST dataset. The XDA heatmaps indicate, that in the strongest proponent the part missing part of the digit to close the loop negatively relevant for the classification class "0".
46

---

Published in Transactions on Machine Learning Research (12/2025)
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
3
Blownn 10.0.
Riwerr 18%
Relewen: CPA
Retwwer
Riiwerr zert
Rimere seat
2
5
3
3
3
3
3
Test
XDA (ours)
Figure G.2: DualXDA explanations for the numeral "3" from the MNIST dataset. This example shows how DualDA be used to detect and understand mistakes in the training data: The most negatively relevant.
training sample is image of a "3" mistakenly labeled 55"
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
List
Libe cille
Likil
Label calle
Libe denaie
Libet de miie
Libel de mation
Libel damratin
Test
XDA (ours)
Figure G.3: DualXDA explanations for dalmatian from the Aw dataset. Note that the most negatively relevant training sample is picture labelled as "collie" that also contains dalmatian next to collie breed dog. This the interaction of those class-dependent features with the model can be observed in the XDA heatmap for the training sample: The part of the image that contains the dalmatian looks similarly to the test sample, but because the image is labeled "collie", this area negatively relevant. for the classification of the test sample a dalmatian. On the other hand, the part that contains the collie looks different from the test sample and is therefore not evidence contrary to the model decision. Therefore this area of the image is red. In general, XDA marks black and white spolted areas as influential, either for the decision (if this is the fur of dalmatians) or against the decision (when it is the fur of cows or the pattern of blanket).
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
Laael suaiea
Label
Likel
Labal
Test
XDA (ours)
Figure G.4: DualXDA explanations for rabbit from the AwA2 dataset. For the proponents, the entire body of the rabbit including the cars are relevant. However, the fur is also negatively relevant due to many opponents having a similar fur colour. Yet because few other animals have similarly long ears, the hare's ears are the main identifying feature in the overall XDA map.
47

---

Published in Transactions on Machine Learning Research (12/2025)
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
2.7L
Test
XDA (ours)
Figure G.5: DualXDA explanations for bee house from the ImageNet dataset. Opponents feature box-like structures that look similar apiaries. For the proponents, XDA considers the featured beekeeper hat positive evidence towards the classification only if similar hat is also shown in the training image.
Train
XDA (ours)
NEGATIVELY relevant training samples
LRP
POSITIVELY relevant training samples
2114 Relevseer 23,
1.66
2-f1
Test
XDA (ours)
Figure G.6: DualXDA explanations for scuba diver from the ImageNet dataset. Relevance is predominantly placed the oxygen mask if it is featured prominently in the testing image.
48

---

Published in Transactions on Machine Learning Research (12/2025)
network layer. Heatmaps visualized for the three neurons contributing the highest relevance scores. To determine their corresponding conceptual representations, we present the nine training samples that maximize the activation of each neuron: the first neuron responds predominantly to turtle imagery, the helmets, second cxhibits selectivity for hemispherical structures oriented upward such as shells, domes, and lamps, while the third demonstrales sensitivity to surfaces exhibiting scaled or dimpled textures as on golf balls, reptilian skin, and artichokes. Although neuron-level decomposition provides valuable interpretable insights, individual neurons frequently encode multiple concepts simultaneously, concepts incompletely, which complicates conceptual understanding of the neurons and results in more diffuse, uniformly distributed activation patterns in the corresponding heatmaps. Future research should investigate decomposition approaches utilizing linear probing techniques to identify more semantically coherent and interpretable representational subspaces. Alternatively, methods such as PURE (Dreyer et al. 2024) allow us to first disentangle neurons into multiple monosemantic units, which improves the clarity of neuron-level dccompositions.
encode
Preliminary Experiments on Large Language Models
To assess the applicability of our method to Large Language Models, we conduct preliminary experiments that will be expanded in future work. We fine-tune a pre-trained Llama-3.2-1B model (Dubey et al. (2024), released under the Llama 3.1 Community License, available on Huggingface on the AG News dataset for news topic classification (Zhang et al. (2015), available on Huggingfacd The dataset contains 120,000 training samples, each consisting of a news description and label indicating one of four categories. We assess both the resource consumption for attribution creation and the attribution quality through Shorteut. Detection experiment. All experiments have been run on a NVIDIA H200 Tensor Core GPU.
1.1 Resource Consumption
In this section, we report the computational resources required by DualDA for C' {10-5, (10-5,10-3, 110-$,10111 and compare them to EK-FAC (Grosse et al. 2023), the most computationally efficient variant of Influence Functions available to date. Table presents the results on the caching time, cache space, average explanation time and VRAM usage for both methods.
Generating attribution scores with EK-FAC for the Llama model demands substantial VRAM. The official implementation highlights the parameter per__eeiice lerr_ddiieeeraa__bobee VRAM consumption against minor changes in explanation times. To ensure a fair comparison, we report EK-FAC's results using its most efficient configurations for each metric. Specifically, the VRAM usage reported in Table [.1 corresponds to the absolute minimal setting (i.e. per per_device_ train batch size=1), while the explanation time corresponds to the most efficient setting we obtained with an available VRAM of 145 GB (i.c. per per_device_treee
device. train_ beein_batch___ size, which allows the user to trade-off
batch batchasize=3]] size=32).
The precomputation phase of DualDA involves two steps: (1) caching final hidden features and (2) training the SVM surrogate. Feature extraction dominates both caching time and cache size. Table I reports total caching time and cache size, while the training durations for the SVM surrogate models are as follows: 18 minutes 27 seconds for C = 10-1, 48 seconds for C = 10-3, and 32 seconds for C = 10-5, The storage required for the trained SVM parameters requires only 1.9 MB across all values C.
The results show that DualDA achieves improvements of 42-116x in caching time, 35x in cache size, at least 108, x00x in average explanation time and at least 3x in VRAM usage.
1.2 Shortcut Detection Experiment
We adapt the task of Shorteut Detection Koh & Liang 2017 Hammoudch & Lowd 2022 which identifies spurious correlations in training data that enable models to exploit dataset-specific biases, from the visual
2B nexs
httee://waggngaaaeco///aasssssss htsp:://gthhbb.ccc//mmomaa/////
49

---

Published in Transactions on Machine Learning Research (12/2025)
Test
Train
Samples maximizing neuron activation
Neuron 278 6.03% of attribution
+
+
Neuron 3869 3.13% of attribution
+
+
Neuron 26 3.02% of attribution
+
+
Figure H.l: Attribution decomposition for a turtle image and its hiighet--autributted training sample, with neuron-specifi contributions ranked in descending order of attribution.
50

---

Published in Transactions on Machine Learning Research (12/2025)
DualDA
EK-FAC
Factor of Improvement
Caching Time (s)
620
1695
72460
42x 116x
Cache Size (GB)
0.99
34.60
35x
Explanation Time per test sample (s)
0.04
> 4345
108, 000
VRAM Usage (GB)
8.7
29.5
> 3x
Table I1:: Computational resource comparison between DualDA and EK-FAC for Llama-3.2-1 trained on the AG News dataset. For DualDA, we report the range of caching times, including feature extraction and SVM training, over three tested scttings (C E {10-5, 10-3 10,,0110011, 1)) Results for EK-FAC are reported using its most efficient per _evvice train batch, size for each metrie. DualDA demonst rates substantially lower caching time, cache size, explanation time, and VRAM usage across all settings.
per_device_
Precision-Recha Curve for Shortcut Detection, samples ranked by DualDA (C 10-1)
800
DuaiDA
Handom
Optimel
700
600
Found shortcul samples
400
300
200
500
1000
1500
2000
2500
3000
3500 4000
Queried samples
Figure I.1: Precision-Recal Curve for the task of identifying shortcutted training samples from attribution values.
domain our language setting. We construct a training set 4,000 randomly selected samples and introduce shortcut into 20% of them by randomly reassigning the label to class cls and inserting the corresponding keyword '<ols>> at a random position in the sample text. We apply the same manipulation to create a shortcutted test set consisting of 1,000 test samples. Since models exploiting these shorteuts should rely heavily on the shortcutted training samples, we hypothesize that Data Attribution methods assign high attribution to these samples when explaining predictions on shortcutted test instances. We quantify the overall effect of each training sample on the entire lest set by summing its attributions across all shortcutted test instances.
Consistent with our vision domain experiments, lower sparsity improves Shorteut Detection performance. At sparsity levels of C = 10-5 or higher, DualDA fails to meaningfully differentiale between shortcutted and non-shortcutted samples. However, at C = 10-7 or lower, DualDA successfully identifies shortcuts: it assigns the highest overall attribution to nearly half of all shortcutted samples, and all shortcutted samples are retrieved within the top 50% ranked training samples. The corresponding Precision-Recalt curve (Figure [.1] achieves an AUPRC score of 0.88.
51


Reviews:
[{"review_id": "SLYq6EhfkI", "reviewer": "Reviewer_9qGX", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes DualDA a new approach to data attribution for deep learning models. DualDA trains a SVM on top of the features extracted by the penultimate layer of a classifiers, and then uses the dual variables of the SVM problem to build an attribution value for each training sample given a test sample and a class. Moreover, the paper introduces XDA, a method based on LRP to explain the data attribution scores. In particular, XDA can visualize as heatmaps in the pixel space which features of a pair of training and test images contribute most to the attribution scores. In the experimental evaluation on three image classification datasets, the proposed DualDA achieves competitive or better results on average across various metrics than existing methods, while being more efficient. Finally, the paper provides qualitative examples showing that XDA can provide interpretable heatmaps for the attribution scores.\n\nStrengths\n- The proposed DualDA leverages the structure of SVMs to obtain simple and effective data attribution scores for deep networks, with only a single hyperparameter.\n\n- The experimental evaluation, spanning multiple datasets and architectures, supports the effectiveness of DualDA, which achieves good results across different metrics. Moreover, it is significantly more efficient than the baselines.\n\n- While XDA is tested only via qualitative examples, these suggest that it can capture meaningful explanations for the attribution scores.\n\nWeaknesses\n- The best performing method varies across metrics, so the most suitable approach may depend on the task and metric of interest. Similarly, among the variants of DualDA with different values of $C$, the best performing one varies, as well as their comparison to the baselines (Fig. 3 and Fig. 5).\n\n- A few examples are presented for XDA, but it's hard to get an idea of its effectiveness in the average case, or what are its failure cases, without more extensive (quantitative) evaluations.\n\n- The text suggests that the proposed method is applicable to state-of-the-art scenarios, including LLMs, but the experiments are limited to relatively small datasets and architectures, and only to image classification tasks (and it's not clear how it can be extended to others).\n\n- The presentation is a bit lengthy at times (for example, implementation details can be deferred to the appendix).", "claims_and_evidence_yn": "Yes", "claims_explanation": "The experimental evaluation supports the effectiveness of DualDA, which achieves good results across different metrics. Moreover, it is significantly more efficient than the baselines. XDA examples are promising, but a more extensive evaluation would be more convincing. Finally, the text seems to suggest that the proposed method is applicable to state-of-the-art scenarios, including LLMs, but the experiments are limited to relatively small datasets and architectures, and only to image classification tasks.", "audience_yn": "Yes", "audience_explanation": "Data attribution and explainability are relevant topics, and an efficient method can be of interest especially if extended to more complex tasks and  models.", "requested_changes": "- Since all other baselines are presented with a single set of hyperparameters, I think it would be good if a default value for $C$ in the proposed method was suggested, and convenient for potential users. Studying the effect of hyperparameter is clearly interesting, but having a standard configuration might make the main comparison to the baselines easier.\n\n- I think the claims about the scalability to SOTA setups should be clarified, since no experiments or technical insights are provided in this direction, while there are works tackling them, e.g. [A].\n\n- (minor) I think the presentation could be made more concise.\n\n[A] https://arxiv.org/abs/2504.16430", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "E6B6naMUZd", "reviewer": "Reviewer_ev3s", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors present a framework for scalable and interpretable data attribution in large AI models, called DualXDA. The objective is to identify which training samples influence model predictions both globally, i.e., their effect on the model parameters obtained during training, and locally, i.e., the influence of a training instance on the prediction of a test sample. The literature already provides methods for this task, but existing approaches are computationally expensive and yield dense, hard-to-interpret results.\n\nDualXDA addresses these issues with two components: DualDA, which uses multiclass kernel SVMs as surrogates to provide sparse, faithful, and highly efficient attributions that can be both global and local; and XDA, which integrates feature attribution methods such as Layer-wise Relevance Propagation to explain why certain training samples matter by highlighting relevant input features in the prediction of a test sample.\n\nThe paper presents extensive experiments on three well-known computer vision datasets, state-of-the-art computer vision models, and seven established performance metrics for explainability. Moreover, it includes four guided examples showing how the feature attribution component of the framework can highlight its significance. The experimental results demonstrate that DualDA matches or surpasses state-of-the-art attribution quality while achieving massive speedups, and that XDA provides richer, human-understandable, feature-level interpretability.\n\n**Strengths**\n- Comprehensive review of previous data attribution approaches. The paper also provides detailed explanations of the connections between DualDA and existing methods such as Influence Functions and Representer Points, with formal proofs.\n- Clear motivation and design of the framework.\n- All theoretical results are formally proven in the Appendix.\n- Rich and curated experimental evaluation that strongly motivates the effectiveness of the framework. Moreover, the additional explanations about the metrics used makes a lot more clear the experimental comparison with the baselines.\n- Clear advantages of the proposed framework: the surrogate SVMs are faithful, the achieved speedup is notable, and the framework enables sparse data and feature attributions that are more interpretable.\n\n**Weaknesses**\n- I had a hard time to understand the purpose and the meaning of the first experiment presented in Section 4.4: why is it relevant to evaluate the `overall ration of cumulative positive attribution versus negative attribution` as a measure of attribution sparsity?\n- One of the main goals of the paper is to enable explainable data attributions for large AI models. However, the evaluation focuses on standard computer vision models such as ResNet-50, VGG-16, and ResNet-18, which are also considered in prior work, showing the speed-up of the proposed method against previously proposed attribution methods. However, it would have been interesting to consider even larger models, such as large language models. This limitation is acknowledged by the authors as future work.", "claims_and_evidence_yn": "Yes", "claims_explanation": "In my opinion, the evidence provided by the paper to support its claims is accurate, clear, and convincing, with the exception of one point regarding the evaluation of the sparsity of the attributions, which the authors should clarify. Apart from this, all theoretical results are formally proven, and the experimental evaluation is sufficiently comprehensive to demonstrate the effectiveness of the proposed framework, its efficiency compared to state-of-the-art methods, and the usefulness of the approach. Moreover, an intriguing aspect of the paper is the explanation of the connections between the proposed framework, the Representer Points method, and the Influence Functions method.", "audience_yn": "Yes", "audience_explanation": "The findings of this paper are certainly of interest to the TMLR community. Explainability is a hot topic in artificial intelligence, and providing explainable data attribution is essential for better understanding the training and prediction dynamics of models that are not easily interpretable, such as neural networks. Since this work proposes a method that delivers faithful and explainable data and feature attributions with improved efficiency, it has the potential to become the new state-of-the-art approach for this task and to generate significant impact.", "requested_changes": "**Changes critical for acceptance**\n- I suggest that the authors provide an explanation about the relevance of the `overall ration of cumulative positive attribution versus negative attribution` to evaluate the sparsity of the attributions in Section 4.4.\n\n**Changes that would strengthen the work**\n- Even though the authors indicate this point as a limitation to be addressed in future work, it could be interesting to provide results about the scalability of the framework when applied to models that are even larger than the ones considered before in other papers, like large-language models.", "broader_impact_concerns": "No broader impact concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "IiGjwN0JCI", "reviewer": "Reviewer_tgqp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper talked about the data attribution issue in the LLM context. Specifically, they proposed a novel approach for efficient and effective DA, leveraging Support Vector Machine theory to provide fast and naturally sparse data attributions for AI predictions.", "claims_and_evidence_yn": "No", "claims_explanation": "Frankly, I am NOT successfully completing reading this paper, particularly in section 2 and 3. I found many parts are seemingly missing from my viewpoint. The followings are the reading notes \n\n**s.2 related work** \n\n- definition 2.1 Is not clear from my viewpoint. For example, the \n\n> Global DA methods assign each pair of training sample and class c a real-valued attribution which indicates the relevance or influence of the training datapoint on the fit of the model Φ(·;θ)c \n\nThis sentence does not rigorously describe what is the global DA exactly, for example, what is each pair of training samples? Φ is a model rather than score? What is the relevance or influence of the training datapoint? \n\n> Local DA methods produce a function \\tau for each class c which accepts as input a test sample and the index for a training point. The value \\tau indicates the relevance or influence of the training datapoint zi on the model output.\n\nSo what is \\tau, as input to a test sample and the index for a training point ? \n\n> advocates for a different standpoint, aiming to estimate the changes induced by fine-tuning an already trained model with an altered training set instead. \n\nI would suggest using formal math equations rather than wording descriptions. \n\n- In addition, the gradient based, surrogate based methods require better math equations. \n\n**s 3.1 Using Support Vector Machines for Sparse and Efficient Data Attribution**\n\nIn equation (1), the formulation is standard SVM and equation (5) is about test data. I would suggest a clear consistency check in s.2.\n\nI think the logic transition from the (1) to (7) becomes unclear for me. \n\n**about paper organizations and figures**\n\nIndeed, I think this may be the main reason why I cannot continue reading the paper. The beginning of the paper is filled with large amounts of text. Figure 1 seems intuitive, but it's actually counterproductive because I have to keep checking the figure and the equation back and forth. In particular, certain subtle arrows and shapes make me repeatedly guess the actual meaning of the figure. Combined with the unclear equations, I am finally lost.", "audience_yn": "Yes", "audience_explanation": "I would think the contributions are still valuable for the community\n-  A fast, efficient calculation in data attribution is important in an explainable LLM. \n-  This paper leverages several co-exiting methods and empirical results seem promising.", "requested_changes": "I would strongly suggest a major revision for s 1-3 to make clear\n\n- the math definition\n- a simplified figure without subtle details to quickly understand the overall information. The current figure is counterproductive.\n- the consistent and concise math notation and equation.", "broader_impact_concerns": "NA", "additional_comments": "If the authors make substional major revision by reducing the text and complex illustrative figures. I could recheck this paper.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)