An Efficient Framework for Crediting Data Contributors of Diffusion Models
Authors: MingYu Lu
Chris Lin
Chanwoo Kim
Su-In Lee
Venue: ICLR 2025 Poster (2025)
Keywords: data attribution
diffusion models
Shapley values
Primary area: interpretability and explainable AI
TL;DR: By model pruning and fine-tuning, we efficiently estimate Shapley values and attribute global properties of diffusion models to data contributors

Abstract: As diffusion models are deployed in real-world settings and their performance driven by training data, appraising the contribution of data contributors is crucial to creating incentives for sharing quality data and to implementing policies for data compensation. Depending on the use case, model performance corresponds to various global properties of the distribution learned by a diffusion model (e.g., overall aesthetic quality). Hence, here we address the problem of attributing global properties of diffusion models to data contributors. The Shapley value provides a principled approach to valuation by uniquely satisfying game-theoretic axioms of fairness. However, estimating Shapley values for diffusion models is computationally impractical because it requires retraining and rerunning inference on many subsets of data contributors. We introduce a method to efficiently retrain and rerun inference for Shapley value estimation, by leveraging model pruning and fine-tuning. We evaluate the utility of our method with three use cases: (i) image quality for a DDPM trained on a CIFAR dataset, (ii) demographic diversity for an LDM trained on CelebA-HQ, and (iii) aesthetic quality for a Stable Diffusion model LoRA-finetuned on Post-Impressionist artworks. Our results empirically demonstrate that our framework can identify important data contributors across global properties, outperforming existing attribution methods for diffusion models.

Published as a conference paper at ICLR 2025
AN EFFICIENT FRAMEWORK FOR CREDITING DATA CONTRIBUTORS OF DIFFUSION MODELS
Chris Lin,' Mingyu Lu," Chanwoo Kim, Su-In Lee
Paul G. Allen School of Computer Science & Engineering
University of Washington (clin25, mi mingyulu, chanwkim, suin ee @cs. wwwssswwshnngttne.....
ABSTRACT
As diffusion models are deployed in real-world settings, and their performance is driven by training data, appraising the contribution of data contributors is crucial to creating incentives for sharing quality data and to implementing policies for data compensation. Depending on the use case, model performance corresponds to various global properties of the distribution learned by diffusion model (e.g., overall aesthetic quality). Hence, here we address the problem of attributing global properties of diffusion models to data contributors. The Shapley value provides a principled approach to valuation by uniquely satisfying game- game-theoretic axioms of fairness. However, estimating Shapley values for diffusion models is computa- tionally impractical because it requires retraining on many training data subsets corresponding to different contributors and rerunning inference. We introduce method to efficiently retrain and rerun inference for Shapley value estimation, by leveraging model pruning and tine-tuning. We evaluate the utility of our method with three use cases: (i) image quality for a DDPM trained on CIFAR dataset, (ii) demographic diversity for an LDM trained on CelebA- -HH and (iii) aesthetic qual- ity for Stable Diffusion model
on Post- cest-mppressionstt artworks. Our results empirically demonstrate that our framework can identify important data contributors across models' global properties, outperforming existing attribution methods for diffusion models"
INTRODUCTION
Diffusion models have demonstrated impressive performance on image generation (Ho et al. 2020 Song et al. (2020b), with models such as Dall-E 2 (Ramesh et al. (202)) and Stable Diffusion (Rom- bach al. 2022) showing versatile utilities and enabling downstream applications via customization (Hu et al. 2021 Ruiz et al. 2023). A key driver for the performance of diffusion models is the data used for training and fine- tunetunnng. The data for commercial diffusion models are often scraped from the internet (Schuhmann et al. 2022), raising concerns about credit attribution for those who created the data the first place (e.g., artists and their artworks) (Jiang et al. 2023). Additionally, as labor required to label and curate data to further improve model performance, demands for data labeling platforms have risen, with workers often underpaid (Widder et al. 2023 Wong 2023). To create incentives for sharing quality data and implement policies for data compensation, pressing question arises: How do we fairly credit data contributors of diffusion models?
Data attribution, which aims to trace machine learning model behaviors back to training data, has the potential to address the above question. Indeed, in the context of supervised learning. several works have proposed methods for valuating the contribution of individual datum model performance such as accuracy (Ghorbani & Zou 2019 Kwon & Zou 2021 Wang & Jia, 2023). Some recent work has developed data attribution methods for diffusion models (Dai & Gifford [2033] Georgiev
eet al., 2023 WWan et al.) 2023al Zheng et al.) 2003).. However, two gaps remain when applying these
recent methods credit data contributors of diffusion models. First, these methods focus on local
properties related to the generation of a given image. For example, Exanple.Zhengg et al. 2023) showcase their
Eqqul contribution.
Our code is availahle at httpps//gtthbb.ooo/////////////0

---

Published as a conference paper at ICLR 2025
method D-TRAK to study the changes in pixel values of particular generated images. In contrast, the performance of diffusion models is evaluated based on global properties of the learned generative distributions. For example, the demographic diversity of generated images can be considered a global property for evaluation (Luccioni et al. 2023). Second, existing attribution methods for diffusion models consider the contribution of each training datum instead of each data contributor, but contributor can provide multiple data points. One reasonable approach is to aggregate the valuations of data provided by a contributor the contributor's total contribution. However, previous work has shown this approach to incur errors between the aggregated and actual contribution (Koh et al. 2019). Here, we aim to address these two gaps by attributing global properties of diffusion models to data contributors.
Attribution methods based on cooperative game theory are particularly desirable because of their axiomatic justification. Specifically, the Shapley value provides principled approach to fairly distribute credit among contributors, since is the unique notion that satisfies game-theoretic axioms for equitable valuation (Ghorbani & Zou| 2019 Shapley 1953). Briefly, the Shapley value assesses each contributor based on the average gain incurred by adding the contributor's data to different contributor combinations. To estimate the Shapley values for data contributors in our setting, need to (i) retrain diffusion models on data subsets corresponding to different combinations of contributors; and (ii) measure global properties of the retrained models by rerunning inference. However, training a diffusion model can take hundreds of GPU days, and inference can also be expensive (c.g., approximately GPU days to generate 50,000 images for image quality metries) (Dhariwal & Nichol (2021). Therefore, estimating Shapley values with vanilla retraining and inference computationally impractical. Here, we propose efficiently approximate retraining and inference on retrained models through model pruning and fine-tuning providing framework that enables Shapley value estimation (Figure
Training Stage
Crediting Stage
Data contributor
Fine-ure data conmibuto sutuets
Run inference
Extimate conti Shap
GUG
y F
JoJO
H&
GUC
Diffusion
model
Figure 1: Schematic overview of our proposed framework, where 0* denotes a trained diffusion model for which we aim to credit data contributors, and Ø- denotes the pruned model that approximates 0*. After fine-tuning the pruned model on data corresponding to various subsets of contributors, denoted as gn, and rerumning inference; global model properties (F) are measured to estimate the Shapley value for each data contributor.
Related work. Data attribution methods for diffusion models have been developed by recent work. Some methods require models be trained with specialized paradigms. For example, to assess the importance for training sample, Dai & Gifford (2023) first train an ensemble of diffusion models on data splits, followed by ablating models trained on splits containing the specific sample. Wang et al. (2023a) evaluate data attribution for text-to-image models by customizing a pretrained model toward an exemplar style. In contrast, other methods can be applied to already trained models in a post hoc manner. For example, the TRAK framework has been adapted to find important training data for intermediate latents along generative process (Georgiev et al. 2023), while Zheng et al. (2023) introduce empirical approaches that improve the performance of TRAK for diffusion models. All these methods attribute local model properties to each individual datum, whereas work focuses on attributing global model properties to each data contributor.
Contributions. (1) To our knowledge, are the first to investigate how to attribute global properties of diffusion models to data contributors. (2) We propose a framework that efficiently approximates retraining and rerunning inference for diffusion models, enabling the estimation of Shapley values for
2

---

Published as a conference paper at ICLR 2025
data contributors. (3) We empirically demonstrate that our framework outperforms existing attribution methods across three datasets, model architectures, and global properties.
PRELIMINARIES
This section provides overview for diffusion models, attributing global model properties to data contributors, and existing attribution methods for diffusion models.
2.1 DIFFUSION MODELS
Our research primarily focuses on discrete-time diffusion models, specifically denoising diffusion probabilistic models (DDPMs) (Ho et al. 2020) and latent diffusion models (LDMs) (Rombach et al. 2022). Generally, diffusion models are trained to approximate a data distribution g(xo). To perform learning, a training sample X, ~ ((xo) is sequentially corrupted by additive noise (Ho al. 2020). This procedure is called the forward process and is defined by ((xx|xx-1) == V(x;; v1 -. 1.,3TT,, F), for i I,... T, where {3, ) corresponds to variance schedule. Notably, the forward process allows sampling of X, at any time step from Xo. with the closed form g(xx) xo) = N(x:: vo, (1 - at) where a == 1 - Br and d; == III'_] 0ss. Then, a diffusion model learns to denoise X1.7, following the reverse process defined by (ixx--((x)) == N(x (X, t), of I) where E I2 is the model parameters, and o, corresponds to some sampling schedule (Karras et al., 2022). Instead of modeling the conditional means μa, it is standard to predict the added noises with a neural network eg using the reparameterization trick. The training objective corresponds variational bound and is formulated as
Siimpee (x; 0) E,,, [||- eo ca(vā.x + v11 ãt 01
Once a diffusion model has heen trained, new image can be generated by sampling an initial noise XT NN, I) and iteratively applying cg each step T, ..... 1 for denoising.
2.2 ATTRIBUTING GLOBAL MODEL PROPERTIES TO DATA CONTRIBUTORS
To gain a comprehensive understanding of machine learning models, it is often beneficial to study global model properties (Covert al., 2020), which assess the model's performance across samples, rather than focusing on individual outputs. For example, in the setting of supervised learning, global model property can be test accuracy. In the context of generative models, it can be quality metric for generated samples. For example, in image generation, the global model property of interest can be the Inception Score (Goodfellow et al., 2014) or FID (Heusel et al. 2017). More formally, a global model property for generative models is defined by any application speciffc function F θ - R. which maps a generative model to a scalar value, thereby quantifying the overall distribution learned by the model. Building on this, we introduce the problem of attributing global model properties to data contributors. The goal of contributor attribution is to identify important groups of training data for model's global properties, where each group of data is provided by contributor. More formally, we have the following definition.
Definition 1. (Contributor attribution) Consider n contributors of training sumples DD (2,,Co... ..., C,), with the ith contributor providing a set of data points denoted as C, and a global model property function F. A contributor attribution method is function T TFF, {C,)==) that assigns scores to all contributors to indicate each contributor's 's importance to the global model property F.
2.3 EXISTING ATTRIBUTION METHODS FOR DIFFUSION MODELS
In the context of diffusion models, recent work has focused attributing local model properties to each datum by adapting the TRAK framework (Park et al., 2023), such as D-TRAK (Zheng et al.
2023) and Journey-TRAY (Georgiev et al. 2023). ). Formally, let X' denote the input space and O the parameter space of diffusion model. Suppose there are N training data points (x(1) xx x (N)) }. and & is the generated image of interest for local attribution. Given a loss function L X x e + R, local model property f X θ RR and models (0 trained S data subset
TRAK for
PI practice, we consider the computationally efficient retraining- free setting by Zheng al. 8 and 0 0" is the original model trained the entire dataset.
2023 That is,
3

---

Published as a conference paper at ICLR 2025
diffusion models is defined as:
2 4 2. +1) 7x()), and
5 s=1
Φ = Øs (x( (x(x
(3)
where (x) = P. VoL((: (.). 7 (x(x) = P. Fef(x; 82), Pa is a random projection matrix, and XI serves for numerical stability and regularization. More details on Joumey-TRAL (Georgiev et al. 2023) and D-TRAK Zheng et al. 2023) are in Appendix
3 CREDITING DATA CONTRIBUTORS OF DIFFUSION MODELS USING THE SHAPLEY VALUE
Here, we motivate the use of the Shapley value for crediting data contributors of diffusion models. While estimating Shapley values for diffusion models require computationally expensive retraining and inference, we propose to address the computational challenge with model pruning and fine-tuning.
3.1 THE SHAPLEY VALUE FOR CONTRIBUTOR ATTRIBUTION
The Shapley value was developed in cooperative game theory to fairly attribute credit in coalition games (Shapley (953). In the context of contributor attribution, for training data with n contributors, the Shapley value attributed to the ith contributor is defined as:
3: =
Σ SCD
n ()!)
(F(8sue)) - F(0g))
where Usuc and 03 are models trained on the subsets S U C; and S, respectively. The Shapley value appraises each contributor's 's's contribution based on the weighted marginal contribution, F(fgue,) F(8g). It satisfies axioms desirable for equitable attribution, including linearity, dummy player, symmetry, and efficiency (see Ghorbani Zou (2019) for a succinct summary). There are arguments that the efficiency axiom, >i=1 B; = F(0*) F(D)), where 0* denotes the model trained on the entire dataset, and Og denotes where the primary goal is to remove detrimental data points (Wang & Jia 2023). In that setting, ranking is more important than exact attribution values. However, in the context diffusion models, particularly in applications involving monetary credit allocation related to model performance, having the total attribution matches the model utility can directly quantify reward for each contributor.
model without training, may not always essential in use cases
Despite its advantages, evaluating the exact Shapley value is challenging it involves retraining 2n (all possible subsets) models and computing their corresponding model properties. Fortunately, many hampling-based estimators for Shapley values have been developed (Lundberg & Lee 2017]
Ghorbani & Zou 2019 Ştrumbelj & Kononenko (2010). In particular, we adopt KernelSHAP in the feature attribution literature for contributor attribution, by solving a weighted least squares problem
with sampling (Lundberg & Lee 2017 Covert Lee 2021):
M
8 = min
F(Oo) + -13,8-F(05)) F( s.t. 1 = ====((0)) - -(Oe)
(5)
So.......
M
where Sy is sampled following the distribution y(S) (n for 1 15.3 < n, and 1s
an indicator vector representing the presence of contributors in S. By solving the least squares problem with the constraint, closed-form solution can be derived (Covert & Lee 2021 The obtained parameters 31,... B,, are the contributor attribution scores.
3.2 SPEED UP RETRAINING AND INFERENCE WITH SPARSIFIED FINE- TUN-TUNING
Although using the Shapley value for contributor attribution is well motivated. and existing sampling approach can be used, the main challenge is in computing F(0%,) ). First, diffusion model needs to
4

---

with Published as a conference paper at ICLR 2025
be trained again from random initialization the subset Sj to obtain 83 Then inference needs to be rerun on the retrained model 0's to measure F(Os,) ). For diffusion models, both retraining and rerunning inference can take multiple GPU days (Dhariwal & Nichol 2021). making the estimator in Equation (5) computationally impractical.
To address this computational challenge, we propose to speed up retraining and rerunning inference with sparsified fine-tuning (Figure D Sparsified fine-tuning enhances the efficiency of both retraining and inference, by reducing the number of model parameters through pruning (Tia et al. (023). Specifically, 0* is pruned and initially fine-tuned on the full dataset D to obtain a performant pruned model 0 that approximates 0" To furthermore improve the retraining efficiency, 0* is fine-tuned on each subset S, for & steps obtain 0 S instead of retraining the pruned model on Sj from random initialization. The diffusion loss objective used to train the original model 0" is used for all fine-tuning. Overall, sparsified fine-tuning aims to efficiently achieve
F(OSs (sparsified fine- se F(g (retraining full model from scratch).
(6)
In other words, under the same computational budget and compared retaining the full model from scratch, sparsified fine-tuning can increase the number of sampled subsets S, for estimating Shapley values, thus making feasible framework for crediting data contributors of diffusion models.
In practice, computing F(0) requires sampling initial noises x ~ (O, for = L,..., N. denoising those noises into generated samples with the denoising network co. and computing some quantity over the generated samples (e.g.. FID). Hence, to formally analyze the approximation
in Equation (6), we consider the expected erroi (x)))) [IF(0S,,)) F(03, I[] For notational omitted, and expectations are taken with respect to {x7 ) (N for the rest of the paper unless otherwise noted. We then have the following proposition to formalize the intuition behind Equation (6).
ease, the dependency of F on {x] (x
Proposition 1. Suppose an objective function &. TRd R on the data provided by a given subset of data contributors S is convex and differentiable, and that its gradient is some constant L > 0, i.e.
(V(((1)) ve(02)]2 = L110, 02112
for any 01,02 E Rd. Let PaSK denote a pruned model after l fine-tuniig steps on the given subset S with learning rate a S 1/L, 0' the optimal pruned model trained on S with the same sparsity structure as 82,1x and f''s the optimal full-parameter model trained on S. Furthermore, assume that (() -(0ss)) < for some constant Then the expected error E. ((F(((sss)) F(05))) S B as le 00.
The proof is in Appendix A The assumption that l is convex and differentiable with Lipschitz- continuous gradient is a standard setting for the theoretical analysis of approximating retraining with fine-tuning, instantiated
quadratic objective"] in Golatkar al. (0200) and Georgiev et al. 2024). The assumption that E| [( F(03)) I] B corresponds the lottery ticket hypothesis that sparser neural network can approximate the performance of a dense, full-parameter network (Frankle Carbin 2018), which has been shown hold for diffusion models (Fang et al.) (023). The main takeaway from Proposition iis that more steps of sparsified fine-tuning should lead to bounded approximation error between F(0t, S, and F(0',) in Equation (6). We further relate Proposition to the expected error in Shapley values with the following proposition.
Proposition 2. Let if E Rn be the Shapley values for the data contributors evaluated with (F(8g,))sez) and 3 E 12. the Shapley values evaluated with {F(0g)]se22. Suppose the assumptions on in Proposition holld for all subsets of data contributors. Furthermore, assume that
E ilmaxxe2" (F(0)) - F(B3))l < C. Then E[]]3, 3 S 2/TCC as k + xc.
'We consider the setting where the same (x( (r) > are inputs for both f0 and 603, to represent the use
case generating samples with the same random seed across multiple inference runs.
A quadratic objective has the form {(O) (0 B) A(B B), with lipschitz-continuous gradient such
that (e(()) Vé(02)lla < 20 (A) ||0 0zl where of max ()) the maximum singular value of A.
5

---

Published as a conference paper at ICLR 2025
The proof is in Appendix A The assumption that convexity, differentiabilityy and the Lipschitz continuity of gradient hold for all subsets is implied by the standard setting of quadratic los as in Golatkar et al. (2020) and Georgiev et al. (2024). The key takeaway is that the Shapley value error is bounded with more steps of sparsified fine-tuning. Finally, we note that both Proposition and Proposition 2 are asymptotic results, and that Proposition 2 is based exact Shapley values. We leave theoretical results incorporating finite-step bounds and Shapley value estimation for future work. Nevertheless, we verify the insights from our theoretical results under empirical settings for diffusion models in Appendix
4 EXPERIMENTS
In this section, we compare our approach with existing attribution methods across three settings. We employ two metries for evaluation: linear datamodeling score (LDS) and counterfactual analysis. Our results demonstrate that our method outperforms existing attribution methods for diffusion models.
4.1 DATASETS AND CONTRIBUTORS
Datasets used for our experiments include CIFAR-22 220, subset of CIFAR-100 with 20 contributors, one per class, as described in Krizhevsky et al. (2009); CelebA-HO, with 50 celebrity identities as contributors; and ArtBench (Post-lmpressioiism) with 258 artists as contributors. More details about datasets and contributors can be found in Appendix B
4.2 EXPERIMENT SETTINGS
Model training. For CIFAR-20, we follow the original implementation of the unconditional DDPM (Ho al., 2020) where the model has 35.7M parameters. For CelebA-HQ, follow the implemen- tation of LDM (Rombach et al. 2022) with 274M parameters and pre-trained VQ-VAE (Razavi 2019). For ArtBench (Post- Impressionis)) a Stable Diffusion model (Rombach al. 2022) is fine-tuned using LoRA (Hu et al. 2021) with rank = 256, corresponding to 5.1M LoRA parameters. "a Post-Impressionist painting" for each image.
al.
The prompt for the Stable Dilfusion model is for more details about model training and inference.
Please refer to Appendix C
Sparsified fine-tuning. Magnitude- based pruning (Han al., 2013) is used to remove model weights according to their magnitudes, resulting in sparse diffusion models with reduced parameters: from 35.7M to 19.8M for CIFAR 20, from 274M 70.9M for CelebA-HQ. and from M.1M to 2.6M for ArtBench (Post-Impressionism) To estimate Shapley values, the pruned models are fine-tuned on data subsets corresponding to different contributor combinations, with 1,000 steps, 500 steps, and 200 steps for CIFAR-20, CelebA-HQ, and ArtBench (Post- (opresmppressiiiimm)) respectively.
Global model properties. For CIFAR-20, we aim to study the contribution of each labeler. As each labeler is tasked to filter out noisy samples in each class (Krizhevsky et al. 2209), high-quality labeling should ensure that generated images are well separated with respect to image classes. Therefore, for the global model property, choose the Inception Score (Salimans et al. 2016):
IS = exp (Ex (LL(() (xLx())))ll(()))) (P)()))
where p(y) represents the marginal class distribution over the generated data and p(y represents the conditional class distribution given generated image
For CelebA-HO. our goal is to investigate how individual celebrities contribute to the demographic diversity of the model. Following Luccioni et al. (2023), we measure diversity using entropy:
K
H Σ Pk log(p));
where pa is the proportion of generated samples in the kth demographic cluster. We generate images using the full model and extract their embeddings with BLIP-VQA (Li et al. 2022) to create 20
'A Lipschitz constant applicable across all subsets be defined as max (As), where As defined by the subset S. For example, in linear regression, Ag X3 Xs, where X5 denotes the feature matrix contributed by the data contrihutors in

---

Published as a conference paper at ICLR 2025
reference clusters. Subsequently, assign these reference clusters images generated from various retrained models to calculate the entropy.
For ArtBench (Post-Impp Impressionism) consider the case where images generated and the most aesthetically pleasing ones are kept. We simulate this use case by computing aesthetic scores 16 for 50 generated images and considering the 90th percentile the global model property.
More details about these global model properties are in Appendix B
4.3 BASELINE METHODS
We mainly categorize baseline attribution methods into three categories (i) similarity methods (e.g., pixel similarity between the generated and training images); (ii) leave-one-out (LOO) and its approximate variants such as the influence functions (Koh Liang 2017) and TRAK-base methods (Park al. 2023); and (iii) application-specife metrics such as the aesthetic score of each training image for ArtBench (Post-Impressionism) Because retraining diffusion models is computationally expensive, gradients are computed using the original model 0* for TRAK-based methods, as done in Zheng et al. (2023). Local attribution scores are averaged across the images generated by the original model for computing global model properties. To aggregate datum-level attributions, the sum of attributions corresponding to each contributor is taken for the influence functions and TRAK- based methods, since this the principled aggregation (Koh al. 2019). For similarity-ban -based and rpplication-specifie methods, datum-level attribution scores are aggregated for contributor attribution by taking either the average or maximum. More details about baseline methods can be found in Appendix B
4.4 EVALUATING THE PERFORMANCE OF CONTRIBUTOR ATTRIBUTION
Linear datamodeling score (LDS). From Definition 1 for a given subset of contributors S C D, where D = {C1,... J. we can define an additive datamodel of global properties based on a given set of attribution scores 7 for the contributors:
g(S,T)= = Σ Ti-
&CEE
Therefore, the evaluation for an attribution method 7 can be constructed as follows:
Definition 2. (Linear datamodeling score) Contributor attribution performance is measured using the linear datamodeling score (LDS) (Ilaa al.) (022). which evaluates an attribution method by comparing predicted model properties based the additive datamodel against actual retrained model properties. Let Si,..,,p be randomly sampled subsets of D, each of size a n for some E (0, 1). The LDS for a contributor altribution score T E RR" is defined as
LDS (1F (g(S,, r)) where So Uniformm DD: D: a n), (10)
where p is the Spearman rank correlation SSeaaman 1961, and 0%, denotes a model retrained from scratch with the contributor subset Sp-
We evaluate the LDS using 100 held-out subsets So, each sampled from the datamodel distribution with a = 0.25, 0.5, 0.75 for each dataset. We report the LDS means and 95% confidence intervals across three independent sets of [S.
Counterfactual evaluation. Following Zheng et al. (2023). we also apply counterfactual evaluation (Hooker al. 2019). We assess the relative change in model property, AF T(0%) (00)) by
F(0*)
comparing the models trained before and after excluding (or retaining only) the top most influential contributors identified by each attribution method. Counterfactual evaluation requires retraining models on different subsets for each method, so only baseline methods with the best LDS in each category are chosen for computational feasibility.
'tttps: wtwpbcoom/oooocctcccooooocottttt I, 10N-01 I wtwpesthettticpreeee ctor
TracIn (Pruhi al. 2020) not considered since intermediate checkpoints may be available in practice.
7

---

Published as a conference paper at ICLR 2025
4.5 EXPERIMENT RESULTS
SHAPLEY ATTRIBUTION OUTPERFORMS BASELINE METHODS IN CONTRIBUTOR ATTRIBUTION
In Table I we present the LDS results for baseline methods and our approach. Interestingly, similarity- based methods such as raw pixel similarity, CLIP similarity, and gradient similarity occasionally outperform TRAK-based methods. We observe that TRAK-based methods sometimes yield poor or even negative correlations. Among TRAK-based approaches, attribution using noisy latents during generation (i.e., Journey-TRAA TRAK) can result in a negative LDS for model properties such as the Inception Score and aesthetic score. Our findings indicate that simply aggregating individual attributions derived from diffusion loss its alternative functions is insufficient for accurately determining contributor attribution of global properties. Such an approach can perform worse than simple heuristics based model properties (e.g., the average aesthetic score).
Table I: LDS (%) results with a = 0.5. Means and 95% confidence intervals across three random
initializations are reported.
ArtBench (Post-mmpressoonsm))
Method
CIFAR-20
CelebA-HQ
Pixel similarity (average)
-11.81 4.56
-8.91 + 0.93
11.24 + 0.63 14.61 + 2.72
Pixel similarity (max)
-31.80 + 2.90
21.70 + 2.05
Embedding dist. (average) Embedding dist, (max)
13.83 + 1.12
7.32
CLIP similarity (average)
5.79 + 3.67
-32.23 1:3:
-6.96 + 4.08
CLIP similarity (max)
11.31 + 0.37
-0.93 + 3.83
-1.75 + 4.07
Gradient similarity (average)
5.79 + 3.67
-18.22
0.25 = 1.18
Gradient similarity (max)
-0.89 + 3.17
-12.90 1.60
10.48 + 3.11
Aesthetic score (average)
24.85 + 2.30
Aesthetic score (max)
21.36 + 3.70
Relative IF
5.23 5.50
-1.07 1 0.68
-5002 + 1.77
Renormalized IF
6.799
10.17 0.57
-11.41 + 0.93
TRAK
7.94
0.75
-8.18 + 1.30
Joumey-TRAK
42.92
-2.88 4.02
-11.41 4.22
D-TRAK
10,90 1.21
-77.23
11.30 + 3.47
Leave-one-out (LOO)
30.66 1 6.11
-1.22 6.34
3.74 1. 8.00
Sparsified-FT Shapley (Ours)
61.48 + 2.27
26.34 + 3.42
61.44 = 2.04
In contrast, our approach, sparsified fine-tuning (sparsified-FT) Shapley, computes contributor attribution using the Shapley value, achieving the highest LDS results of 61.48%, 26.34% and 61.46% for CIFAR-20, CelebA-HQ, and ArtBench Post-Iiprrssionism)) respectively. While leave-one-out (LOO) achieves 30.66% LDS on CIFAR-20, its performance declines as the number of contributors increase, e.g., CelebA-HQ and ArtBench (opessmppressiiiimmm)) This shows that attribution based on the marginal contribution of Shapley subsets with respect to F provides the most accurate importance the bascline methods,
score. Despite achieving the best results for CelebA-Ht -QQ compared observe that the LDS performance of sparsified-FT Shapley, 26.32%, is relatively low compared to those of CIFAR-20 and ArtBench. We also perform evaluation with additional datamodel subset sizes (a = 0.25, 0.75), and our approach consistently outperforms others (Appendix
ENHANCING RETRAINING AND INFERENCE EFFICIENCY THROUGH SPARSIFIED FINE-TUNING
computing F(B3,) is the primary computational bottleneck. Our sparsified fine-tuning approach significantly reduces the runtime required to obtain 0' compared to retraining and fine-tuning without sparsification. On average, retraining and inference with sparsified fine-tuning for Shapley subset take 18.3 minutes for CIFAR-20, 22.9 minutes for CelebA-HQ, and 10.5 minutes for ArtBench (Post- mass--pprressonismm) making it 5.3, 10.4, and 18.6 times faster than retraining, respectively (Table 2). By enabling faster computation and obtaining more models retrained on different subsets, sparsified FT yields the best LDS results under the same computational budgets (Figure 2). This demonstrates that sparsified-FT Shapley is both more computationally
As described in Section 3.2
8

---

Published as a conference paper at ICLR 2025
feasible and accurate with limited computational resources. To the best of our knowledge, the first to overcome the computational bottleneck and enable contributor attribution using Shapley values for diffusion models.
CIFAR-20
CalebA-HQ
ArtBanch [Post-lmpnssion)
Rttiinn
03
LDS
LDS
00
Computational Budget
Computational Budget.
Computational Budget
Figure 2: Comparison of LDS (%) with a = 0.5 among Shapley values estimated with sparsified fine-tuning (FT), fine-tuning (FT), and retraining under the same computational budgets (1 unit runtime to retrain and run inference on a full model). Specific runtimes are shown in Table
IMPACT OF CONTRIBUTORS ON MODEL PROPERTY
Here we present the results of our counterfactual evaluation, analyzing changes in model behavior after either removing the top contributors or including only the top contributors identified by each method. In CIFAR-20, our approach shows a change -23.23% compared -14.95% for CLIP similarity and -17.00% for LOO, as shown in Figure 3 (top). For CelebA-HB the changes are -7.83% for our method, -6.64% for pixel similarity, and 0.21% % for renormalized IF. For ArtBench fosts-mmpressioissmm) the changes are 0.58%, -0.05%, -1 -1.27%, and -1.66% for D-TRAK, maximum pixel similarity, average aesthetic score, and sparsified-FT Shapley, respectively.
Conversely, when retaining the top 60% of contributors, model properties improve. In the CIFAR-20 dataset, our method leads to a 16.98% positive change in model properties, compared -9.45% for CLIP similarity and 9.51 % for LOO. For CelebA-HQ, the model behavior change was 20.0% for our method, 8.89% for pixel similarity, and 6.02% for renormalized IF (bottom of Figure 3). These
findings, along with the results in Section 4.5 demonstrate that our approach effectively identifies the top contributors and provides accurate attribution.
WHO ARE THE TOP CONTRIBUTORS?
We analyze the top contributors identified by sparsified-FT Shapley in CIFAR-20, including con- tributors to classes such as motorcycles, buses, and lawnmowers (Figure J. High-quality labeling by these contributors should ensure that the training images contain clear and meaningful objects. which should result in low entropy (i.e., high confidence) of a classifier (i.e., Inception v3) (Barratt & Sharma 2018). We therefore compute the entropy of each image and find that the top 20% classes have a lower average entropy of 5.18, compared to 6.03 for the remaining classes (Figure T6]. For CelebA-HQ. the most important celebrities ranked by sparsified-F" Shapley encompass diverse range of demographics and tend to belong to non-majority demographic clusters (Figure Exclud- ing images corresponding to these celebrities can have a negative impact on the diversity score. For ArtBench, we observe that images from the top contributors more vivid and exhibit more vibrant colors, as shown in Figure
DISCUSSION
In this work, we introduce the problem of attributing global model properties to data contributors of diffusion models. We develop an efficient framework to estimate the Shapley values for data contributors by leveraging model pruning and fine-tuning, speeding up retraining and inference runtime while ensuring accurate attribution. Our framework can have range of implications, such as creating incentives for data contributors, rewarding data labelers in a way that satisfies game-theoretic faimess,, assessing label quality, and improving model performance and fairness. Empirical results for multiple datasets and global model properties show that our framework outperforms existing attribution approaches based on the diffusion loss, such TRAK (Park et al., 2023) and its variant

---

Published as a conference paper at ICLR 2025
CHAA-20
Celeb/-110
Arttlench (Post
Roletvve Change 1%0 Iniccpption Scare
Ralarive Change 196 Swnlhellee Scote
Rotertve Change in Entropy
-10
-20
-22
Spaply
F Sicarity
150
Recertalive
Part maletty
Sinddy
Save Pown FT)
trcuptton State
10
lurty miler
Resonie Chinges in 290
20.0
Meaettu Charge semergg... (h4)
PAcsheell Score
anauly spasilis
lak i
Recotvee
10.0
7.5
5.0
20 Porcentage Top
90
60
90
B0
60
90
80
70
60
Parcentage Contributios:
Percontago Contributors
Figure 3: Relative percentage changes in global properties, comparing the original fully trained models to models retrained after removing the top 40% contributors (top) and including only the top contributors (bottom), as identified by various attribution methods.
"a Post-Impressionist painting"
exclude contributors
all contributors
contributors included
Figure 4: Top contributors and their corresponding training images for each dataset (top). Pairs of generated images above and below the 90th percentile of aesthetic score from Stable Diffusion models LoRA-finetuned under three conditions: excluding the data from the top 40% of artists, using the data from all the artists, and including only the data from the top 60% of artists (bottom).
D-TRAK (Zheng al. 2023). crediting data contributors. There are several promising directions for future research. Beyond fine-tuning, recent unlearning methods designed for diffusion models (Gandikota et al., 2023 Heng Soh. (023) could be explored in combination with various pruning strategies (Ding et al. 2019 Fang et al. 2024 Liu et al.) 2021). Beyond diffusion models, our proposed framework can also be extended to models that are expensive to retrain, such as large language models (LLMs), whenever appropriate pruning and unlearning approaches are available. Finally, our approach and existing attribution methods for diffusion models assume access to training data and model parameters, which are not necessarily available for deployed models. For text- to-image models, membership inference and in-context learning may be useful addressing this challenge (Hu & Pang. 2023
Wang et al. 2023b).
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
As diffusion models become increasingly deployed in real-world applications, there is a notable lack of clear guidelines for crediting data contributors. This work focuses the fair attribution of data contributors for diffusion models, which is crucial step towards creating incentives for sharing quality data and implementing policies for equitable compensation. By efficiently estimating Shapley values, our approach can promote faimess in recognizing the importance of data contributors. Importantly, our approach motivated by positive attribution, such as incentivizing quality labeling and improving the demographic diversity of generated images. However, we also recognize that our approach could potentially be used to remove data from certain data contributors. We view this as an opportunity to ensure higher data quality and transparency in model development, particularly when addressing harmful or irrelevant contributions. Ensuring data contributors are fairly compensated and informed is critical to avoid exploitation. Our cases, such demographic diversity for CelebA- HQ, highlight the need for transparency in data usage and advocate for diverse and representative datasets. At the same time, acknowledge that considerations related data privacy, consent, and the potential amplification of biases, especially in socially sensitive applications, may not be easily quantified model properties.
ACKNOWLEDGEMENTS
We thank members the Lee lab for providing feedback on this project. This work was funded by the National Science Foundation [DBI-1759487]; and the National Institutes of Health [RO1 AG061 132, RO1 EB035934, and RFI AG088824).
REFERENCES
Shane Barratt and Rishi Sharma. note on the inception score. arXiv preprint /rXiv:/801.0197.
2018.
Elnaz Barshan, Mare-Etienne Etienne Brunet, and Gintare Karolina Dziugaite. Relatif: Identifying explanatory training samples via relative influence. In International Conference on Artificial Intelligence and Statistics, pp. 1899-1909. PMLR, 2020.
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka. Input similarity from the neural network perspective. Advances in Neural Information Processing Systems, 32, 2019.
Ian Covert and Su-In Lee. Improving KerneISHAP: Practical Shapley value estimation using linear regression. In International Conference Artificial Intelligence and Statistics, pp. 3457-3465. PMLR, 2021.
Ian Covert, Scott Lundberg, and Su-In Lee. Feature removal is unifying principle for model explanation methods. arXiv preprint arXiv:2011.03625 2020.
Zheng Dai and David K Gifford. Training data attribution for diffusion models. arXiv preprint arXiv:2306. 02t1:2306600174, 02174, 2023.
Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wine wise quantization. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net, 2022. URL http:////////eeeeee/ee/m net/
Forum??ic-shppppXXoo33
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-877 8794, 2021.
Xiaohan Ding, Guiguang Ding, Yuchen Guo, and Jungong Han. Centripetal sgd for pruning very deep convolutional networks with complicated structure. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4943 2019.
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. arXiv preprint 2.t::2305.10924 2023.
11

---

Published as a conference paper at ICLR 2025
Gongfan Fang, Xinyin Ma, and Xinchao Wang. Structural pruning for diffusion models. Advances in neural information processing systems, 36, 2024.
Jonathan Frankle and Michael Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks, arXiv preprini 1oX3vv/80000333: 2018.
Rohit Gandikota, Joanna Materzynska, Jaden Sootto-Kaufman and David Bau. Erasing concepts from diffusion models. arXiv preprint arXiv:2303.07335, 2023.
Kristian Georgiev, Joshua Vendrow, Hadi Salman, Sung Min Park, and Aleksander Madry. The journey, not the destination: How data guides diffusion models, arXiv preprint arXiv:2312.06205, 2023.
Kristian Georgiev, Roy Rinberg, Sung Min Park, Shivam Garg, Andrew Ilyas, Aleksander Madry, and Seth Neel. suribute-to-deled Machine unlearning via datamodel matching. arXiv preprint arXiv:2410.2322 232322, 2024.
Amirata Ghorbani and James Zou, Data shapley: Equitable valuation of data for machine learning. In International conference machine learning, pp. 2242-22511 PMLR, 2019.
Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting in deep networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 9304-9312, 2020.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. Explaining and harnessing adversarial examples. arXiv preprint 141.:61112.67., 2014.
Laura Graves, Vineel Nagisetty, and Vijay Ganesh. Amnesiac machine learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 11516-11524, 2021.
Zayd Hammoudeh and Daniel Lowd. Identifying a training-set attack's target using renormalized influence estimation. In Proceedings of the 2022 ACM SIGSAC Conference Computer and Communications Security, pp. 1367-1381, 2022.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in neural information processing systems, 28, 2015.
Alvin Heng and Harold Soh. Selective amnesia: A continual learning approach to forgetting in deep generative models. arXiv preprint arXiv:2305.10122, 2023.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851. 2020.
Sara Hooker, Dumitru Erhan, Pieter-Jan Kindermans, and Been Kim. A benchmark for interpretability methods in deep neural networks. Advances in neural information processing systems, 32, 2019.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint
(rXiv:2106009681 2021.
Hailong Hu and Jun Pang. Membership inference of diffusion models. arXiv:2301.09956 2023.
arXiv preprint
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Data- models: Predicting predictions from training data. arXiv preprint arXiv:2202.00622. 2022.
Zachary Izzo, Mary Anne Smart, Kamalika Chaudhuri, and James Zou. Approximate data deletion from machine learning models. In International Conference on Artificial Intelligence and Statistics, pp. 2016-2016. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2025
Ward Jh Jr. Hierarchical grouping to optimize objective function. J. Amsr. Statis. Assoc., 58:
1963.
Jinghan Jia, Jiancheng Liu, Parikshit Ram, Yuguang Yao, Gaowen Liu, Yang Liu, Pranay Sharma, and Sijia Liu. Model sparsity can simplify machine unlearning. In Annual Conference on Neural Information Processing Systems, 2023.
Harry H Jiang, Lauren Brown, Jessica Cheng, Mehtab Khan, Abhishek Gupta, Deja Workman, Alex Hanna, Johnathan Flowers, and Timnit Gebru. Ai art and its impact on artists. In Proceedings of the 2023 AAAI/ACM Conference on AI, Ethics, and Society, pp. 363-374, 2023.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation, In 6th International Conference on Learning Representa- tions, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL http//ppeeevviieee /ttttpp:///penzeviee... wwt. id= 99
Tero Karras, Miika Aittala, Timo Aila, and Samuli Laine. Elucidating the design space of diffusion- based generative models. Advances Neural Information Processing Systems, 35:26565-26577, 2022.
Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pp. 1885-1894. PMLR, 2017.
Pang Wei W Koh, Kai-Siang Ang, Hubert Teo, and Percy Liang. On the accuracy of influence functions for measuring group effects. Advances in neural information processing systems, 32, 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yongchan Kwon and James Zou. Beta shapley: a unified and noise-reduced data valuation framework for machine learning. arXiv preprint arXiv:21 I0.4499, 2021.
14049,
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre- training for unified vision-language understanding and generation. In International conference on machine learning, pp. 12888-12900. PMLR, 2022.
Peiyuan Liao, Xiuyu Li, Xihui Liu, and Kurt Keutzer. The artbench dataset: Benchmarking generative models with artworks. arXiv preprint arXiv: 11404, 2022.
Liyang Liu, Shilong Zhang, Zhanghui Kuang, Aojun Zhou, Jing-Hao Xue, Xinjiang Wang, Yimin Chen, Wenming Yang, Qingmin Liao, and Wayne Zhang. Group fisher pruning for practical network compression. In International Conference on Machine Learning. pp. 7021-7032. PMLR. 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1001 2017.
Alexandra Sasha Luccioni, Christopher Akiki, Margaret Mitchell, and Yacine Jemnite, Stable bias: Analyzing societal representations in diffusion models. arXiv preprint 1rXiv:2303.11408, 2023.
Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. In Advances in Neural Information Processing Systems, pp. 4765-4774, 2017.
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior scale. arXiv preprint 11X 2023.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-8766 PMLR, 2021.
13

---

Published as a conference paper at ICLR 2025
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3 2022.
Ali Razavi, Aaron Van den Oord, and Oriol Vinyals. Generating diverse high-fidelity images with vq-vae-2. Advances in neural information processing systems, 32, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yacl Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Proceed- ings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 22500-22510, 2023.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans, Advances in neural information processing systems, 29, 2016.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 35:25278 22222222222944222-22222095992429 2022.
Lloyd Shapley. A value for n-person games. Contributions the Theory of Games, 2(28):307-317, 1953.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020a.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456, 2020b.
Charles Spearman. The proof and measurement of association between two things. 1961.
Erik Štrumbelj and Igor Kononenko. An efficient explanation of individual classifications using game theory. Journal of Machine Learning Research, 11:1-18, 2010.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 6388-6421. PMLR, 2023.
Sheng- Yu Wang. Alexci A Efros, Jun- Yan Zhu, and Richard Zhang. Evaluating data attribution for text-to-image models. arXiv preprint arXiv:2306.09 09345, 2023a.
Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, al. In-context learning unlocked for diffusion models. Advances in Neural Information Processing Systems, 36:8542-855 -5622 2023b.
David Gray Widder, Sarah West, and Meredith Whittaker. Open (for business): Big tech, concentrated power, and the political economy of open ai. Concentrated Power, and Politicul Economy of Open Al (August 17, 2023), 2023.
Matteo Wong. America already has ai underclass. The Atlantic, 2023.
Xiaosen Zheng, Tianyu Pang, Chao Du, Jing Jiang, and Min Lin. Intriguing properties of data attribution on diffusion models. arXiv preprint arXiv:2311.00500 2023.
14

---

Published as a conference paper at ICLR 2025
APPENDIX
A PROOFS
Before proving Proposition I we first provide the following lemma.
Lemma 1. Suppose objective function é : IRd + IR is convex and differentiable, and that its gradient is -ipsching-contimuous with some constant L > 0, i.e.
[(V(())) ve(02)]l2 I L| - 82ll2
for any 01,0g 02 E Rd. Then rumning gradient descent for k steps with learning rate S L satisfies
|10(0) 0
((0(K)) {(0))
2ka
where g(h) is the model parameter after the kth step, 8" is the optimum, and ( is the initial parameter.
Proof. We provide the proof for this well-known result for completeness. The assumption that Vi(0)
is L-Lipschitz continuous implies that (=((0) 3 LI, so can perform quadratic expansion of &
around E(O) to obtain the following inequality:
1 [1 - -112, ((O()) < (()) + VE(O) (0 - 0) +
(11)
2
for any 0, g(0 E Rd. With the gradient descent update g(0) = 0li-1) - a V((((ii))) and setting
0 = 0(i Equation [1 we obtain
((0()) - (((()- + Ve((((-1)) (vv((((-))) + マ L
)
(12)
1(0(i-11) all v + 2 =
(13)
((0li-1))
-
1 -1 a a a/[Ve(0(i-1))) 13
1
(14)
Using a < 1/L, we have
1 1
1 - 2
aL 2 1
L 2 T.
(15)
2
Hence
e(B(0) S ((8(1-1)) a ((((((11)) ) 12
(16)
which implies that the objective decreasing until the optimal value is reached, because |[VE(O(E-))]] positive unless vi((ti-1)) O.
Now since is convex, we have
((0(t-1)) + ()( (0+ 2(i < <((0),
(17)
and
((0(i-1)) < {(0') + VE(Oli-1)) T(0T)) - 0*)
(18)
after rearrangement, Plugging this inequality into Equation (66), we have
((010) < E(0') + ve(o(i-1))T (ole-1) 0²) 2 (
(19)
which implies that ((0())) - E(O')
(20)
S (( - 2 (e(0(i-1)) 11
(21)
(2ave(((--1)) ( - 0r) a² ([(0)(-1))]3 1111
(22)
2a
(2 (g(6-1) (4) a² V | 110(1-1) -0 + ||0(i-1)
0 112
2a
(23)
1
(10(1-2)-
(
-0-112 10(--1) (((0lo-1)) - 0'133)
(24)
20
15

---

Published as a conference paper at ICLR 2025
Plugging in the gradient descent update rule into this inequality, get
E(B()) - e(00) - 2a
(10(6-1) - -8 - - 00113)
(25)
Summing over & steps, we have
Σ ((0() {(0)) v
1
(1186-11 - -0 1/2 110 0 0-
(26)
2a
1=1
1
(
(10(0)
-0111 ||0( - 00
(27)
2a
1 |0(0)
00 1
(28)
20
where the telescoping sum results in the equality. Finally, the term in the summation on the LHS is decreasing because of Equation (T6), hence
|10(0) 0- 11
((0()) - -(0) S
Σ - -((0))
(29)
2ka
We then proceed to prove Proposition
Proof of Proposition
L By the triangle inequality, we have
BI = -
F(03))
(30)
S E E| - -F +
(31)
where the second inequality comes from the assumption that ([[F())) F(0)))) B.
Note that 0 and 03 are in Rd, with zeros for the pruned weights. Hence the assumptions on hold for f 03. Applying Lemma we have
0 0 11
(
((0g) S
(32)
2ka
where we recall that Ō+ denotes the pruned model before fine-tuning. Taking k: + co, we obtain ((OG,) - ((gg) = O, and it follows that 0 S.s = 0 because é convex. Therefore, as - co, we have
BI F(03)11 < EIIF(OS F(0 + B = =1|F(0G) - =(03))] +
(33)
(34)
B
(35)
Before proving Proposition 2 we provide the following lemma.
Lemma 2. The exact Shapley values 3 E F. for the data contributors evaluated with (F(0s)) se2D
can be represented as B At, where A E Rux2"" is a matrix with rows indexed by the data contributors and columns indexed by all possible contributor subsets, and v E 122" is a vector indexed by all possible contributor subsets. Specifically,
JCES
Alr.S] =
n 11-1
(36)
otherwise
[S]
and = =(()s)
16

---

Published as a conference paper at ICLR 2025
Proof. For each data contributor i, we have
B Σ
S[i . .(S]
(37)
SE20
Σ S:C.ES
1
|1| -1
n - 1
F(0s)- Σ
1 (n-
|54|
F(0s))
(38)
8':C
S' " |S 11==1)
Σ
1
F(0suci)- - Σ
1
11--1)
F(0s))
(39)
|S
"
S'CPic:
1 Σ
(117))
F)0s))),
(40)
S'EDIC,
which is the Shapley value for contributor i. The third equality follows from the change of variable S' = S C such that |S*| |S| - 1.
Finally, we prove Proposition as follows.
Proof of Proposition 2 Based on Lemma i = Aöt and 8*+ = Ae=, where i[[ss] F(6 and
0" [S] = F(83) for all S e 2P. Hence we have
E Bll - [12] All - A0
(41)
E
(Ali] (og =)))²
(42)
E
|
11ef - u* 11.
(43)
0" 1l
Σ |
(44)
where the inequality follows from Holder's inequality. For each i I,... n, we have
|| Σ ||i
(45)
SE2
Σ
n - 1
Σ
n - 1
(46)
22
2
==1
S:C;ES,|S|==
0 S:C, $S
Σ 1
n n-
Σ
n n-
n - 1)
(47)
(1-1)
==
n n 2,
(48)
n
n
where the third equality follows from counting the number of subsets. Hence, we have
E|| [1] x 2 E
vn 10
(49)
=2nn E
max SE20
- -(09))
(50)
$ 2/n.
E
mmx SE20
[F(0%))) F(03)11
+ E max [F(0z) F(03)1)
(51)
SE20
= 22/n E.
max Se2n
[F(03) - -F(()))
(52)
17

---

Published as a conference paper at ICLR 2025
as k + oc, where the last equality follows the same steps in Proposition assumption that E|max [F(03)
|
Finally, with the
F _ C, we have
E|l - 3 [2] - 2/TCC
(53)
B DATASETS AND MODEL PROPERTIES
CIFAR-20 (32 32). The full CIFAR- 100 datase d contains 50,000 training samples across 100 classes, with each class labeled by a labeler (Krizhevsky et al. 2009). These labelers were com- pensated to ensure the removal of initially mislabeled images. For computational feasibility in the LDS evaluations, follow similar approach in Zheng al. (2023) create smaller subset, C-FAR-20. This subset includes 10,000 training samples randomly selected from four superclasses in CIFAR-100: "large carnivores", "flowers". "household electrical devices". and "vehicles 1". For Inception Score calculation, we generate 10,240 samples and use Inception Score Pytorch
CelebA-HQ (256 256). The original CelebA-HQ dataset (Karras al. 2018) comprises of 30,000 face images with 6,217 unique identities. We downloaded a preprocessed dataset from https 7/github. com ndb7 HSTt which contain 5,478 images from 307 identities, each with 15 or more images. For our experiment, we randomly sample 50 identities, totaling 895 images. Also, match the input dimension required by the implementation of latent diffusion model, we resize the original images from a resolution of 1024 x 1024 to 256 x 256. To calculate the diversity score based on Equation (8), we first generate 1, 1,024 samples from the model trained on the full dataset and extract image embeddings using the pretrained BLIP-VQA image encode (M Li et al. 2022). These embeddings are then clustered into 20 distinct groups as the reference clusters using the Ward linkage criterion (Jh Jr 1963). representing groups with demographic characteristics. Subsequently, for models trained on various subsets of celebrities, we generate 1,024 samples and assign them into these 20 reference clusters.
ArtBench mattesspressinnsmm (256 x 256). ArtBench- 10 a dataset for benchmarking artwork generation with 10 art styles, consisting of 5,000 training images per style (Liao et al. 2022). For our experiments, we consider the 5,000 training images corresponding to the art style "post- impressionism" from 258 artists. The aesthetic score predictor línear model built on top of CLIP to predict the aesthetic quality of images as a proxy to how much people, on average, perceive an image aesthetically pleasing.
C ADDITIONAL DETAILS ON TRAINING AND INFERENCE
CIFAR-20 and CelebA-HQ. For CIFAR-20, we follow the original implementation of the uncondi- tional DDPMs (Ho et al. 2020) where the model has 35.7M parameters. For CelebA-HQ, we follow the implementation of LDM (Rombach al. 2022 with 274M parameters and a pre-trained VQ- VAE (Razavi et al.) 2019). For both datasets, the maximum diffusion time step is set to T = 1000 during training. with a linear variance schedule for the forward diffusion process ranging from 3 10 to 3T = 0.02. For both CIFAR-20 and CelebA-HQ, we use the Adam optimizer (Loshchilov & Hutter 2017) and apply random horizontal flipping as data augmentation. Both the DDPM and LDM are trained for 20,000 steps with batch size of 64 and learning rate of 10 During inference, images are generated using the 100-step DDIM solver (Song et al. 2020a).
ArtBench Post-Impressionism)) A Stable Diffusion mode Rombach et al. 2022) is fine-tuned using LoRA (Hu et 2021) with rank = 256, corresponding to 5.1M LoRA parameters. The prompt set to "a Post-Impressiooist painting" for each image. The LoRA parameters are trained using the AdamW optimizer Loshhhilov & Hutter 2017) with weight decay of 10-6, for 200 epochs and
18

---

Published as a conference paper at ICLR 2025
batch size of 64. Cosine learning rate annealing is used, with 500 warm-up steps and initial learning rate of 3 10"h At inference time, images are generated using the PNDM scheduler with 100 steps (Karras et al. 2022
For training diffuusio models, we utilize NVIDIA GPUs; RTX 6000, A40, and A100, equipped with 24GB, 48GB, and 80GB of memory, respectively. This study employs the Diffusers package version 0.24.0 to train models across. Per-sample gradients are computed using the techniques outlined in the PyTorch package tutorial (version 2.2.1) [4 Furthermore, we apply the TRAK package to project gradients with random projection matrix. All experiments are conducted on systems equipped with 64 CPU cores and the specified NVIDIA GPUs.
For CelebA-HQ, we implement two techniques to reduce GPU memory usage during training. First, we pre-compute the embeddings of VQ-VAE for all training samples. since the VQ-VAE kept frozen during training. This approach allows us to avoid loading the VQ-VAE module into GPU memory during training. Second, we use the 8-bit Adam optimizer implemented the bitsandbytes Python package (Dettmers al., 2022).
D EMPIRICAL VERIFICATION OF PROPOSITION 1 AND COROLLARY 1
Proposition D suggests that increasing the number of fine-tuning steps can lead to a better approxima- tion of the retrained model property F(O3,) using the sparsified fine-tuned model property F(0S)
Here, we empirically measure the similarity between F(f's) ) and F( with 100 subsets Sj sampled from the Shapley kernel and varying the number of sparsified fine-tuning steps k. As Proposition [ provides an upper bound on the estimation error, the possibility of a constant shift should be considered
16 Therefore, focus the Pearson correlation between (F(65,)) ) and
(F(OS, )) } as the similarity metric. As shown in Figure 5 the approximation generally becomes better with more fine-tuning steps.
CIAAN 20
Aitherch geet iroresiviini
0.59
0.00
6.5
2008 100 2000
60 2p0 800
200 100
210 20 AG 400
Figure 5: Pearson correlation between model behaviors evaluated with retraining vs. sparsified fine-tuning, with varying number of fine-tuning steps. Models are retrained or fine-tuned on 100 contributor subsets sampled from the Shapley kernel.
Proposition suggests that increasing the number of fine-tuning steps can also lead to a better approximation of the retrained Shapley values (denoted as 3*) using Shapley values estimated through sparsified fine-tuning (denoted as 3.). Here, we compare the correlation between 3 and 3 both estimated with 500 contributor subsets, with varying number of fine-tuning steps k. As shown in Figure 6 the approximation tends to become better with more fine-tuning steps.
Finally, we note there is a trade-off between better approximation and computational efficiency. As shown in Figure 7 average runtime increases linearly with more fine-tuning steps.
httpa://pytorchh.ogg cdu/ wwtpaat....uuuuucttttraar
For example, E| F(ss) < B still holds when F(00)..) ) F(B2,) ) B' for some - B
B' < B
19

---

Published as a conference paper at ICLR 2025
CFAR-22
CelesA/11Q
Artendh Passimpraiinnmn
0.90
6.4
0.0
6-0
655
1300
1200
2000
1506
3500
185 350
650
100
300
300
150
800
400
Figure 6: Pearson correlation between the Shapley values estimated with retraining vs. sparsified fine- tuning, with varying number of fine-tuning steps. Shapley values estimated with 500 contributor subsets sampled from the Shapley kernel.
CIMA-33
Aedtonch (Part Intremlantmi
30
18-
Niabber Taninn 5.053
Number of Yru Taning Skige
Mamber hru Staps
Figure 7: Average runtime of sparsified fine-tuning for the Shapley kernel.
E DATA ATTRIBUTION METHODS FOR DIFFUSION MODELS
Here we provide definitions and implementation details of the baselines used in Section For TRAK-based approaches, follow the implementation details in Zheng al. 2023 17
Raw pixel similarity. This is a simple baseline that uses each raw image as the representation and then computes the cosine similarity between the generated sample of interest and each training sample as the attribution score.
Embeddings distance. This method computes the distances between training samples encoded using BLIP-VQA [8] Li et al. 2022), and the average embedding of each reference cluster as the
attribution score.
CLIP similarity. This method utilizes CLIF Radford et al. 2021 to encode each sample into an embedding and then compute the cosine similarity between the generated sample of interest and each training sample the attribution score.
Leave-one-out. Leave-one-out (LOO) evaluates the change in model property due to the removal of a single data contributor i from the training set (Koh & Liang 2017). Formally, it is defined as:
[Fo D); = -(fx)))((
F(B
(54)
where F(O*) denotes the global property for the model trained with the full dataset, and F(O2).c.) represents the global property of the mode trained with the data provided by the ith contributor.
Gradient similarity. This is gradient-based influence estimator (Charpiat et al. 2019 which computes the cosine similarity using the gradient representations of the generated sample interest, and each training sample x(i) as the attribution score:
P o Simple (x()) 8*) PTV, 9 Simpple (3;; 8*)
(55)
I V Ciimpte (1; )T PT V Simple (x; 0+)
I hnttee///atthbb..coo///lla////// /bl baae httpss///iithub.ccom//mm/m
20

---

Published as a conference paper at ICLR 2025
TRAK. Park et al. (2023) propose approach that aims to enhance the efficiency and scalability of data attribution for classifiers, Zheng et al. (2023) adapt TRAK for attributing images generated by diffusion models, defined follows:
Σ
+
x1) -1 Po Velimipple (x; 0g), and
(56)
Φ. - d. (x
so (x where d,(x) = P. V Cimple (x; 02),
(57)
where Simple is the diffusion loss used during training, P, random projection matrix, and XI serves for numerical stability and regularization.
D-TRAK. Zheng
(2023) find that, compared the theoretically motivated setting where the loss function is defined as L == Simple. using alternative functions- ssuc-suchh as the squared loss L to replace both the loss function and local model property result improved performance:
D. ④ D. AT) Pp VQuuuee (x;0)) and
(58)
2, (1) ................................ o (x (x ),.,,x"))....,,'''
[m)) where (x(x) = P. V VoLSquere (x; 07).
(59)
Journey-TRAA TRAK. Georgiev et al. (2023) focus on attributing noisy images x, during the denoising process. It attributes training data only for the final generated sample but also for the intermediate noisy samples throughout the denoising process. At each time step, the reconstruction loss is treated
local model property. Following (Zheng al. 2023), we compute attribution scores by averaging over the generation time steps, as shown below:
11 S
+ D. AT) P. V Vinisi (Rc; 02); and
(60)
4. = 0s
2 (x))
where (x = P. V Cimpte (x; 07).
(61)
Here, Simple is the diffusion loss restricted to the time step t (hence without taking the expectation over the time step).
In practice, follow the main implementation by Zheng al. (2023 that considers TRAK, D- TRAK, and Joumey-TRAA as retraining- free methods. That is, S and 07 0* is the original diffusion model trained on the entire dataset.
Relative Influence. Proposed by Barshan et al. (2020), the d-relative influence functions estimator Liang(2011) by the HVP magnitude. Follow- normalizes the influence functions estimator of Koh Liang (2017)
ing Zheng et al. (2023), we combine this estimator with TRAK dimension reudction. The attribution for each training sample x(i) as follows:
PT Vo Lsimpte (x;; 0*) (DTRAK" TTRAKK AT)
PT PTV Liimll (x0);
(62)
(DIRAK" T. PIRAK + +A)=
.PTV SSimple (x(1); (m))
and
@TRAK [)(x())..... [ where (bx) =P= TTV V Vo (x; 0*).
(63)
Renormalized Influence. Introduced by Hammoudeh Lowd (2022). this method renormalizes the influence functions by the magnitude of the training sample's gradients. Similar to relative influence, made an adaptation following Zheng et al. 2023). The attribution for each training sample x(6) is as follows:
PT CSimpte (x; 0*) (PTRAK" PTRAK x1) .TT Simple (x())00)
(64)
| V Ciimple x01;0'))
21

---

Published as a conference paper at ICLR 2025
and
PIRAK [ )... [ where o(x) P V Simpie (x: 8*).
(65)
For TRAK-based methods and gradient similarity, we compute gradients using the final model checkpoint, meaning that S in Equation (56). We select 100 time steps evenly spaced within the interval []] for computing the diffusion loss. At each time step, we introduce one instance of noise. The projection dimensions are set k 4096 for CIFAR-22 and k 32768 for CelebA-HQ and ArtBench (Pot--lmpressooiimm). For D D-TRAK, we use the best-performing output function Square
and choose À = 5e as in Zheng et al. (203)).
TracIn. Pruthi al. 20200) propose attribute the training sample's influence based on first-order approximation with saved checkpoints during the training process, However, this is also a major limitation because sometimes it impossible obtain checkpoints for models such Stable
Diffusion RRombach et al. 2022).
22

---

Published as a conference paper at ICLR 2025
F ADDITIONAL EXPERIMENT RESULTS
In this section, we present additional experiment results, including 1) LDS results across different datamodel alpha and computational budgets, 2) generated images after removing top contributing groups, 3) the performance of alternative unlearning approaches in CIFAR-20, and 4) analysis for top contributors in each dataset.
Table 2:
Average runtime for data subset in minutes (training + inference) for Shapley values estimated with retraining, fine-tuning (FT), sparsified FT, and LoRA fine-tuning.
Method
CIFAR-20
CelebA-HQ
ArtBench (Post-lmpressionism))
Retrain
77.4 + 20.8
213.4 26.5
190.6 + 6.4 4.4 6.4
FT
6.06 18.9
17.5 + 27.0
Sparsified FT LoRA
4.37 14.0
11.0 + 11.9
4.5 6.1
4.21 + 19.0
5.5 + 26.7
First, we provide the training cost incurred for each method. Utilizing 8 RTX-6k GPUs, the total runtime cost for sparsified-FT Shapley is under one day for each dataset. contrast, the corresponding runtime costs for retraining-based Shapley are 4.3 days, 10.4 days, and 17.1 days. This highlights that sparsified-FT Shapley is more computationally feasible than retraining-ban Shapley.
F.1 CIFAR-20
Table 3: LDS (%) results with 0.25, 0.5, 0.75 CIFAR-20 with the Inception Score of 10,240 generated images as the global model property. Means and 95% confidence intervals across three random initializations reported.
Method
a = 0.25
a = 0.5
a = 0.75
Pixel similarity (average)
-10.88 + 9.61
-11.81 + 4.56
-00.82 + 5.57
Pixel similarity (max)
-17.42 + 0.83
31.80
-23.88 + 1.28
CLIP similarity (average)
-0.34 +: 4.65
-21.27
-13.13
CLIP similarity (max)
- 15.19
11.31
16.05
Gradient similarity (average)
= 10.98 6
3.79
10.75
Gradient similarity (max)
7
+3.1
#4.71
Relative IF
1.14 13.51
5.50
5.21 3.78
Renormalized IF
9,09 + 13.40
6.79
7.99 + 4.02
TRAK
3.62 14.02
5.67
6.59
Journey-TRAL TRAK
30.009 3.85
42.92
-40.43 3.46
D-TRAK
21.02 + 6.76
10.90 1.21
21.90 + 5.02
LOO
17.01 +: 5.29
30.66 6.11
13.64 14.99
Sparsified-FT Shapley (Ours)
51.24 3.39
61.48 2.27
59.15 4.24
23

---

Published as a conference paper at ICLR 2025
Table LDS (%) results on Shapley value with different number of subset S.
Method
= 100
S = 200
S = 300
S 400
== 500
Retraining
43.88 (3.97)
49.95 (3.20)
57.56 (2.52)
62.68 (1.85)
65.90 (1.52)
0.25
Sparsified-FT
8.46 (13.41)
23.92 (6.27)
33.88 (8.33)
54.88 (7.44)
51.24 (3.39)
FT
13.57 (7.09)
40.01 (8,30)
(8.59)
24.58 (6.07)
20.57 (3.02)
LoRA
12.00 (0.65)
29.61 (1.37)
54.26 (3.38)
51.55 (6.40)
61.60 (7.44)
Retraining Sparsified-FT FT
48.84 (2.00)
39.55 (3.17)
57.24 (2.69)
66.81 (2.99)
70.58 (2.05)
0.5
-5,41 (4.78)
38.00 (3.06)
47.74 (1.49)
63.62 (2.30)
61.48 (2.27)
FT
13.57 (7.09)
40.01 (8.30)
28.94 (8.59)
24.58 (6.07)
39.60 (3.03)
LoRA
29.90 (2.96)
41.41 (1.28)
41.27 (0.59)
34.20 (0.53)
30.86 (1.07)
Retraining Sparsified-FT
51.39 (3.21)
45.85 (3.99)
61.91 (4.62)
70.59 (3.13)
72.07 (4.83)
0.75
-11.44 (2.26)
33.98 (0.98)
48.16 (5.18)
59.65 (5.34)
59.15 (4.24)
FT
33,90 (5.88)
26.09 (8,55)
45.49 (6.83)
46,92 (5.96)
38,51 (4.62)
LoRA
24.55 (2.16)
31.07 (7.83)
32.67 (7.94)
35.01 (7.05)
38.16(8.14)
0.25
0.5
0.75
Spardtied FT
1D5
Retrain
Lonk
7
Compulational Budgel
Compulational Budget.
Computational Budgel
Figure 8: LDS (%) results on CIFAR-20 for Shapley values estimated with sparsified tne-tuning (FT), finet- finet-tuning (FT), LoRA fine-tuning, and retraining under the same computational budgets (1
unit runtime to retrain and run inference on full model).
Table 5: LDS (%) results for retraining-based attribution across Shapley, Leave-One-Out (LOO), and Banzhaf distributions with 0.25, 0.5, 0.75 CIFAR-20. The global model behavior is evaluated using the Inception Score of 10,240 generated images. Means and 95% confidence intervals across three random initializations are reported. For sparsified fine-tuning (sFT) and fine-tuning (FT), the number of fine-tuning steps is set to 1000.
Method
a 0.25
0.5
Q 0.75
LOO (retraining)
17.01 + 5.29
30.66 6.11
13.64 +4.99
Banzhaf (retraining)
10.59 + 8.64
37.11 + 2.33
42.78 + 1.85
Shapley (retraining)
65.90 + 1.52
70.58 2.05
72.07 +4.83
LOO (FT)
-55.00 - 11.76
-66.06 + 2.28
-54.58 +7.02
Banzhaf (FT)
-11.20 6.42
9.09 2.67
16.79 = 1.12
Shapley (FT)
20.57 + 3.02
39.60 + 3.03
38.51 + 4.62
LOO (sFT)
29.45 5.96
27.43 4.20
19.58 + 0.35
Banzhaf (sFT)
5.44 + 8.59
22.55 5.07
31.44 + 0.27
Shapley (sFT)
51.24 3.39
61.48 2.27
59.15 +-2 4.24
24

---

Published as a conference paper at ICLR 2025
Sparsified-FT Shapley
D-TRAK
CLIP Similarity (Max)
IS = 6.134
IS =.16.100
IS = 6.258
IS = 5.604
6.143
IS = 6.552
IS = 5.352
=
IS = 6.365
Figure 9: Inception scores and example images generated with the same initial noises for DDPMs trained without the 10%, 20%, and 40% (top to bottom) most important CIFAR-20 classes based on sparsified-FT Shapley, D-TRAK, and CLPP similarity (max).
25

---

Published as a conference paper at ICLR 2025
F.2 CELEBA-HO
Table 6: LDS (%) results with 0.25, 0.5, 0.75 on CelebA-HQ, with the cluster entropy of 1,024
generated images as the global model property. Means and 95% confidence intervals across three random initializations reported.
Method
= 0.25
= 0.5
a = 0.75
Pixel similarity (average)
-1.52 = 3.14
-8.91 + 0.93
-4.19 + 2.25
Pixel similarity (max)
9.53 6.35
21.70 2.005
24.76 + 0.96
Embedding dist. (average)
3.59 + 6.47
13.83
10.89 + 2.04
Embedding dist. (max)
2.17
3.11
2.87
CLIP similarity (average)
-
-3887
-2..29
CLIP similarity (max)
-.55
-388
2..6
Gradient similarity (average)
-1.32
-21.29 # 1.76
Gradient similarity (max)
15.98
- 12.90 1.60
25.41 2.42
Relative IF
-..14
-1.07 + 0.68
-5.0 +1.13
Renormalized
-5.27 6.15
10.17
0.39
TRAK
-6.59 1:59::50 5.04
3.22
-3.65
Journey-RRAK
16.82
2.88
2.226
D-TRAK
-30.77 = 4.26
-7.233
-27.67
LOO
11.9 8.32
-1.22 6.34
8.47 -7.93
Sparsified-Fi Shapley (Ours)
10.05 + 5.33
26.34 + 3.34
24.08 + 1.52
Table 7: LDS (%) results on Shapley value with different number of subsets S
Method
= 100
5 = 200
$ = 300
S = 400
$ = 500
Retraining
3.54 (0.59)
8.91 (2.61)
16.77 (4.75)
15.58 (6.23)
31.52 (5.56)
0.25
Sparsified FT
9.95 (5.77)
15.93 (6.29)
9.90 (8.17)
16.12 (6.40)
10.05 (5.33)
FT
-10.52 (4.26)
2.77 (5.79)
12.29 (4.50)
9.86 (4.84)
13.91 (3.91)
LoRA
3.07 (1.17)
19.03 (9.47)
26.20 (9.73)
27.17 (7.72)
31.09 (7.72)
Retraining
19.01 (1.60)
28.43 (2.71)
35.06 (3.48)
23.48 (2.95)
28.58 (3.91)
Sparsified FT
20.73 (2.89)
26.83 (3.96)
12.71 (2.06)
27.23 (2.94)
26.34 (3.42)
FT
1.82 (3.36)
9.25 (2.59)
12.17 (2.83)
8.63 (0.52)
13.77 (1.42)
LoRA
10.66 (3.56)
(2.72)
13.99 (1.13)
21.12 (0.85)
22.37 (0.82)
Retraining
21.14 (0.93)
26.29 (1.37)
31.87 (2.85)
31.60 (2.23)
32.26 (2.08)
0.75
Sparsified FT
16.05 (0.31)
29.35 (1.43)
9,49 (2.03)
17.70 (1.30)
24.08 (1.52)
FT
-11.02 (1.51)
-12.07 (4.30)
1.93 (4.82)
0.96 (4.16)
7.37 (3.79)
LoRA
6.05 (1.17)
9.66 (1.43)
1..270
9.00 (2.18)
14.60 (2.33)
- 0.5
0.5
0.75
Sparatted IT
IDD.
Retrii
350
152
300
200
330
Computational Budget
Computati Budget
Computational Budget
Figure 10: LDS (%) results on CelebA-HQ for Shapley values estimated with sparsified fine-tuning (FT), fine-tuning (FT), and retraining under the same computational budgets (1 unit = runtime to retrain and run inference full model).
26

---

Published as a conference paper at ICLR 2025
Table 8: LDS (%) results for retraining-base attribution across Shapley, Leave-One-Ous (LOO), and Banzhaf distributions with 0.25, 0.5, 0.75. The global model behavior is calculated with the cluster entropy of 1,024 generated images. Means and 95% confidence intervals across three random initializations reported. For sparsified fine-tuning (sFT) and fine-tuning (FT), the number of training steps is set to 500.
Method
0.25
0.5
0.75
LOO (retraining)
11.90 8.32
-1.22 6.34
-8.20 +0.27
Banzhaf (retraining)
7.87 1.31
6.79 + 1.27
12.54 + 2.60
Shapley (retraining)
31.52 5.56
28.58 + 3.91
32.26 +2.08
LOO (FT)
-2.70 + 1.84
-10.15 + 1.12
-10.09 + 1.70
Banzhaf (FT)
1.50 7.36
-2.78 0.69
3.23 + 1.91
Shapley (FT)
13.91 + 3.91
13.77 + 1.42
7.37 = 3.79
LOO (sFT)
0.93 2.83
-9.26 + 0.39
-18.80 + 1.28
Banzhaf (sFT)
7.87 = 1.31
6.79 + 1.27
12.54 + 2.60
Shapley (sFT)
10.05 + 5.33
26.34 + 3.34
24.08 + 1.52
27

---

Published as a conference paper at ICLR 2025
F.3 ARTBENCH PPOTT-MPPRESSIONISM
Table 9: LDS (%) results with 0.25, 0.5, 0.75 on ArtBench (Post-Impredd (ost-Impresssonismm) with the 90th percentil of aesthetic scores for 50 generated images as the global model property. Means and 95% confidence intervals across three random initializations are reported.
Method
a 0.25
a = 0.5
at = 0.75
Pixel similarity (average)
16.57 (3.16)
11.24 (0.63)
-1.19 (5.71)
Pixel similarity (max)
16.62 (2.95)
14.61 (2.72)
(10.57)
CLIP similarity (average)
-6.03 (1.61)
-6.96 (4.08)
-9.27 (2.74)
CLIP similarity (max)
8,94 (0.43)
-1.75 (4.07)
1..00
Grad similarity (average)
4.14 (5.14)
0.25 (1.18)
-7.81 (0.89)
Grad similarity (max)
22.10 (9.02)
10.48 (3.11)
1.32 (4.19)
Aesthetic score (average)
24.81 (3.00)
24.85 (2.30)
13.21 (7.82)
Aesthetic score (max)
31.73 (3.04)
21.36 (3.70)
7.45 (11.66)
Relative IF
3.30 (4.98)
-5.02 (1.77)
-3.77 (12.05)
Renormalized IF
1.71 (4.37)
-1.41 (0.93)
-8.96 (12.86)
TRAK
-0.85 (4.74)
-8.18 (1.30)
-6.78 (13.37)
Journey-TRAK
-14.10 (5.09)
-11.41 (4.22)
-6.83 (2.80)
D-TRAK
19.37 (3.29)
11.30 (3.47)
17.72 (6.87)
Sparsified-FT Shapley (Ours)
52.83 (3.58)
61.44 (2.04)
32.24 (10.93)
8 0.25
= 0.75
LDS (96)
LOS 196)
Sparste-FF
Retran
Spareted-1
Renh
50
Computational Budget
Computational Budget
Computational Budget
Figure 11: LDS (%) results with a = 0.25, 0.5, 0.75 on ArtBench Post-Impressionine for Shapley values estimated with sparsified fine-tuning (FT), fine-tuning (FT), and retraining under the same computational budgets (1 unit runtime to retrain and run inference on full model).
Table 10: Aesthetic Score (90th percentile of 50 generated images) as the global model behavior. 100 datamodel subsets with alpha = 0.25, 0.5, 0.75 are used for evaluation. Means and 95% confidence intervals across three random initializations are reported. All Shapley values are estimated with 1,000 subsets.
Ct
Retrain
Sparsified - FT
FT
0.25
53.25 (6.75)
51.15 (7.54)
52.83 (3.58)
0.50
52.30 (2.32)
51.61 (3.21)
61.44 (2.04)
0.75
23.78 (14.07)
33.77 (6.65)
32.24 (10.93)
28

---

Published as a conference paper at ICLR 2025
Table 11: LDS (%) results with a = 0.25, 0.5, 0.75 on ArtBench (Pott-Impressionism)) with the 90th percentile of aesthetic score for 50 generated images the global model property. Means and 95% confidence intervals across three random initializations are reported. For sparsified fine-tuning (sFT) and fine-tuning (FT), the number of fine-tuning steps is set to 200. For Banzhaf and Shapley values, 1000 sbusets are used for estimation.
Method
0.25
0.5
0.75
LOO (retraining)
2.69 + 4.57
3.74 + 8.00
-1.13
15.78
Banzhaf (retraining)
4.70 = 7.28
16.19 +1.98
-10.16 +9.49
Shapley (retraining)
53.25 6.75
52.30 + 2.32
23.78 + 14.07
LOO (FT)
-14.13 5.36
-0.84 + 1.44
6.62 + 6.72
Banzhaf (FT)
3.55 + 6.99
15.30 + 2.09
-10.79 9.40
Shapley (FT)
61.44 + 2.04
32.24 + 10.93
LOO (sFT)
2.94 + 2.64
2.84 + 7.33
1.00 + 4.45
Banzhaf (sFT)
3
15.51 + 2.15
-10.43 9.16
Shapley (sFT)
51.15 = 7.54
51.61 + 3.21
33.77 + 6.65
29

---

Published as a conference paper at ICLR 2025
Aesthetic Score
Aesthetic 7.489
Aesthetic Score
Aesthetic 7.120
D-TRAK
Aesthetic Score 7.520
Aesthetic Score 7.498
Aesthetic Score 7.477
Aesthetic Score 7.428
Pixel Similarity (Max)
Aesthetic Score 7.289
Aesthetic Score 7.258
Aesthetic Score 7,217
Aesthetic Score 7.122
Aesthetic Score (Average)
Aesthetic Score = 7.343
Aesthetic Score 7.196
Aesthetic Score = 7.180
Aesthetic Score = 7.168
Shapley (Sparsified FT)
Figure 12: Two generated images above and two generated images below the 90th percentile of aesthetic scores, for Stable Diffusion models LoRA-finetueed without the top 40% most important artists based on D-TRAK, pixel similarity (max), training image aesthetic score (average), and sparsified-FT Shapley.
30

---

Published as a conference paper at ICLR 2025
Aesthetic Score 7.467
Aesthetic 7.464
Aesthetic Score
Aesthetic 7.270
D-TRAK
Aesthetic Score 7.448
Aesthetic Score 7.311
Aesthetic Score 7.263
Aesthetic Score 7.119
Pixel Similarity (Max)
Aesthetic Score 7.260
Aesthetic Score 7.255
Aesthetic Score 7.180
Aesthetic Score 7.167
Aesthetic Score (Average)
Aesthetic Score = 7.259
Aesthetic Score 7.239
Aesthetic Score = 7.171
Aesthetic Score 7.102
(Sparsified FT)
Shapley
Figure 13: Two generated images above and two generated images below the 90th percentile of aesthetic scores, for Stable Diffusion models LoRA-finetueed without the top 40% most important artists based on D-TRAK, pixel similarity (max), training image aesthetic score (average), and sparsified-FT Shapley.
31

---

Published as a conference paper at ICLR 2025
F.4 ALTERNATIVE UNLEARNING APPROACHES FOR REMOVAL
In this section, we present the performance of unlearning approaches: fine-tuning (FT), gradient ascent (GA) (Graves et al., 202T), and influence unlearning (IU) (Izzo al. (02T). To evaluate how well these approaches approximate retraining, we randomly selected 100 subsets from the Shapley sampling distribution, and we computed the FID score between the samples generated by the model retrained on each subset vs. models undergone by the unlcarning methods. The resulting average FID scores are 25.5, 185.9, and 101.3 for sparsified fine-tuning (FT), sparsified GA, and sparsified IU, respectively (Table 12). In contrast, the average FID scores without sparsification are 43.1, 145.2, and 67.7 for fine-tuning (FT), GA, and IU, respectively (Table (2). These results indicate that while both GA and IU can effectively unlearn the target set, they significantly degrade overall image quality, Figure 14 and Figure I5 In contrast, sparsified FT demonstrates superior performance, producing better image quality than both GA and IU.
Table 12: FID scores between generated samples from different unlearning approximations and retrained models. Mcans and 95% confidence intervals across 100 random sampled subsets from Shapley distribution are reported.
Unlearning Method
FID Score
With Sparsification
Fine-Tuning (FT)
25.5 + 10.5 185.9 + 45.7
Gradient Ascent (GA)
Influence Unlearning (IU)
101.3 32.1
Without Sparsification
Fine-nuning (FT)
43.1 + 10,9
Gradient Ascent (GA)
145.2 + 15.4
Influence Unlearning (IU)
67.7+ 15.9
Figure 14: Sample images generated using various unlearning approaches on CIFAR-20 with identical noise inputs: retraining. sparsified fine-tuning (FT), sparsified gradient ascent (GA), and sparsified influence unlearning (IU) (from left to right).
Figure 15: Sample images generated using various unlearning approaches on CIFAR-20 with identical
noise inputs: retraining, fine-tuning (FT), gradient ascent (GA), and influence unlearning (IU) (from left to right).
32

---

Published as a conference paper at ICLR 2025
F.5 ANALYSIS OF TOP CONTRIBUTORS
Here, we present an additional analysis of the top contributors identified by sparsified FT. For CIFAR- 20, compute the entropy of the training images for each class based the softmax outputs of InceptionNet. In the case of CelebA-HQ, the top contributors are predominantly drawn from non-majority clusters.
235
60-
Figure 16: Entropy of each class in CIFAR-20 (left), redline indicates top 20% groups; Distribution of the demographic clusters in the CelebA-HQ dataset (middle). Top 4 celebrity identities by sparsified- FT Shapley (right).
33

---

Published as a conference paper at ICLR 2025
F.6 DATA QUALITY DISTRIBUTION
Enstrup
Data Contributor
Figure 17: Entropy distributions across data contributors in CIFAR-20.
18
Emieddinng Distance to Majority Claster
14-
Celebrity Identity
Figure 18: Distributions of embedding distance the average of majority clusters across celebrity identities in CelebA-HQ.
Anstweii Score
Artist
Figure 19: Distributions of aesthetic score across artists in ArtBench (Post-mmpressionsmm)
34


Reviews:
[{"review_id": "5RGLiQuEOo", "reviewer": "Reviewer_KRfe", "summary": "This paper presents a framework for attributing the contributions of data providers in diffusion models. The authors propose a framework that efficiently approximates retraining and rerunning inference for diffusion models, thus enabling the estimation of Shapley values for data contributors. This is achieved by investigating how global properties of diffusion models are influenced by data contributors. Empirically, it is demonstrated that the proposed framework outperforms existing data attribution\nmethods across three datasets, model architectures, and global properties.", "questions": "1. in definition 1, you said the function $\\tau(\\mathcal{F}, \\\\{C_i\\\\}_{i=1}^n)$ is supposed to assign a score to each contributor $i$. I'm wondering how this notation reflects this idea. Maybe it's a typo and it should be $\\tau(\\mathcal{F}, C_i)$?\n2. if I understand it correctly, in the experiment results (section 4.5), the LDS is computed with different baseline methods for computing $\\tau$. Then how are $\\{\\mathcal{F}(\\theta^*_{S_b})\\}_{b=1}^B$ computed? Are they computed by Sparsified-FT Shapley? If this is the case, why it is a fair comparison if the proposed approach is served as the benchmark in the evaluation metric?\n3. is the proposed approach applicable to any data-driven machine learning model? I don't see any reason why it must be restricted to the diffusion model. If this is the case, why this paper focus on the application of diffusion models?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, I think this paper is organized and written clearly and I enjoyed reading most of the parts. In particular, the conceptual strengths of this paper include:\n1. The proposed framework utilizes Shapley values, a game-theoretic approach, to assign fair credit to data contributors based on their influence on model performance. This methodology uniquely meets the fairness principles in valuation.\n2. To address the high computational cost of Shapley value calculations, the paper introduces a model-pruning and fine-tuning method, which significantly accelerates retraining and inference processes. \n3. The approach has the potential to be applicable in various scenarios, such as incentivizing quality data sharing, creating compensation policies, and improving model diversity and fairness, making it a nice tool for real-world diffusion model deployments​.", "weaknesses": "1. despite the computational efficiency of the proposed speed up method with sparsified fine-tuning (section 3.2), it seems to me that there is no in-depth discussion about its accuracy. The core idea of the approximation is Eq. (6), but there is no discussion or empirical evidence to support how well the approximation (6) is. Given that the idea is straightforward, it would be much more convincing if the author can provide additional justifications for Eq. (6), besides its superior empirical performance compared to baseline methods. Otherwise, it is difficult to digest why the proposed approach outperforms other methods with such a dominant advantage (Table 1).  \n2. I do not see any particular reason the approximation for computing Shapley value has to be restricted in the diffusion model applications. Whether this is true or not, it would be better to include additional discussions in this regard.\n3. I feel the contributions summarized at the end of the introduction is a bit over-claimed. For example, the first claimed contribution is not surprising from my perspective, as it is well-acknowledged that the performance of any ML model relies heavily on the sources of its training data set. For the second claim, I'm not so sure what does \"efficiently approximate\" mean. From the paper I get that the proposed approximation framework indeed reduces the computational efficiency of solving the least square problem (5), however, there is no evidence of how well the approximation is. In my opinion, an \"efficient\" approximation should somehow provide a trade-off between computational cost and approximation accuracy. That said, an approximation method with only a computational cost guarantee makes it less convincing and lacks insight. I think this paper can benefit more from an in-depth discussion of the proposed approximation approach. \n4. the focus of technical writing is not well-balanced. In my opinion, the entire section 2 and section 3.1 are known results (which do not contribute to the novelty of this work) and should be shortened significantly. However, unfortunately, the core novel part of the proposed method (section 3.2) is not discussed in depth.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EhJaUxwgZS", "reviewer": "Reviewer_f56T", "summary": "The paper entitled \"An Efficient Framework for Crediting Data Contributors of Diffusion Models\" has focused on diffusion models and presented a method to fairly attribute data contributions using Shapley values. To address computational inefficiencies, the authors employ model pruning and fine-tuning, enabling practical Shapley value estimation. Their method is validated across multiple datasets, demonstrating improved attribution accuracy and efficiency over existing techniques.", "questions": "How does the framework handle cases where data contributors have varying data quality, styles, etc. and could this affect the accuracy of attribution?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1- The Shapley theorem has been proven to be an effective solution for calculating the contribution and is widely used in data valuation. The author has properly utilised this theorem as part of the methodology. \n2- Regarding quality, the methodology is well-executed, with rigorous evaluations across multiple datasets that demonstrate the proposed framework’s superior performance.\n3- About the clarity, the paper is well-written, with structured explanations and nice visualization,", "weaknesses": "1- It is not a weakness, but the author could also provide an evaluation against data poisoning, which could make it even stronger. \n2- The provided code is well-structured but it could be improved by providing further demo Jupyter notebooks that make it easier for others to test and run the model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "v2KLQMhJV4", "reviewer": "Reviewer_2xYg", "summary": "This paper proposes to use a combination of kernel-based Shapley value and sparse fine-tuning as a new method to credit the data contributor in diffusion models. The authors evaluated their approach on CIFAR-20 CelebA-HQ and Art", "questions": "Listed in the weaknesses", "limitations": "", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "If the comparison and contribution calculation methods are indeed correct and reasonable, the numerical results look good", "weaknesses": "I'm not very convinced about the overall proposed method for the following reasons.\n\nFor the sparsified FT part. some design choice can be elaborated and some comparisons with alternative methods can make the method more convincing.\n\n1. Why is training on the full data and then finetuning on the subset comparable with training with subset from scratch?\n2. Why pruning? Why not suing alternative solutions like full model with LoRA instead?\n3. When applying sparsified FT, what are the contribution score formulas?\n\nFor the Shapley value part.\n\n4. It will be better to elaborate on the similarity and differences between the conventional prediction tasks and the generation tasks, why the kernel based Shapley value can still work well in the current situation needs more explanation as well\n\nFor the numerical results, the ablation study is needed.\n\n5. In table 1, the baselines used models trained with subset from scratch but the proposed method used sparsified FT. The results will be more straight forward when using retraining from scratch for all these scoring method or use sparsified FT for all these methods, that will show the ablation for each component in the new method \n\nOverall speaking, the novelty seems fair.\n\n6. The proposed method is a combination of Shapley value and sparse FT, and I think the reasoning for using this method will be much stronger if the authors can provide evidence showing that each component is better (either efficiency or effectiveness) than their alternatives in this task, like for contribution measurements, Shapley values vs LIME, PFI, etc., and for speeding up model retraining sparse FT vs unlearning, etc.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)