Nonparametric Data Attribution for Diffusion Models

## Abstract

Abstract Data attribution for generative models seeks to quantify the influence of individual training examples on model outputs. Existing methods for diffusion models typically require access to model gradients or retraining, limiting their applicability in proprietary or large-scale settings. We propose a nonparametric attribution method that operates entirely on data, measuring influence via patch-level similarity between generated and training images. Our approach is grounded in the analytical form of the optimal score function and naturally extends to multiscale representations, while remaining computationally efficient through convolution-based acceleration. In addition to producing spatially interpretable attributions, our framework uncovers patterns that reflect intrinsic relationships between training data and outputs, independent of any specific model. Experiments demonstrate that our method achieves strong attribution performance, closely matching gradient-based approaches and substantially outperforming existing nonparametric baselines.

## 1 Introduction

Recent advances in generative models, particularly diffusion models (Ho et al., 2020; Song et al., 2021), have led to significant progress in image synthesis and editing (Rombach et al., 2022; Ramesh et al., 2022; Meng et al., 2022). As these powerful models are trained on increasingly large-scale datasets that often contain private, copyrighted, or low-quality content, concerns around data transparency, accountability, and ethical use are growing (Carlini et al., 2023; Saveri & Butterick, 2023). These concerns motivate the study of *data attribution*: identifying the influence of individual training examples on a given generation. Effective data attribution not only supports the responsible deployment of generative models but also enables various downstream applications, including interpreting model behavior (Koh & Liang, 2017; Sui et al., 2021; Ilyas et al., 2022), detecting mislabeled or poisoned data (Jia et al., 2021), guiding data valuation (Nohyun et al., 2023), and improving dataset quality through informed curation (Khanna et al., 2018; Jia et al., 2021; Liu et al., 2021).

Much progress has been made in attributing image generations to training data. Retraining-based methods (Ghorbani & Zou, 2019; Ilyas et al., 2022) assess how generations change when specific training data are removed. While effective, these methods typically require retraining the model tens of thousands of times on different data subsets (Ghorbani & Zou, 2019), making them computationally expensive. To improve efficiency, recent works (Zheng et al., 2024; Lin et al., 2025; Mlodozeniec et al., 2025) adopt approximations based on additive attribution scores (Park et al., 2023a), enabling scalable attribution on large datasets. A common assumption in these methods is access to model gradients, i.e., full access to the generative model. However, this assumption is not always practical. For example, in scenarios where users seek copyright protection against infringement by proprietary models (Somepalli et al., 2023; Zhao et al., 2024), the model gradients may not be accessible. Furthermore, when attribution is intended to support tasks such as data selection (Gu et al., 2025), training a generative model solely for attribution may be prohibitively costly or infeasible.

These challenges call for an attribution method that does not require access to the generative model, either to handle proprietary or black-box models or to serve as a fast surrogate without the cost of training or accessing the model.
Existing methods (Zheng et al., 2024; Mlodozeniec et al., 2025)
typically measure the similarity between generated images and training data in feature spaces, but often perform poorly as they disregard the behavior of the generative model.
Effective attribution for diffusion models in this restricted-access setting remains an open challenge.

In this paper, we present a nonparametric approach to image data attribution in diffusion models, which directly quantifies the influence of training samples on generated outputs through patch-level comparisons. Drawing inspiration from analytical expressions of the score function in diffusion models (Gu et al., 2024; Kamb & Ganguli, 2024), we extract local patches from both generated images and training samples and compute attribution scores based on pairwise distances, as illustrated in Figure [1](https://arxiv.org/html/2510.14269v1#S1.F1). Unlike prior methods that require retraining or gradient access, our method operates entirely on data, without any reliance on model architecture or training.
Moreover, it provides fine-grained, patch-level attribution, enabling the localization of influential training regions and offering interpretable insights into the generation behaviors.
Due to space constraints, related work is deferred to Appendix [A](https://arxiv.org/html/2510.14269v1#A1).

Figure: Figure 1: Schematic illustration of attribution methods. Left: Model-based attribution relies on gradients and requires parameter access. Right: Our *Nonparametric Diffusion Attribution* (NDA) compares local image patches via an analytic score function, enabling attribution without model access.
Refer to caption: x1.png

## 2 Preliminaries

Our approach is grounded in the optimal score function of diffusion models. In this section, we begin with a brief overview of diffusion models and their optimal empirical score functions, followed by a review of data attribution techniques and evaluation protocols.

### 2.1 Diffusion models

Diffusion models (Ho et al., 2020; Song et al., 2021) are a class of probabilistic generative models that learn to approximate a data distribution $q({\bm{x}})$ by modeling a parametrized Markovian process $p_{\theta}({\bm{x}})$. Specifically, it defines a forward process that transforms a clean data sample ${\bm{x}}\sim$ $q({\bm{x}})$ into a noisy sequence ${\bm{x}}_{1:T}={\bm{x}}_{1},\cdots,{\bm{x}}_{T}$ by gradually adding Gaussian noise. The transition probability is given by
$q\left({\bm{x}}_{t}|{\bm{x}}_{t-1}\right)\triangleq\mathcal{N}\left({\bm{x}}_{t}|\sqrt{1-\beta_{t}}{\bm{x}}_{t-1},\beta_{t}\mathbf{I}\right)$, where $\{\beta_{t}\}_{t=1}^{T}$ denotes a predefined variance schedule. A key property of the forward process is that ${\bm{x}}_{t}$ can be sampled in closed form at any timestep $t$:

$$ $q({\bm{x}}_{t}|{\bm{x}})=\mathcal{N}({\bm{x}}_{t}|\sqrt{\bar{\alpha}_{t}}{\bm{x}},(1-\bar{\alpha}_{t})\mathbf{I})\textrm{,}$ (1) $$

where $\bar{\alpha}_{t}\triangleq\prod_{s=1}^{t}\alpha_{s}$ and $\alpha_{t}\triangleq 1-\beta_{t}$.
As $t$ increases from 0 to $T$, the noise level increases and $\bar{\alpha}_{t}$ decays from $1$ to 0. Consequently, the marginal distribution $q_{t}({\bm{x}}_{t})$ approaches a standard Gaussian, i.e., $q_{T}({\bm{x}}_{T})\approx\mathcal{N}(0,\mathbf{I})$.
The reverse process is learned by approximating the reverse conditionals $q({\bm{x}}_{t-1}|{\bm{x}}_{t})$ using a neural network model $p_{\theta}({\bm{x}}_{t-1}|{\bm{x}}_{t})\triangleq\mathcal{N}({\bm{x}}_{t-1}|\bm{\mu}_{\theta}({\bm{x}}_{t},t),\sigma_{t}^{2}\mathbf{I})$, where the variance $\sigma_{t}^{2}$ are typically chosen as hyperparameters (Bao et al., 2022).
In practice, diffusion models are usually trained to predict the added noise $\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)$, which relates to the mean via $\bm{\mu}_{\theta}({\bm{x}}_{t},t)=\frac{1}{\sqrt{\alpha_{t}}}({\bm{x}}_{t}-\frac{\beta_{t}}{\sqrt{1-\bar{\alpha}_{t}}}\bm{\epsilon}_{\theta}({\bm{x}}_{t},t))$ (Ho et al., 2020).
The model is learned by minimizing a variational bound on the negative log-likelihood:

$$ $\mathcal{L}_{\text{ELBO}}({\bm{x}};\theta)=\mathbb{E}_{\bm{\epsilon},t}\left[\frac{\beta_{t}^{2}}{2\sigma_{t}^{2}\alpha_{t}\left(1-\bar{\alpha}_{t}\right)}\left\|\bm{\epsilon}-\bm{\epsilon}_{\theta}\left(\sqrt{\bar{\alpha}_{t}}{\bm{x}}+\sqrt{1-\bar{\alpha}_{t}}\bm{\epsilon},t\right)\right\|_{2}^{2}\right]\textrm{,}$ (2) $$

where $\bm{\epsilon}\sim\mathcal{N}(\bm{\epsilon}|\mathbf{0},\mathbf{I})$. Let $\mathbb{S}\triangleq\{{\bm{z}}^{(n)}|{\bm{z}}^{(n)}\sim q\}_{n=1}^{N}$ be the training dataset. The empirical training objective is then $\mathcal{L}_{\text{ELBO}}(\mathbb{S};\theta)=\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}_{\text{ELBO}}\left({\bm{z}}^{(n)};\theta\right)$.
To enhance sample quality, a simplified training objective is often used (Ho et al., 2020):

$$ $\mathcal{L}_{\text{Simple}}({\bm{x}};\theta)=\mathbb{E}_{\bm{\epsilon},t}\left[\left\|\bm{\epsilon}_{\theta}\left({\bm{x}}_{t},t\right)-\bm{\epsilon}\right\|^{2}\right]\textrm{,}$ (3) $$

with the empirical objective on $\mathbb{S}$ given by $\mathcal{L}_{\text{Simple}}(\mathbb{S};\theta)=\frac{1}{N}\sum_{n=1}^{N}\mathcal{L}_{\text{Simple}}({\bm{z}}^{(n)};\theta)$.

### 2.2 Optimal score functions

A key property (Song et al., 2021; Kingma & Gao, 2023) of diffusion models is that the optimal noise prediction function $\bm{\epsilon}^{*}({\bm{x}}_{t},t)$ is closely related to the *score function* ${\bm{s}}({\bm{x}}_{t},t)\triangleq\nabla_{{\bm{x}}_{t}}\log q_{t}({\bm{x}}_{t})$ via:

$$ ${\bm{s}}({\bm{x}}_{t},t)=-\dfrac{\bm{\epsilon}^{*}({\bm{x}}_{t},t)}{\sqrt{1-\bar{\alpha}_{t}}}.$ (4) $$

Intuitively, for a finite dataset $\mathbb{S}=\{{\bm{z}}^{(n)}\}_{n=1}^{N}$, the marginal distribution $q_{t}\left({\bm{x}}_{t}\right)$ becomes a Gaussian mixture centered at the scaled data points $\sqrt{\bar{\alpha}_{t}}{\bm{z}}^{(n)}$:
$q_{t}({\bm{x}}_{t})=\frac{1}{N}\sum_{n=1}^{N}\mathcal{N}\left({\bm{x}}_{t}|\sqrt{\bar{\alpha}_{t}}{\bm{z}}^{(n)},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)$.
The score function ${\bm{s}}({\bm{x}}_{t},t)$ therefore admits an analytical form:

$$ ${\bm{s}}({\bm{x}}_{t},t)=\nabla_{{\bm{x}}_{t}}\log q_{t}({\bm{x}}_{t})=\dfrac{1}{1-\bar{\alpha}_{t}}\sum_{n=1}^{N}(\sqrt{\bar{\alpha}_{t}}{\bm{z}}^{(n)}-{\bm{x}}_{t})W_{t}({\bm{z}}^{(n)}|{\bm{x}}_{t})\textrm{,}$ (5) $$

where $W_{t}({\bm{z}}^{(n)}|{\bm{x}}_{t})$ is a weighting term defined as:

$$ $W_{t}({\bm{z}}^{(n)}|{\bm{x}}_{t})=\frac{\mathcal{N}\left({\bm{x}}_{t}|\sqrt{\bar{\alpha}_{t}}{\bm{z}}^{(n)},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)}{\sum_{n^{\prime}=1}^{N}\mathcal{N}\left({\bm{x}}_{t}|\sqrt{\bar{\alpha}_{t}}{\bm{z}}^{(n^{\prime})},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)}.$ (6) $$

This score function ${\bm{s}}({\bm{x}}_{t},t)$ can be interpreted as a conditional average over the added noise, where the residual term ${\bm{x}}_{t}-\sqrt{\bar{\alpha}_{t}}{\bm{z}}^{(n)}$ is averaged across training examples, weighted by the posterior probability $W_{t}({\bm{z}}^{(n)}|{\bm{x}}_{t})$ that ${\bm{x}}_{t}$ was transformed from ${\bm{z}}^{(n)}$ at $t=0$ under the forward process.

### 2.3 Data attribution and evaluation metrics

Given a training dataset $\mathbb{S}$ and a generated sample ${\bm{x}}$, the goal of data attribution is to quantify the influence of each training example in $\mathbb{S}$ on the generation of ${\bm{x}}$. Formally, this involves assigning an attribution score $\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})$ to each training example, reflecting its relative importance in generating ${\bm{x}}$.

We follow Zheng et al. (2024) and adopt the linear datamodeling score (LDS; Park et al., 2023a) as our evaluation metric, which quantifies how well an attribution method aligns with the ground-truth influence of training data on model outputs. Specifically, for a training dataset $\mathbb{S}=\{{\bm{z}}^{(n)}\}^{N}_{n=1}$ of size $N$, LDS evaluates an attribution method $\tau$ by first sampling multiple random subsets $\{\mathbb{S}_{m}\subset\mathbb{S}\}_{m=1}^{M}$.
Let $\theta^{*}(\mathbb{S}_{m})$ denote the generative model trained on subset $\mathbb{S}_{m}$, and let $\bm{\mathcal{F}}({\bm{x}};\theta^{*}(\mathbb{S}_{m}))$ denote the model output on a test input ${\bm{x}}$.(^1^11For diffusion models, we set $\bm{\mathcal{F}}=\mathcal{L}_{\text{Simple}}$, representing the model output used in LDS evaluation.)
Then, for each subset $\mathbb{S}_{m}$, the attribution scores $\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})$ assigned to training examples in $\mathbb{S}_{m}$ are summed to form an attribution-based prediction $g_{\tau}({\bm{x}},\mathbb{S}_{m};\mathbb{S})$:

$$ $g_{\tau}({\bm{x}},\mathbb{S}_{m};\mathbb{S})\triangleq\sum_{{\bm{z}}^{(n)}\in\mathbb{S}_{m}}\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})\textrm{.}$ (7) $$

Finally, the LDS score for attribution method $\tau$ on test input ${\bm{x}}$ is calculated as the Spearman rank correlation $\rho(\cdot,\cdot)$ between the ground-truth outputs from the $M$ retrained models (each trained on a different subset $\mathbb{S}_{m}$) and the corresponding attribution-based predictions:

$$ ${\text{LDS}}(\tau,{\bm{x}})\triangleq\rho\Big(\{\bm{\mathcal{F}}({\bm{x}};\theta^{*}(\mathbb{S}_{m})):m\in[M]\},\{g_{\tau}({\bm{x}},\mathbb{S}_{m};\mathbb{S}):m\in[M]\}\Big)\textrm{.}$ (8) $$

## 3 Methodology

Our approach is motivated by the weighting term in the optimal score function of diffusion models, which naturally encodes the relative importance of each training example during generation. Leveraging this insight, we build upon the recent theoretical framework of Kamb & Ganguli (2024), which extends the analysis of optimal score functions to capture inductive biases that promote generalization in diffusion models. By bridging this framework with the problem of data attribution, we develop a nonparametric data attribution method that does not require access to model gradients or retraining.

### 3.1 Equivariant and local score machines

While Eq. ([5](https://arxiv.org/html/2510.14269v1#S2.E5)) defines the optimal score function, it relies solely on distance-based weighting in image space and does not generalize beyond the training set. Kamb & Ganguli (2024) derive an analytical form under inductive biases of *locality* and *equivariance*. This formulation preserves the structure of the empirical score but yields meaningful similarity by incorporating spatial structure and symmetry.

Let ${\bm{x}}_{t}\in\mathbb{R}^{C\times L\times L}$ denote a noisy image at diffusion time $t$, with $C$ channels and spatial resolution $L$.
For a pixel location $\ell\in[L]\times[L]$, let ${\bm{x}}_{t,\ell}\in\mathbb{R}^{C}$ denote its pixel value. Define $\Omega_{\ell}$ as the $P\times P$ neighborhood centered at $\ell$, and ${\bm{x}}_{t,\Omega_{\ell}}\in\mathbb{R}^{C\times P\times P}$ as the corresponding local patch.
Let $\mathbb{P}_{\Omega}(\mathbb{S})$ be the set of all such patches extracted from the training dataset $\mathbb{S}$. Each patch ${\bm{u}}\in\mathbb{P}_{\Omega}(\mathbb{S})$ is thus a local crop of some training image ${\bm{z}}\in\mathbb{S}$, and we denote its center pixel as ${\bm{u}}_{0}$.
Under the locality and equivariance assumptions, Kamb & Ganguli (2024) show that the optimal MMSE estimator of the score function at pixel location $\ell$, denoted by ${\bm{s}}({\bm{x}}_{t},t,\ell)\in\mathbb{R}^{C}$, takes the form:

$$ ${\bm{s}}({\bm{x}}_{t},t,\ell)=\sum_{{\bm{u}}\in\mathbb{P}_{\Omega}(\mathbb{S})}\dfrac{\sqrt{\bar{\alpha}_{t}}{\bm{u}}_{0}-{\bm{x}}_{t,\ell}}{1-\bar{\alpha}_{t}}W_{t}({\bm{u}}|{\bm{x}}_{t,\Omega_{\ell}})\textrm{,}$ (9) $$

where the weighting term $W_{t}({\bm{u}}|{\bm{x}}_{t,\Omega_{\ell}})$ is defined as:

$$ $W_{t}({\bm{u}}|{\bm{x}}_{t,\Omega_{\ell}})=\frac{\mathcal{N}\left({\bm{x}}_{t,\Omega_{\ell}}|\sqrt{\bar{\alpha}_{t}}{\bm{u}},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)}{\sum_{{\bm{v}}\in\mathbb{P}_{\Omega}(\mathbb{S})}\mathcal{N}\left({\bm{x}}_{t,\Omega_{\ell}}|\sqrt{\bar{\alpha}_{t}}{\bm{v}},\left(1-\bar{\alpha}_{t}\right)\mathbf{I}\right)}\textrm{.}$ (10) $$

This formulation generalizes Eq. ([5](https://arxiv.org/html/2510.14269v1#S2.E5)) by measuring similarity at the patch level rather than over full images, allowing it to exploit fine-grained local structure.

### 3.2 Patch-based data attribution scores

The weighting term in Eq. ([10](https://arxiv.org/html/2510.14269v1#S3.E10)) can be expressed as a softmax over quadratic distances $\frac{\|{\bm{x}}_{t,\Omega_{\ell}}-\sqrt{\bar{\alpha}_{t}}{\bm{u}}\|^{2}}{2(1-\bar{\alpha}_{t})}$, which naturally reflects the contribution of each training patch ${\bm{u}}$ to the generation of pixel ${\bm{x}}_{t,\ell}$. We reinterpret this term as a *patch-wise influence* score that quantifies the influence of a training patch ${\bm{u}}$ on the local region of a generated image. By aggregating these local scores across spatial locations, we obtain a nonparametric, spatially interpretable attribution measure.

Patch-wise influence. For a noisy patch ${\bm{x}}_{t,\Omega_{\ell}}$ centered at $\ell$ in a generated image ${\bm{x}}$ at timestep $t$, we adopt the local weighting from Eq. ([10](https://arxiv.org/html/2510.14269v1#S3.E10)) and define the patch-wise influence score as:

$$ $\!\!\tau({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))\triangleq\exp\!\left(-\frac{\left\|{\bm{x}}_{t,\Omega_{\ell}}\!-\!\sqrt{\bar{\alpha}_{t}}{\bm{u}}\right\|^{2}}{2(1-\bar{\alpha}_{t})}\right)\!\cdot\!\left(\sum_{{\bm{v}}\in\mathbb{P}_{\Omega}(\mathbb{S})}\!\exp\!\left(-\frac{\left\|{\bm{x}}_{t,\Omega_{\ell}}\!-\!\sqrt{\bar{\alpha}_{t}}{\bm{v}}\right\|^{2}}{2(1-\bar{\alpha}_{t})}\right)\!\right)^{\!-1}\!\textrm{,}$ (11) $$

which is a normalized similarity score over all patches in $\mathbb{P}_{\Omega}(\mathbb{S})$ extracted from the training set.

Image-level attribution.
Given a generated image ${\bm{x}}$, we apply the forward diffusion process to obtain noisy samples ${\bm{x}}_{t}$. For each pixel location $\ell$, we extract the $P\times P$ patch ${\bm{x}}_{t,\Omega_{\ell}}$ from ${\bm{x}}_{t}$, using zero-padding near boundaries to handle incomplete patches. The patch-wise influence scores in Eq. ([11](https://arxiv.org/html/2510.14269v1#S3.E11)) is then computed for all training patches ${\bm{u}}\in\mathbb{P}_{\Omega}(\mathbb{S})$.

To aggregate into an image-level attribution score $\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})$, we proceed as follows:
(1) For each patch ${\bm{x}}_{t,\Omega_{\ell}}$ of ${\bm{x}}_{t}$, we select the $k$ most influential patches from each training image ${\bm{z}}^{(n)}$, denoted by $\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})$.(^2^22Formally, $\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})\subset\mathbb{P}_{\Omega}(\{{\bm{z}}^{(n)}\})$ with $|\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})|=k$, and for all ${\bm{u}}^{\prime}\in\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})$ and ${\bm{u}}^{\prime\prime}\in\mathbb{P}_{\Omega}(\{{\bm{z}}^{(n)}\})\setminus\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})$, we have $\tau({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}}^{\prime};\mathbb{P}_{\Omega}(\mathbb{S}))\geq\tau({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}}^{\prime\prime};\mathbb{P}_{\Omega}(\mathbb{S}))$.)
(2) We sum the influence scores of these top-$k$ patches to estimate the contribution of the training image ${\bm{z}}^{(n)}$ to generating the local region around $\ell$.
(3) Finally, we aggregate across all spatial locations $\ell$ and average over a set of timesteps $\mathcal{T}$:

$$ $\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})\triangleq\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\sum_{\ell}\sum_{{\bm{u}}\in\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})}\tau({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))\textrm{.}$ (12) $$

This attribution score is both *spatially interpretable*, as it aggregates patch-wise influence with local meaning, and *nonparametric*, as it operates entirely on training data without relying on the model.

### 3.3 Attribution with multiscale patch-wise influence

The attribution method described above relies on quadratic Euclidean distances between fixed-size patches in the original image space. However, images may have varying resolutions, and a single patch size may fail to capture different levels of information: from fine-grained textures to higher-level structures (Adelson et al., 1984; Lin et al., 2017). Moreover, diffusion models are known to generate different levels of detail across timesteps: early (high-noise) stages capture coarse structures, while later (low-noise) stages refine local details (Jing et al., 2022; Park et al., 2023b).

To account for these effects, we introduce a *multiscale* extension that computes patch-wise influence across multiple resolutions. Specifically, we downsample both generated and training patches and evaluate distances in the lower-resolution space. Let $\text{D}(\cdot)$ denote a downsampling operator, and define $\widehat{{\bm{x}}}_{t,\Omega_{\ell}}\triangleq\text{D}({\bm{x}}_{t,\Omega_{\ell}})$, $\widehat{{\bm{u}}}\triangleq\text{D}({\bm{u}})$, and $\widehat{{\bm{v}}}\triangleq\text{D}({\bm{v}})$. The low-resolution patch-wise influence score is then:

$$ $\!\!\widehat{\tau}({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))\triangleq\exp\!\left(-\frac{\left\|\widehat{{\bm{x}}}_{t,\Omega_{\ell}}\!-\!\sqrt{\bar{\alpha}_{t}}\widehat{{\bm{u}}}\right\|^{2}}{2(1-\bar{\alpha}_{t})}\right)\!\cdot\!\left(\sum_{{\bm{v}}\in\mathbb{P}_{\Omega}(\mathbb{S})}\!\exp\!\left(-\frac{\left\|\widehat{{\bm{x}}}_{t,\Omega_{\ell}}\!-\!\sqrt{\bar{\alpha}_{t}}\widehat{{\bm{v}}}\right\|^{2}}{2(1-\bar{\alpha}_{t})}\right)\!\right)^{\!-1}\!\textrm{.}$ (13) $$

We then combine the original and low-resolution influence measures into a multiscale score:

$$ $\tau^{\text{ms}}({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))\triangleq\gamma_{t}\tau({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))+(1-\gamma_{t})\widehat{\tau}({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))\textrm{,}$ (14) $$

where $\gamma_{t}\in[0,1]$ is a timestep-dependent weighting factor that balances the contribution of the original and low-resolution influence.
Finally, we extend this to multiscale image-level attribution by aggregating over timesteps $\mathcal{T}$ and spatial locations $\ell$:

$$ $\tau^{\text{ms}}({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})\triangleq\frac{1}{|\mathcal{T}|}\sum_{t\in\mathcal{T}}\sum_{\ell}\sum_{{\bm{u}}\in\mathbb{P}^{k}_{\Omega}({\bm{x}}_{t,\Omega_{\ell}},{\bm{z}}^{(n)})}\tau^{\text{ms}}({\bm{x}}_{t,\Omega_{\ell}},{\bm{u}};\mathbb{P}_{\Omega}(\mathbb{S}))\textrm{.}$ (15) $$

### 3.4 Convolution-based acceleration

Directly evaluating Eq. ([11](https://arxiv.org/html/2510.14269v1#S3.E11) & [13](https://arxiv.org/html/2510.14269v1#S3.E13)) on large-scale datasets is computationally challenging. In a naive implementation, each test patch ${\bm{x}}_{t,\Omega_{\ell}}$ is broadcast against all unfolded training patches ${\bm{u}}\in\mathbb{P}_{\Omega}(\mathbb{S})$, leading to peak memory consumption of $\mathcal{O}(NL^{2}CP^{2})$, which is $P^{2}$ times larger than the dataset itself.

To address this, we propose a memory-efficient implementation that avoids explicit patch unfolding by leveraging convolutional operators. Specifically, to compute $\|{\bm{x}}_{t,\Omega_{\ell}}-\sqrt{\bar{\alpha}_{t}}{\bm{u}}\|^{2}$ for all $L\times L$ patches ${\bm{u}}$ from a training image ${\bm{z}}^{(n)}$, we treat the patch ${\bm{x}}_{t,\Omega_{\ell}}\in\mathbb{R}^{C\times P\times P}$ as a convolutional kernel. Applying this kernel to the training image yields the inner-products $\langle{\bm{x}}_{t,\Omega_{\ell}},{\bm{u}}\rangle$ over all spatial locations in a single convolution pass. The quadratic distance is then obtained as $\|{\bm{x}}_{t,\Omega_{\ell}}\|^{2}-2\sqrt{\bar{\alpha}_{t}}\langle{\bm{x}}_{t,\Omega_{\ell}},{\bm{u}}\rangle+\bar{\alpha}_{t}\|{\bm{u}}\|^{2}$.

This approach leverages GPU-optimized convolutions to reduce memory usage. We further parallelize across the $L^{2}$ test patches by batching $B$ patches into a convolutional kernel with $B$ output channels. This yields peak memory $\mathcal{O}(BNL^{2})$, which does not explicitly scale with patch size $P$, thereby avoiding the prohibitive $P^{2}$ factor of naive unfolding and enabling scalable attribution on large datasets.

### 3.5 Discussions

Data attribution is often regarded as a lens for understanding model behavior, quantifying the causal influence of individual training examples on model predictions through the learning process. From this perspective, it might seem paradoxical to speak of attribution without specifying a model: if no model is given, what exactly is being attributed?

Our work invites a broader interpretation. In a narrow sense, our nonparametric method can be understood as a principled guess of how training data might have influenced a generated sample, assuming it was produced by some model trained on the same data. Since our method does not access model parameters, it produces the same attribution for all possible models trained on the dataset. At first glance this may appear counterintuitive, because different models can generalize in different ways and might be expected to yield different attribution patterns. However, our empirical results show that the attribution scores remain consistent across a variety of architectures and training regimes. This suggests that there exists a shared, model-agnostic structure to generalization, which reflects intrinsic relationships between training data and outputs, independent of any specific model.

Seen from an even broader perspective, this idea resonates with how humans attribute provenance. People are often able to determine which training images most likely inspired a generated image even without any knowledge of the mechanism that produced it. This form of attribution is grounded in *perceived similarity* rather than *parametric causality*, and it applies whether the image was produced by a neural network, a human artist, or some natural process. In this light, nonparametric data attribution can be viewed as an attempt to formalize this intuitive, model-independent notion of influence: some training examples are simply more responsible for a given generation than others.

## 4 Experiments

We evaluate our method, *Nonparametric Diffusion Attribution* (NDA), against both nonparametric and gradient-based attribution approaches using two complementary protocols: the linear datamodeling score (LDS), which quantifies alignment with ground-truth influence, and counterfactual evaluation, which assesses the effect of removing influential training data on generated outputs. Our results and ablations show that NDA achieves strong attribution performance without accessing model parameters or gradients, while qualitative visualizations highlight its spatial interpretability and visual consistency.(^3^33Code is available at [https://github.com/sail-sg/NDA](https://github.com/sail-sg/NDA).)

### 4.1 Experimental setup

Datasets.
We conduct experiments on CIFAR-10 (Krizhevsky, 2009) and CelebA (Liu et al., 2015). For efficient ablation studies, we follow Zheng et al. (2024) and construct a CIFAR-2 subset consisting of $5{,}000$ images from CIFAR-10. Additional dataset details are provided in Appendix [B.1](https://arxiv.org/html/2510.14269v1#A2.SS1).

Target models.
For each dataset, we train a diffusion model to serve as the target model for data attribution. On CIFAR-2 and CIFAR-10, we adopt the original DDPM implementation (Ho et al., 2020) to train an unconditional diffusion model with a U-Net backbone containing $35.7$M parameters. For CelebA, we use the same implementation but with a modified architecture of $118.8$M parameters to accommodate the $64{\times}64$ resolution. The number of diffusion steps is fixed to $T{=}1{,}000$. Further training details are provided in Appendix [B.2](https://arxiv.org/html/2510.14269v1#A2.SS2). Note that gradient-based attribution methods directly use the target model parameters, whereas our NDA does not access model parameters or architectures.

LDS evaluation.
Following Zheng et al. (2024), we sample $M{=}64$ random subsets of the training set $\mathbb{S}$, each containing $50\%$ of the samples. For each subset $\mathbb{S}_{m}$, we train three models with different random seeds and average their $\mathcal{L}_{\text{Simple}}$ losses as the model output $\bm{\mathcal{F}}({\bm{x}};\theta^{*}(\mathbb{S}_{m}))$ for a test input ${\bm{x}}$. Additional implementation details are provided in Appendix [B.3](https://arxiv.org/html/2510.14269v1#A2.SS3). To ensure a fair comparison, we use the same $1{,}000$-image held-out validation set and $1{,}000$-image generation set as Zheng et al. (2024).

NDA setup.
For patch-wise influence, we use patch sizes $P\in[3,21]$ for different timesteps $t\in\mathcal{T}$ and resolutions, as determined by ablation studies in Sec. [4.5](https://arxiv.org/html/2510.14269v1#S4.SS5). To compute low-resolution patch-wise influence, we apply an average-pooling downsampling operator $\text{D}(\cdot)$ with a window size of $2$, reducing patch resolution by half. In all experiments, we select the top-$k$ most influential patches per training image with $k{=}100$ to obtain image-level attribution scores. Due to low signal-to-noise ratios at large timesteps, we restrict the set of timesteps to $\mathcal{T}{=}\{100,200,300,400,500\}$ in our experiments.

**Table 1: LDS (%) of different attribution methods on CIFAR-2, CIFAR-10, and CelebA.**
| Method | CIFAR2 | CIFAR10 | CelebA |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Validation | Generation | Validation | Generation | Validation | Generation |  |
| Without Model Access |  |  |  |  |  |  |
| Raw pixel (dot prod.) | 7.77$\pm$0.57 | 4.89$\pm$0.58 | 2.50$\pm$0.42 | 2.25$\pm$0.39 | 5.58$\pm$0.73 | 4.94$\pm$1.58 |
| Raw pixel (cosine) | 7.87$\pm$0.57 | 5.44$\pm$0.57 | 2.71$\pm$0.41 | 2.61$\pm$0.38 | 6.16$\pm$0.75 | 4.38$\pm$1.63 |
| CLIP similarity (dot prod.) | 6.51$\pm$1.06 | 3.00$\pm$0.95 | 2.39$\pm$0.41 | 1.11$\pm$0.47 | 8.87$\pm$1.14 | 2.51$\pm$1.13 |
| CLIP similarity (cosine) | 8.54$\pm$1.01 | 4.01$\pm$0.85 | 3.39$\pm$0.38 | 1.69$\pm$0.49 | 10.92$\pm$0.87 | 3.03$\pm$1.13 |
| NDA (Ours) | 24.88$\pm$0.42 | 15.91$\pm$0.49 | 11.81$\pm$0.30 | 7.41$\pm$0.45 | 16.89$\pm$0.59 | 13.92$\pm$0.68 |
| Using Model Gradients |  |  |  |  |  |  |
| Gradient (dot prod.) | 5.14$\pm$0.60 | 2.80$\pm$0.55 | 0.79$\pm$0.43 | 0.74$\pm$0.45 | 3.82$\pm$0.50 | 3.83$\pm$1.06 |
| Gradient (cosine) | 5.08$\pm$0.59 | 2.78$\pm$0.54 | 0.66$\pm$0.43 | 0.58$\pm$0.42 | 3.65$\pm$0.52 | 3.86$\pm$0.96 |
| TracInCP | 6.26$\pm$0.84 | 3.76$\pm$0.61 | 0.98$\pm$0.44 | 0.96$\pm$0.40 | 5.14$\pm$0.75 | 5.18$\pm$1.05 |
| GAS | 5.78$\pm$0.82 | 3.34$\pm$0.56 | 0.89$\pm$0.48 | 0.90$\pm$0.41 | 5.44$\pm$0.68 | 4.69$\pm$0.97 |
| TRAK | 11.42$\pm$0.49 | 5.78$\pm$0.48 | 2.93$\pm$0.46 | 2.20$\pm$0.38 | 11.28$\pm$0.47 | 7.02$\pm$0.89 |
| D-TRAK | 26.79$\pm$0.33 | 18.82$\pm$0.43 | 14.69$\pm$0.46 | 11.05$\pm$0.43 | 22.83$\pm$0.51 | 16.84$\pm$0.54 |

Figure: Figure 2: Counterfactual evaluation of $\ell_{2}$ distance (Left) and CLIP similarity (Right) between original and regenerated images on CIFAR-2 and CelebA after removing the most influential training samples identified by different attribution methods and retraining the model.
Refer to caption: x2.png

### 4.2 Main results: evaluating LDS for attribution methods

We compare NDA against a range of attribution baselines; detailed descriptions are provided in Appendix [B.4](https://arxiv.org/html/2510.14269v1#A2.SS4). Our primary comparison focuses on representative approaches that do not require access to model parameters, with gradient-based methods included as references. Table [1](https://arxiv.org/html/2510.14269v1#S4.T1) reports the results on CIFAR-2, CIFAR-10, and CelebA.

Compared to the strongest baseline based on CLIP cosine similarity, NDA achieves consistent and substantial gains across both validation and generation sets. On CIFAR-2, NDA improves over CLIP by $+16.32$ (validation) and $+11.90$ (generation); on CIFAR-10, by $+8.42$ and $+5.72$; and on CelebA, by $+5.97$ and $+10.89$, respectively.
When compared to strong gradient-based methods tailored for diffusion models, such as D-TRAK, NDA substantially closes the gap while accessing no model information. For example, on CelebA, the gap is reduced to $5.94$/$2.92$ (validation/generation), much smaller than the CLIP gap of $11.91$/$13.81$.
Overall, NDA approaches the performance of strong parametric baselines while markedly outperforming CLIP similarity on LDS.

### 4.3 Counterfactual evaluation

To evaluate the faithfulness and practical effectiveness of NDA, we conduct a counterfactual influence experiment on CIFAR-2 and CelebA.
For each generated test image, we first identify the top-$1{,}000$ most positively influential training samples according to each attribution method, remove these samples from the training set, and retrain the diffusion model from scratch. The test image is then regenerated using the retrained models under the same random seed, and the impact of removal is quantified using pixel-wise $\ell_{2}$ distance and CLIP cosine similarity. We repeat this procedure for $60$ randomly generated test images and report the average results.

We compare NDA against three baselines: Random removal, CLIP similarity (cosine), and D-TRAK. As shown in Figure [2](https://arxiv.org/html/2510.14269v1#S4.F2), NDA achieves median $\ell_{2}$ distances of $8.43$ and $14.38$ on CIFAR-2 and CelebA, respectively, outperforming CLIP ($6.68,10.30$) and D-TRAK ($7.68,13.18$). For CLIP similarity, NDA attains $0.92$ and $0.94$, approaching D-TRAK ($0.91,0.93$). These results indicate that NDA effectively identifies and removes training samples with strong influence over the generated output, providing a compelling nonparametric alternative to gradient-based attribution. Qualitative visualizations in Figure [3](https://arxiv.org/html/2510.14269v1#S4.F3) further show that the generated images exhibit significant distortions after removing the training samples deemed most influential by NDA.

Figure: Figure 3: Counterfactual visualization on CIFAR-2 (Top) and CelebA (Bottom). Images are compared to those generated by retrained models using the same seed. See Appendix [D.1](https://arxiv.org/html/2510.14269v1#A4.SS1) for more cases.
Refer to caption: fig/counter/cifar2/136/136.png

### 4.4 Spatial interpretability

Since NDA computes attribution by aggregating patch-wise influence, it naturally offers an additional spatial level of interpretability. Intuitively, a training image with a high attribution score must contain patches that strongly align with influential regions of the test image. To visualize this correspondence, we highlight the most influential patches from each of the top attributed training images in Figure [4](https://arxiv.org/html/2510.14269v1#S4.F4).
We observe that top-ranked training images consistently contain local patches that are visually similar to those in the test image, offering a natural explanation for their high attribution scores.

Figure: Figure 4: Spatial interpretability of NDA. The patches with the highest patch-wise influence scores (w.r.t. patches in the generated image) are highlighted in the top attributed training images.
Refer to caption: x6.png

Figure: Figure 5: LDS (%) on the CIFAR-2 validation set across different patch sizes and timesteps at the original resolution. Left: small timesteps ($t{\leq}300$), where moderate patch sizes achieve the highest scores. Middle: medium timesteps ($t{=}400,500$), where larger patches outperform smaller ones due to higher noise. Right: large timesteps, where noise dominates and informative signal is minimal.
Refer to caption: x7.png

### 4.5 Ablation studies

We conduct ablation studies on key hyperparameters of our method. All studies are performed on the validation sets, which are i.i.d. samples drawn from the same distribution as the training sets. The selected hyperparameters are then applied to the generation sets without further tuning to produce the results reported in Sec. [4.2](https://arxiv.org/html/2510.14269v1#S4.SS2).
In all figures where applicable, solid lines show the mean, and shaded regions indicate $\pm 1$ standard deviation.

Patch size selection.
We study the effect of patch size by evaluating LDS performance across $P\in[3,21]$ at different timesteps. To isolate the impact of timestep, we fix $\mathcal{T}$ to contain a single timestep (e.g., $\mathcal{T}{=}\{100\},\dots,\{900\}$) and vary $P$. Figure [5](https://arxiv.org/html/2510.14269v1#S4.F5) shows LDS curves as a function of $P$ for each $\mathcal{T}$ on CIFAR-2 (see Appendix [C](https://arxiv.org/html/2510.14269v1#A3) for CelebA).
We observe that the optimal patch size varies with timestep and generally increases as $t$ grows. At early timesteps ($t{\leq}300$), small to moderate patches ($P{=}5,7,9$) yield the highest LDS scores, suggesting that local patterns dominate when noise levels are relatively low during reverse diffusion. For mid-range timesteps ($400{\leq}t{\leq}500$), larger patches perform better, due to their ability to aggregate more contextual information under higher noise levels. In the high-noise regime ($t{\geq}600$), LDS scores drop significantly across all patch sizes. Based on these results, we adopt a timestep-dependent patch size selection strategy: using smaller patches for early timesteps and progressively larger patches for later timesteps, choosing the $P$ that maximizes LDS for each $t$.

Multiscale influence.
We first find the optimal patch size for the low-resolution patch-wise influence in Eq. ([13](https://arxiv.org/html/2510.14269v1#S3.E13)) using the same procedure as above and observe a similar trend, where early timesteps favor smaller patches. Interestingly, the optimal patch size differs between the original and low-resolution cases (see Appendix [C](https://arxiv.org/html/2510.14269v1#A3) for details).
We then evaluate the proposed multiscale attribution in Eq. ([15](https://arxiv.org/html/2510.14269v1#S3.E15)) by varying the weighting factor $\gamma\in\{0,0.25,0.5,0.75,1\}$ across timesteps. As shown in Figure [8](https://arxiv.org/html/2510.14269v1#S4.F8), combining multiple scales (e.g., $\gamma{=}0.75$) consistently improves LDS, particularly in the low-noise regime, suggesting that multiscale features provide complementary information that enhances attribution. Further improvements may be possible by incorporating more scales, which we leave for future work.

Top-$k$ patch selection.
We examine how the number of top-$k$ patches affects attribution. Figure [8](https://arxiv.org/html/2510.14269v1#S4.F8) shows LDS as a function of $k$ on lower-noise timesteps, which contribute more significantly to the final attribution. We test $k$ ranging from $1$ to $500$ and find that an intermediate value of $k{=}100$ generally achieves optimal performance across timesteps and datasets and generalizes well to generation sets.

Timestep selection.
Since lower-noise timesteps provide stronger attribution signal, we define $\mathcal{T}=\{100,200,\ldots,\tau_{\max}\}$ and study the effect of varying the max aggregation timestep $\tau_{\max}$. At each $t\in\mathcal{T}$ we use the optimal patch sizes and weighting factors identified above. Figure [8](https://arxiv.org/html/2510.14269v1#S4.F8) shows that enlarging $\mathcal{T}$ improves LDS up to mid-range timesteps, after which gains saturate or degrade. On CIFAR-2, performance rises from $t{=}100$ to a peak near $\tau_{\max}{=}400$ before dropping by ${\sim}0.5$ at $\tau_{\max}{\geq}600$; On CelebA, performance saturates near $\tau_{\max}{=}400{\sim}500$ with negligible gains beyond. These results suggest that late (high-noise) timesteps contribute little useful signal and may instead inject noise. We therefore fix $\tau_{\max}{=}500$ for all subsequent experiments, which achieves near-optimal performance while avoiding unnecessary computation and variability.

Figure: Figure 6: LDS (%) on CIFAR-2 across different weighting factor $\gamma$ at varying timesteps.
Refer to caption: x10.png

## 5 Conclusion

We present a nonparametric method for data attribution in diffusion models that measures the influence of training examples via patch-level similarity. Our approach leverages the analytical form of the optimal score function, extends to multiscale representations, and remains efficient through convolution-based operations. It provides spatially interpretable attributions and uncovers intrinsic relationships between training data and generated outputs, independent of any specific model. Experiments on CIFAR-2, CIFAR-10, and CelebA show that our method closely matches gradient-based approaches and outperforms existing nonparametric baselines, demonstrating that model-agnostic, data-driven attribution is a practical tool for understanding generative models.

## References

- Adelson et al. (1984)
Edward H Adelson, Charles H Anderson, James R Bergen, Peter J Burt, and Joan M Ogden.
Pyramid methods in image processing.
*RCA engineer*, 29(6):33–41, 1984.
- Bao et al. (2022)
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang.
Analytic-DPM: an analytic estimate of the optimal reverse variance in diffusion probabilistic models.
In *International Conference on Learning Representations*, 2022.
- Carlini et al. (2023)
Nicolas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace.
Extracting training data from diffusion models.
In *32nd USENIX security symposium (USENIX Security 23)*, pp. 5253–5270, 2023.
- Charpiat et al. (2019)
Guillaume Charpiat, Nicolas Girard, Loris Felardos, and Yuliya Tarabalka.
Input similarity from the neural network perspective.
In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett (eds.), *Advances in Neural Information Processing Systems*, volume 32. Curran Associates, Inc., 2019.
- Feldman & Zhang (2020)
Vitaly Feldman and Chiyuan Zhang.
What neural networks memorize and why: Discovering the long tail via influence estimation.
In *Advances in Neural Information Processing Systems*, 2020.
- Ghorbani & Zou (2019)
Amirata Ghorbani and James Zou.
Data shapley: Equitable valuation of data for machine learning.
In *International conference on machine learning*, pp. 2242–2251. PMLR, 2019.
- Gu et al. (2024)
Xiangming Gu, Chao Du, Tianyu Pang, Chongxuan Li, Min Lin, and Ye Wang.
On memorization in diffusion models, 2024.
- Gu et al. (2025)
Yuxian Gu, Li Dong, Hongning Wang, Yaru Hao, Qingxiu Dong, Furu Wei, and Minlie Huang.
Data selection via optimal control for language models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Hammoudeh & Lowd (2022)
Zayd Hammoudeh and Daniel Lowd.
Identifying a training-set attack’s target using renormalized influence estimation.
In *Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security*, CCS ’22, pp. 1367–1381, New York, NY, USA, 2022. Association for Computing Machinery.
ISBN 9781450394505.
- Ho et al. (2020)
Jonathan Ho, Ajay Jain, and Pieter Abbeel.
Denoising diffusion probabilistic models.
*Advances in neural information processing systems*, 33:6840–6851, 2020.
- Ilyas et al. (2022)
Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.
Datamodels: Predicting predictions from training data.
In *ICML*, 2022.
- Jia et al. (2021)
Ruoxi Jia, Fan Wu, Xuehui Sun, Jiacen Xu, David Dao, Bhavya Kailkhura, Ce Zhang, Bo Li, and Dawn Song.
Scalability vs. utility: Do we have to sacrifice one for the other in data importance quantification?
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)*, 2021.
- Jing et al. (2022)
Bowen Jing, Gabriele Corso, Renato Berlinghieri, and Tommi Jaakkola.
Subspace diffusion generative models.
In *European conference on computer vision*, pp. 274–289. Springer, 2022.
- Kamb & Ganguli (2024)
Mason Kamb and Surya Ganguli.
An analytic theory of creativity in convolutional diffusion models, 2024.
- Khanna et al. (2018)
Rajiv Khanna, Been Kim, Joydeep Ghosh, and Oluwasanmi Koyejo.
Interpreting black box predictions using fisher kernels.
In *International Conference on Artificial Intelligence and Statistics*, 2018.
- Kingma & Gao (2023)
Diederik P Kingma and Ruiqi Gao.
Understanding diffusion objectives as the ELBO with simple data augmentation.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
- Koh & Liang (2017)
Pang Wei Koh and Percy Liang.
Understanding black-box predictions via influence functions.
In *Proceedings of the 34th International Conference on Machine Learning*, 2017.
- Krizhevsky (2009)
Alex Krizhevsky.
Learning multiple layers of features from tiny images.
2009.
- Lin et al. (2025)
Jinxu Lin, Linwei Tao, Minjing Dong, and Chang Xu.
Diffusion attribution score: Evaluating training data influence in diffusion model.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Lin et al. (2017)
Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie.
Feature pyramid networks for object detection.
In *Proceedings of the IEEE conference on computer vision and pattern recognition*, pp. 2117–2125, 2017.
- Liu et al. (2021)
Zhuoming Liu, Hao Ding, Huaping Zhong, Weijia Li, Jifeng Dai, and Conghui He.
Influence selection for active learning.
In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, pp. 9274–9283, October 2021.
- Liu et al. (2015)
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang.
Deep learning face attributes in the wild.
In *Proceedings of the IEEE International Conference on Computer Vision (ICCV)*, 2015.
- Loshchilov & Hutter (2019)
Ilya Loshchilov and Frank Hutter.
Decoupled weight decay regularization, 2019.
- Meng et al. (2022)
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon.
SDEdit: Guided image synthesis and editing with stochastic differential equations.
In *International Conference on Learning Representations*, 2022.
- Mlodozeniec et al. (2025)
Bruno Kacper Mlodozeniec, Runa Eschenhagen, Juhan Bae, Alexander Immer, David Krueger, and Richard E. Turner.
Influence functions for scalable data attribution in diffusion models.
In *The Thirteenth International Conference on Learning Representations*, 2025.
- Nohyun et al. (2023)
Ki Nohyun, Hoyong Choi, and Hye Won Chung.
Data valuation without training of a model.
In *The Eleventh International Conference on Learning Representations*, 2023.
- Park et al. (2023a)
Sung Min Park, Kristian Georgiev, Andrew Ilyas, Guillaume Leclerc, and Aleksander Madry.
TRAK: Attributing model behavior at scale.
In *Proceedings of the 40th International Conference on Machine Learning*. PMLR, 2023a.
- Park et al. (2023b)
Yong-Hyun Park, Mingi Kwon, Jaewoong Choi, Junghyo Jo, and Youngjung Uh.
Understanding the latent space of diffusion models through the lens of riemannian geometry.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023b.
- Pruthi et al. (2020)
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan.
Estimating training data influence by tracing gradient descent.
In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), *Advances in Neural Information Processing Systems*, volume 33, pp. 19920–19930. Curran Associates, Inc., 2020.
- Radford et al. (2021)
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.
Learning transferable visual models from natural language supervision.
In *International conference on machine learning*, pp. 8748–8763. PmLR, 2021.
- Ramesh et al. (2022)
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.
Hierarchical text-conditional image generation with clip latents.
*arXiv preprint arXiv:2204.06125*, 2022.
- Rombach et al. (2022)
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.
High-resolution image synthesis with latent diffusion models.
In *Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition*, pp. 10684–10695, 2022.
- Saveri & Butterick (2023)
Joseph Saveri and Matthew Butterick.
Image generator litigation.
[https://imagegeneratorlitigation.com/](https://imagegeneratorlitigation.com/), 2023.
Accessed: 2024-07-06.
- Somepalli et al. (2023)
Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein.
Understanding and mitigating copying in diffusion models.
In *Thirty-seventh Conference on Neural Information Processing Systems*, 2023.
- Song et al. (2021)
Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.
Score-based generative modeling through stochastic differential equations.
In *International Conference on Learning Representations*, 2021.
- Sui et al. (2021)
Yi Sui, Ga Wu, and Scott Sanner.
Representer point selection via local jacobian expansion for post-hoc classifier explanation of deep neural networks and ensemble models.
In *Advances in Neural Information Processing Systems*, 2021.
- Zhao et al. (2024)
Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Ngai man Cheung, and Min Lin.
A recipe for watermarking diffusion models, 2024.
- Zheng et al. (2024)
Xiaosen Zheng, Tianyu Pang, Chao Du, Jing Jiang, and Min Lin.
Intriguing properties of data attribution on diffusion models.
In *The Twelfth International Conference on Learning Representations*, 2024.

## Appendix A Related works

### A.1 Data attribution

Training data plays a critical role in shaping the behavior of machine learning models. Data attribution aims to measure the contribution of individual training samples to model predictions. Existing methods can be broadly categorized into two classes: retraining-based and retraining-free approaches.

Retraining-based methods, such as empirical influence functions (Feldman & Zhang, 2020), Shapley value estimators (Jia et al., 2021), and datamodels (Ilyas et al., 2022), provide high-fidelity attributions by measuring the effect of removing or modifying each training example. However, these methods typically require retraining the model tens of thousands of times on different data subsets to achieve reliable results, making them computationally infeasible for large-scale settings.
Retraining-free methods aim to approximate influence scores without additional retraining, offering a more scalable alternative. These approaches fall into two main categories. The first comprises gradient-based methods without kernelization, which rely solely on first-order gradient signals and avoid computing second-order derivatives. Representative approaches include Gradient (Charpiat et al., 2019), TracInCP (Pruthi et al., 2020) and GAS (Hammoudeh & Lowd, 2022).
The second category includes gradient-based methods with kernelization, which incorporate curvature information by constructing kernels, typically using the inverse of the Hessian matrix.
However, Hessian inversion is often numerically unstable.
To mitigate this, recent work approximates the Hessian with the Fisher information matrix (FIM). For instance, TRAK (Park et al., 2023a) introduces a kernel-based approximation that efficiently estimates influence scores via random projections, which is both accurate and computationally tractable for large-scale models. D-TRAK (Zheng et al., 2024) further improves attribution performance by modifying the output function and training loss in TRAK, resulting in more effective LDS scores. Based on empirical analysis, D-TRAK recommends a specific configuration that achieves the best overall performance.

### A.2 Nonparametric data attribution methods

While most data attribution methods rely on parametric models and require gradient computations or model retraining, an alternative line of research explores nonparametric approaches that operate directly on training data without fitting parametric functions. These are typically similarity-based approaches, which estimate the influence of a training sample based on its similarity to the target using a predefined metric.
For example, one can leverage pretrained vision-language embeddings such as CLIP (Radford et al., 2021) features to measure high-level semantic similarity of images (Zheng et al., 2024; Mlodozeniec et al., 2025), or compare raw pixel values directly. However, these methods overlook the internal dynamics of the generative model. As a result, effective data attribution for diffusion models in settings without model access remains an open and challenging problem.

## Appendix B Implementation details

### B.1 Datasets

CIFAR-10 ($32{\times}32$). CIFAR-10 contains $50{,}000$ training and $10{,}000$ test images. For LDS evaluation, we randomly sample $1{,}000$ validation images from the test set. To reduce computational overhead, we additionally construct CIFAR-2 ($32{\times}32$), a subset consisting of $5{,}000$ training images randomly selected from the “automobile” and “horse” classes and $1{,}000$ validation images from the corresponding classes in the test set.

CelebA ($64{\times}64$). We sample $5{,}000$ training and $1{,}000$ validation images from the official CelebA splits (Liu et al., 2015). Following the preprocessing steps outlined by Song et al. (2021), we center-crop all images to $140{\times}140$ pixels and resize them to $64{\times}64$.

### B.2 Models

CIFAR-10 and CIFAR-2. We adopt the original unconditional DDPM implementation (Ho et al., 2020), using a U-Net backbone with approximately $35.7$M parameters. Models are trained for $200$ epochs with a batch size of $128$ and cosine annealing learning rate schedule. We use AdamW (Loshchilov & Hutter, 2019) with weight decay $10^{-6}$ and a dropout rate of $0.1$. To enhance robustness, random horizontal flips are applied for data augmentation.

CelebA. We use the same DDPM framework but scale the U-Net to $118.8$M parameters to accommodate $64{\times}64$ inputs. All other hyperparameters, including the variance schedule, optimizer configurations, and training procedure, follow the CIFAR setup.

### B.3 LDS evaluation setup

For LDS evaluation, we sample $M{=}64$ random subsets $\mathbb{S}_{m}$ from the training set, each containing $50\%$ of the training data. For each subset $\mathbb{S}_{m}$, three models are trained using different random seeds to reduce evaluation variance. For each input (sample of interest) ${\bm{x}}$, the linear datamodeling score is computed as the Spearman rank correlation between model outputs and attribution scores, as described in Eq. ([8](https://arxiv.org/html/2510.14269v1#S2.E8)). Specifically, we compute the model output $\bm{\mathcal{F}}=\mathcal{L}_{\text{Simple}}({\bm{x}},\theta)$ in Eq. ([3](https://arxiv.org/html/2510.14269v1#S2.E3)) over $1{,}000$ timesteps uniformly spaced in the interval $[1,T]$. At each timestep, we approximate the expectation $\mathbb{E}_{\bm{\epsilon}}$ by sampling three standard Gaussian noise $\bm{\epsilon}\sim\mathcal{N}(0,\mathbf{I})$. Final LDS performance is reported by averaging scores over both validation and generation samples.

### B.4 Baselines

We compare NDA against two major categories of attribution methods: (1) nonparametric similarity-based methods, which do not rely on model parameters and instead operate directly on image features and representations, and (2) post-hoc retraining-free methods, which leverage gradient-based representations of a trained model to estimate the attribution score without retraining. Within the first category, we evaluate two similarity-based methods: Raw Pixel and CLIP (Radford et al., 2021). Within the second category, we further divide the methods into *(*gradient-based methods without kernels) and *(*gradient-based methods with kernels). The first group includes Gradient (Charpiat et al., 2019), TracInCP (Pruthi et al., 2020), and GAS (Hammoudeh & Lowd, 2022). The second group includes representative kernel-based methods such as TRAK (Park et al., 2023a) and D-TRAK (Zheng et al., 2024).

Raw Pixel. A naive similarity-based method that directly treats the raw image as the feature representation. Attribution scores are computed by measuring similarity (e.g., dot product or cosine similarity) between the query sample and each training sample.

CLIP Similarity. Each target and training sample is embedded using CLIP (Radford et al., 2021). Attribution is then computed as the cosine similarity (or dot product of normalized embeddings) between pairs of embeddings. This provides a simple estimate of training data influence, independent of the target diffusion model.

Gradient. A gradient-based influence estimator introduced by Charpiat et al. (2019). The attribution score for each training sample is computed as the dot product or cosine similarity between its gradient (w.r.t. model parameters) and that of the test sample:

$$ $\displaystyle\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})$ $\displaystyle=\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{x}};\theta^{*})^{\top}.\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{z}}^{(n)};\theta^{*})\textrm{,}$ $\displaystyle\textrm{or}\quad\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})$ $\displaystyle=\frac{\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{x}};\theta^{*})^{\top}.\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{z}}^{(n)};\theta^{*})}{\left\|\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{x}};\theta^{*})\right\|\left\|\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{z}}^{(n)};\theta^{*})\right\|}\textrm{,}$ $$

where $\mathcal{P}$ is a Gaussian random projection matrix that projects the gradient into low-dimensional subspace.

TracInCP. We adopt the TracInCP estimator introduced by Pruthi et al. (2020), formulated as
$\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})=\frac{1}{C}\sum_{c=1}^{C}\left(\mathcal{P}_{c}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}\left({\bm{x}};\theta^{c}\right)\right)^{\top}\cdot\left(\mathcal{P}_{c}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}\left({\bm{z}}^{(n)};\theta^{c}\right)\right)$,
where $C$ denotes the number of uniformly sampled checkpoints along the training trajectory, and $\theta^{c}$ denotes parameters at the $c$-th checkpoint. We use four checkpoints for each experiment. For example, we select checkpoints at epochs $\{50,100,150,200\}$ on CIFAR-2.

GAS. A “renormalized” version of TracInCP that replaces dot products with cosine similarity (Hammoudeh & Lowd, 2022).

TRAK. TRAK (Park et al., 2023a) can be extended to diffusion models following Zheng et al. (2024).
The retraining-free TRAK can be implemented as:

$$ $\displaystyle\Phi_{\text{TRAK}}=\left[\phi\left({\bm{x}}^{1}\right),\cdots,\phi\left({\bm{x}}^{N}\right)\right]^{\top}\textrm{, where}\ \phi({\bm{x}})=\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{x}};\theta^{*});$ $\displaystyle\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})=\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{x}};\theta^{*})^{\top}\cdot\left(\Phi_{\text{TRAK}}{}^{\top}\Phi_{\text{TRAK}}+\lambda I\right)^{-1}\cdot\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Simple}}({\bm{z}}^{(n)};\theta^{*})\textrm{,}$ $$

where the term $\lambda I$ is included for numerical stability and regularization effect.

D-TRAK (Zheng et al., 2024). Similar to TRAK but uses the squared-error objective:

$$ $\displaystyle\Phi_{\text{D-TRAK}}=\left[\phi\left({\bm{x}}^{1}\right),\cdots,\phi\left({\bm{x}}^{N}\right)\right]^{\top}\textrm{, where}\ \phi({\bm{x}})=\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Square}}({\bm{x}};\theta^{*});$ $\displaystyle\tau({\bm{x}},{\bm{z}}^{(n)};\mathbb{S})=\left(\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Square}}({\bm{x}};\theta^{*})\right)^{\top}\cdot\left(\Phi_{\text{D-TRAK}}{}^{\top}\Phi_{\text{D-TRAK}}+\lambda I\right)^{-1}\cdot\mathcal{P}^{\top}\nabla_{\theta}\mathcal{L}_{\text{Square}}({\bm{z}}^{(n)};\theta^{*})\textrm{,}$ $$

with $\mathcal{L}_{\text{Square}}=\|\bm{\epsilon}_{\theta}({\bm{x}}_{t},t)\|^{2}$.

## Appendix C Additional ablation studies

We provide additional ablation studies on patch size selection and multiscale influence.

Figure: (a)
Refer to caption: x21.png

Figure: (a)
Refer to caption: x24.png

Figure: (a)
Refer to caption: x27.png

Patch size selection. For completeness, we also study the effect of patch size by evaluating LDS across $P\in[3,21]$ at different timesteps on CelebA. To isolate timestep effects, we fix $\mathcal{T}$ to a single $t$ and vary $P$. Figure [9](https://arxiv.org/html/2510.14269v1#A3.F9) shows LDS as a function of $P$ for each $t$. The trend mirrors CIFAR-2: the optimal patch size generally grows with $t$. At eraly timesteps ($t{\leq}300$), small to moderate patches (e.g., $P{=}5,7,9$) yield the highest scores, indicating local patterns dominate when noise is low. At mid-range timesteps ($400{\leq}t{\leq}500$), larger patches perform better (peaks shift to $P{=}11,13$), likely due to the need for more contextual information under higher noise. In the high-noise regime ($t{\geq}600$), LDS drops and curves flatten across all patch sizes, suggesting informative signal is limited.

Multiscale influence. We further compare the optimal patch size between the original and the low-resolution settings on CIFAR-2. As shown in Figure [5](https://arxiv.org/html/2510.14269v1#S4.F5) and Figure [10](https://arxiv.org/html/2510.14269v1#A3.F10), at the early timestep $t{=}100$, the optima differ notably: the original resolution peaks at $P{=}5$, whereas the low-resolution curve peaks at $P{=}8$, indicating that coarser inputs benefit from slightly larger spatial context. By contrast, at $t{=}200$ and $t{=}300$, the two resolutions behave very similarly: on the original scale it peaks at $P{=}7$ and $P{=}9$ while low-resolution peaks at $P{=}8$ and $P{=}10$, with near-overlapping curves around their maxima. At mid timesteps ($t{=}400,500$), both resolution favors large patch size with $P{=}21$. These results show that downscaling mainly shifts optimal to large patch size $P$ at small timestep, while mid/large timesteps show qualitatively similar trends across resolutions.

## Appendix D Visualization

We provide additional visualizations, including counterfactual examples and proponent-opponent analyses.

### D.1 Counterfactual visulization

We include more counterfactual visualizations in Figures [12](https://arxiv.org/html/2510.14269v1#A4.F12) and [13](https://arxiv.org/html/2510.14269v1#A4.F13). As shown, NDA identifies training examples whose removal leads to marked changes in the output of a model retrained with the same seed. In Figure [12](https://arxiv.org/html/2510.14269v1#A4.F12) (right column, third row), models retrained after removing images selected by Random and CLIP still synthesize an “automobile”. In contrast, removing images selected by NDA or D-TRAK yields an image that resembles a mixture of “automobile” and “horse”. Notably, the image after NDA’s removal shows clearer horse morphology (e.g., head and leg outlines) than that after D-TRAK. These results suggest that NDA is an effective nonparametric approach for identifying training images with strong influence on a given target.

### D.2 Proponents and opponents visulization

Following Pruthi et al. (2020), we refer to training examples with positive influence scores as *proponents* and those with negative scores as *opponents*. For each target, we retrieve and visualize the top-$5$ proponents and the top-$3$ opponents. Qualitative results on CIFAR-2, CIFAR-10, and CelebA are shown in Figures [14](https://arxiv.org/html/2510.14269v1#A4.F14), [15](https://arxiv.org/html/2510.14269v1#A4.F15), and [16](https://arxiv.org/html/2510.14269v1#A4.F16). Across datasets, NDA consistently retrieves proponents that are visually more similar to the target than those selected by CLIP, and its opponents are correspondingly more dissimilar.

Figure: Figure 12: Counterfactual visualization on CIFAR-2 dataset. We compare samples generated by retrained models with different attribution methods using the same seed.
Refer to caption: fig/counter/cifar2/660/660.png

Figure: Figure 13: Counterfactual visualization on CelebA dataset. We compare samples generated by retrained models with different attribution methods using the same seed.
Refer to caption: fig/counter/celebA/862/862.png

Figure: Figure 14: Proponents and opponents visualization on CIFAR-2 using CLIP, NDA and D-TRAK (using averaged timesteps). For each target sample, the 5 most positively influential training samples (Left) and the 3 most negatively influential samples (Right) are shown.
Refer to caption: fig/vis/cifar2_gen_171.png

Figure: Figure 15: Proponents and opponents visualization on CIFAR-10 using CLIP, NDA and D-TRAK (using averaged timesteps). For each target sample, the 5 most positively influential training samples (Left) and the 3 most negatively influential samples (Right) are shown.
Refer to caption: fig/vis/cifar10_gen_56.png

Figure: Figure 16: Proponents and opponents visualization on CelebA using CLIP, NDA and D-TRAK (using averaged timesteps). For each target sample, the 5 most positively influential training samples (Left) and the 3 most negatively influential samples (Right) are shown.
Refer to caption: fig/vis/celebA_gen_138.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 4
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. Novel Non-Parametric Framework: The paper makes a valuable contribution by framing the attribution problem from a non-parametric, data-driven perspective.
2. Theoretical Motivation: The method is motivated by reinterpreting the weighting term (W_t) from the analytical form of the optimal score function (Eq. 5-6) as a measure of influence. This provides a strong theoretical justification for why this specific form of similarity should be effective.
3. Strong Empirical Performance (vs. Chosen Baselines): On the LDS and counterfactual retraining metrics, the proposed NDA method is shown to be highly effective. It substantially outperforms other full-image non-parametric baselines (Raw Pixel, CLIP).
summary: This paper introduces Nonparametric Diffusion Attribution (NDA), a novel, model-agnostic method for data attribution in diffusion models. The key insight is to reinterpret the weighting term from the analytical, optimal score function of a diffusion process as an influence score. By grounding this in a local score function, the method derives a patch-wise similarity metric that is computationally efficient and requires no access to model parameters. The method demonstrates strong empirical performance on the Linear Datamodeling Score (LDS) when compared to several baseline methods.
weaknesses: 1. Unsupported Claim of Model-Independence: The paper makes a strong, highly valuable claim that "our empirical results show that the attribution scores remain consistent across a variety of architectures and training regimes" (Page 6, Line 281). However, there are no empirical results presented in the manuscript (including the appendix) to substantiate this.

2. Missing "Apples-to-Apples" Baselines: The paper's core method (NDA) is patch-based. However, the non-parametric baselines it is compared against ("Raw Pixel" and "CLIP Similarity") are full-image-based. This is an "apples-to-oranges" comparison that inflates the perceived contribution of NDA. The strong performance seen might not come from the specific "score-function" formulation but simply from the act of using patches, which is a known technique. Some "naive patch-based" baselines include:
    * Patch-wise Raw Pixel: A method that computes L2 similarity on top-k raw patches and aggregates them.
    * Patch-wise Feature Similarity: A method that uses a standard feature extractor (e.g., CLIP, a pretrained CNN, or DINOv2) to embed patches and then computes a top-k aggregated similarity.

3. Failure to Disentangle Causal Factors: The paper's novelty rests on two assumptions: theoretical motivation via optimal score function and localization. The current experiments do not disentangle these. It is unclear whether the method's good performance comes from the "locality" assumption (Eq. 9) or the "optimal score" formula itself (Eq. 6). The authors should provide an ablation study for an "Image-wise NDA" baseline. This method would be derived directly from the full-image optimal score function (Eq. 5) and would use the full-image weighting term $W_t$ (Eq. 6) as the influence score, with no patch decomposition.

---

confidence: 4
contribution: 3
initial_rating: 4
presentation: 3
questions: 1. Please explain what are the requirements for performing the attribution?
2. How the ground-truth was generated?
soundness: 3
strengths: 1. Proposing an efficient query-only/nonparametric, data attribution approach for diffusion models.
2. The method is based on analytical properties of the diffusion score function.
3. The patch-level mapping can be used for understanding the relationship between training and generated images.
4. Experiments show consistent improvements over the baselines.
summary: The paper proposes a method for quantifying the relevancy of individual training samples to the outputs generated by diffusion models without requiring model access or gradients. The method performs patch-level similarity comparisons between generated and training images to quantify the attribution. Evaluation results using CIFAR-2, CIFAR-10, and CelebA datasets, show that NDA performance is close to gradient-based approaches and outperforming existing nonparametric baselines.
weaknesses: 1. Figure 1 – The proposed method is not clear from the figure. I suggest improving the figure.
2. No clear description of what are the requirements for performing the attribution
3. Not clear how the ground-truth was generated?
4. Experiments should be extended to larger and more complex datasets (e.g., ImageNet) to prove scalability and generalization.
5. Comparisons with other strong nonparametric or model-free baselines is missing, for example: 

[a] CustomMark: Customization of Diffusion Models for Proactive Attribution
[b] Montrage: Monitoring training for attribution of generative diffusion models

6. Method's runtime analysis is missing.

---

confidence: 2
contribution: 2
initial_rating: 6
presentation: 3
questions: As the authors also point out, patch-level attribution seems to be interpretable. Do you think that for diffusion model kind image generation models, patch-level attribution is just fundamentally a better approach compared to image-level attribution, as it provides more fine-grained signals?
soundness: 3
strengths: The paper is well motivated and well presented. The proposed approach is solid, although more explanations will be helpful (see weakness). In the evaluated scenarios, the methods seem to perform well.
summary: The paper introduces a nonparametric approach to data attribution for diffusion models motivated by scalability. It computes patch-level influence scores derived from the optimal score-function formulation. Effectiveness is evaluated via Linear Datamodeling Score (LDS) and counterfactual removal-and-retrain experiments.
weaknesses: 1. While the method is motivated by a score-function view, the estimator still just behaves like a complicated version of a similarity measure against training patches. Additional ablations/theory to isolate what truly helps will be very appreciated. For example, do naive patch-level similarities across timesteps already recover most of the effect? 
2. The paper is motivated by scalability. However, it seems that the paper still only evaluates on smaller models that generate low-res images.

---

confidence: 3
contribution: 2
initial_rating: 2
presentation: 2
questions: (number references to the Weekness section)

On (2). It would be valuable to report wall-clock vs. dataset size, have some theoretical scaling analysis on compute for data size/resolution, and give scaling guidances. 

On (3). Are the multiscale tricks and hyperparameters stable at higher resolutions or for more scales?—an experiment at ≥256×256 would be very valuable. 

On (4). How does the algorithm behave with more complex distributions (e.g. Imagenet)?
soundness: 2
strengths: The paper seems practical and easy enough to follow in situations where no model access is available. The authors also report spatially interpretable results, and the method is shown to outperform significantly the tested prior baselines.
summary: The authors propose Nonparametric Diffusion Attribution (NDA), a gradient-free data-attribution method for diffusion models that scores patch-level influence using the analytic optimal score, and is optimized with convolution-based operations. They target black-box/proprietary settings with no access to a model's parameters, and show attributions that outperform prior baselines and approach gradient-based methods on CIFAR-10 and CelebA.
weaknesses: 1. A few core assumptions and scope limits should be made explicit. Most derivations rely on naive score-matching or score identities for standard diffusion losses, so general statements about “model-agnostic” influence seem like an overreach.  For example, if we inject an exact-reconstruction penalty in the loss, the results may no longer hold.

2. Compute requirements seem concerning. The convolutional trick proposed by the authors helps with peak memory blow-up, but the per-image convolution over all training patches and multiple timesteps/scales won't scale very well. 

3. The authors test their work in relatively low-resolution settings (up to 64×64).

4. The visual distribution tested a relatively simple, with generally smooth contours and simple patches.


Decision: Reject