Data Attribution for Text-to-Image Models by Unlearning Synthesized Images
Authors: Sheng-Yu Wang
Aaron Hertzmann
Alexei A Efros
Jun-Yan Zhu
Richard Zhang
Venue: NeurIPS 2024 poster (2024)
Keywords: Data Attribution
Influence Estimation
Text-to-Image models
Machine Unlearning
Primary area: generative_models
TL;DR: We identify influential training images that make a synthesized image possible, using our proposed attribution method by unlearning a synthesized image.

Abstract: The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image. Influence is defined such that, for a given output, if a model is retrained from scratch without the most influential images, the model would fail to reproduce the same output. Unfortunately, directly searching for these influential images is computationally infeasible, since it would require repeatedly retraining models from scratch. In our work, we propose an efficient data attribution method by simulating unlearning the synthesized image. We achieve this by increasing the training loss on the output image, without catastrophic forgetting of other, unrelated concepts. We then identify training images with significant loss deviations after the unlearning process and label these as influential. We evaluate our method with a computationally intensive but "gold-standard" retraining from scratch and demonstrate our method's advantages over previous methods.

Data Attribution for Text-to-lmage Models by Unlearning Synthesized Images
Sheng-Yu Yu Wang' Aaron Hertzmann Carnegie Mellon University
Alexei A. Efros
Jun-Yan Zhu Richard Zhang?
Adobe Research
UC Berkeley
Abstract
The goal of data attribution for text-to-image models is to identify the training images that most influence the generation of a new image. Influence is defined such that, for given output, if model is retrained from scratch without the most influential images, the model would fail to reproduce the same output. Unfortu- nately, directly searching for these influential images is computationally infeasible, since it would require repeatedly retraining models from scratch. In our work, we propose an efficient data attribution method by simulating unlearning the synthe- sized image. We achieve this by increasing the training loss on the output image, without catastrophic forgetting of other, unrelated concepts. We then identify train- ing images with significant loss deviations after the unlearning process and label these as influential. We evaluate our method with a computationally intensive but "gold-stane standard" retraining from scratch and demonstrate our method's s advantages over previous methods.
Introduction
Data attribution for text-to-image generation aims identify which training images "influenced" given output. The black-box nature of modern image generation models [ 3
4
5
6 7 8).
together with the enormous datasets required [9], makes extremely challenging understand the contributions of individual training images. Although generative models can, at times, replicate training data
they typically create samples distinct from any specific training image.
We believe that a counterfactual definition of "influence" best matches the intuitive goal of attribu- tion 212 [3]. Specifically, collection of training images is influential for given output image if removing those images from the training set and then retraining from scratch makes the model unable reproduce the same synthesized image. Unfortunately, directly searching for the most influential images according to this definition is computationally infeasible since would require training an exponentially large number of new models from scratch.
Hence, practical influence estimation requires effective approximations. For example, many ap- proaches replace retraining with closed-fon form approximation, computed separately for each training image 12 14 15 [3 16]. For text-to-image attribution, these methods are outperformed by simple matching of off-the-sher shelf image features [17]. Wang al. (18) model customization [I9] to study the effect of training a model towards an exemplar, but find limited generalization to the general large-scale training case. We aim for a tractable method that accurately predicts influence according to the counterfactual definition.
We propose an influence prediction approach with two key ideas (Figure First, we approximate the removal of a training image from a model through an optimization procedure termed unlearning [20 27 22). which increases the training loss of the target image while preserving unrelated concepts. We then compute the training loss for the original synthesized image. However, directly applying this idea would require unlearning separately for every single training image, which is also costly. Our
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Training Dataset
Pretrained
Synthesized
Unlearning
Assess influence (by loss Increase)
(a) Attribution by Unlearning
Remove Top-K Influential Images
Unlearned
(b) Counterfactual Evaluation
Failed Resynthesis
Retrain
8:
Figure I: (a) Our algorithm: We propose a new data attribution method using machine unlcarning. By modifying the pretrained model 0 to unlearn the synthesized result 2, the model also forgets the influential training images crucial for generating that specific result. (b) Evaluation: We validate our method through counterfactual evaluation, where retrain the model without the top K influential images identified by our method. When these influential images are removed from the dataset, the model fails to generate the synthesized image.
second main idea is reverse the roles: unlearn the synthesized image, and then evaluate which training images are represented worse by the new model. This method requires only one unlearning optimization, rather than separate unlearning step for each training image.
The methodology for unlearning is important. Unlearning synthesized image by naively maximizing its loss leads to catastrophic forgetting (23], where the model also fails to generate other unrelated concepts. Inspired by work on unlearning data for classifiers 200 24], we mitigate this issue by regularizing gradient directions using the Fisher information retain pretrained information. Additionally, we find that updating only the key and value mappings in the cross-attention layers improves attribution performance. We show how "influence functions" [3] [1]] can be understood as approximations to unlearning in Appendix but they are also limited by their closed-form approximation nature.
We perform a rigorous counterfactual validation: removing a predicted set influential images from the training set, retraining from scratch, and then checking that the synthesized image is longer represented. We use MSCOCO [25] (~~000 images), which allows for retraining models within a reasonable compute budget. We also test aublicly-vvillble attribution benchmark [18] using customized sot-tto-image models [19] Our experiments show that our algorithm outperforms prior work both benchmarks, demonstrating that unlearning synthesized images is effective way to attribute training images. Our code available at: https: htt:://peeeeran11110 1e..ii..giiii github. io/ AttrilbueeBynnlearniin
In summary, our contributions are:
We propose new data attribution method for text-to-image models, by unlearning the synthesized image and identifying which training images are forgotten.
We find and ablate the components for making unlearning efficient and effective, employing Fisher information and tuning critical set of weights.
We rigorously show that our method counterfactual predictive by omitting influential images, retraining, and checking that the synthesized image cannot be regenerated. Along with the existing Customized Model benchmark, our method identifies influential images more effectively than recent methods based customization and influence functions.

---

2 Related Work
Attribution. Influence functions [15] 13]
approximate how the objective function of a test datapoint would change after perturbing a training datapoint. One may then predict attribution according to the training points that can produce the largest changes. Koh and Liang [3] proposed using influence functions to understand model behavior in deep discriminative models. The influence function requires calculating a Hessian of the model parameters, for which various efficient algorithms have been proposed, such inverse hessian products [13], Arnoldi iteration [26], Kronecker factorization [7 22 29]. Gauss-Newton approximation [16], and nearest neighbor search [3)..
Other methods explore different approaches. Inspired by the game-theory concept of Shapley value [11], several methods train models on subsets of training data and estimate the influence of a
training point by comparing the models with and without that training point [32 33 34 35 Pruthi et
al. [36] estimate influence by tracking train-test image gradient similarity throughout model training.
Recent methods have started tackling attribution for diffusion- bbaed image generation. Wang et al. [18] proposed attribution by model customization [9] where a pretrained model influenced by tuning towards an exemplar concept. Several works adapt TRAK [16], an influence function- -ased method, diffusion models, extending it by attributing specific denoising timesteps [[2], or by improving gradient estimation and using Tikhonov regularization [14). Unlike these methods, our method performs attribution by directly unlearning a synthesized image and tracking the effect on each training image. Our method outperforms existing methods in attributing both customized models and text-to-image models. Concurrent works also apply machine unlearning for attribution tasks (38 (3]. Unlike their approaches, we find that applying strong regularization during unlearning is crucial to obtaining good performance in our tasks.
Machine unlearning. Machine unlearning seeks to efficiently "remove" specific training data points from a model. Recent studies have explored concept erasure for rext-to-image diffusion models, specified by a text request [40] [40]41]] 42 43 44), whereas we remove individual images. While forgetting may be achieved using multiple models trained with subsets of the dataset beforchand 2 doing so is prohibitively expensive for large-scale generative models.
45
Instead, our approach follows unlearning methods that update model weights directly 20 47
48
24). The majority of prior methods use the Fisher information matrix (FIM) to approximate retraining [1]. In particular, we are inspired by the works from Guo el al. (20] and Tanno et al. 24 which draw a connection between FIM-based machine unlearning methods and influence functions. We show that unlearning can be efficiently
without forgetting other training points (20 [22 49
22
50
applied to the attribution problem, by "unlearning" output images instead of training data.
Replication detection. Shen et al. [52) identify repeated pictorial elements in art history. Somepalli et al. I1 and Carlini et al. [10] investigate the text-to-image synthesis of perceptually-exx -exaact copies of training images. Unlike these, our work focuses on data attribution for more general synthesis settings beyond replication.
3 Problem Setting and Evaluation
Our goal is to attribute a generated image to its training data. We represent the training data as
c 1=1 where X' denotes an image and represents its conditioning text. A learning
algorithm A : D + 0 yields parameters of a generative model; for instance, 0 = A(D) is model trained on DD We focus diffusion models that generate an image from noise map e NN I). A generated image from text e is represented as x = Go c). To simplify notation, we write a text-image tuple as single entity. synthesized pair is denoted as (x, c). and a training pair is denoted as z, = (x: Q) ~ D. We denote the loss of an image x conditioned on e as L(z, 0).
Next, we describe the "gold-standard" evaluation method that we use to define and evaluate influence. Section describes our method for predicting influential images.
Counterfactual evaluation. A reliable data attribution algorithm should accurately reflect counterfactual prediction. That is, if an algorithm can identify a set of truly influential training images, then a model trained without those images would be incapable of generating or representing that image. As noted by Ilyas et al. (35) and Park al. [66. counterfactual prediction is computationally intensive to validate. As such, these works introduce the Lincar data modeling (LDS) score as an
3

---

efficient proxy, but with the assumption that data attribution methods are additive, which does not hold for feature matching methods and our method.
In our work, we invest substantial computational resources to the "gold standard" counterfactual evaluation within our resource limits. That is, we use attribution algorithm to identify critical set of K images, denoted as D C D. We then train a generative model without those images from scratch, per synthesized sample and per attribution method. Despite the computational cost, this allows us to provide the community with a direct evaluation of counterfactual prediction, without relying on a layer of approximations. We formalize our evaluation scheme as follows.
Training a counterfactual model. For evaluation, an attribution algorithm given a budget of K images for attributing synthesized sample 2, denoted as D We then train leave- K-out model 02 K from scratch using D K DI, the dataset with the attributed images removed:
0 0T A(Dz*))
(1)
Evaluating the model. We then compare this "leave- K -out" model against 0g = A(D), the model trained with the entire dataset, and assess how much it loses its capability to represent 2 in terms of both the loss change ( 9) and the capability to generate the same sample AG& e) from the same input noise and text c.
First, if the leave- K -out model is trained without the top influential images, should reconstruct synthetic image à more poorly, resulting in higher (2 0):
- K) - C(2, 90)
Second, if properly selected, the leave-K-ou model should no longer be able to generate Gele, c). For diffusion models, we can particularly rely on the "seed consistency" prop- erty 02 54]. Georgiev et al. [12 find that images generated from the same random noise have little variations, even when generated by two independently trained diffusion models on the same dataset. They leverage this property to evaluate attribution via AGo(E,)) c), the difference of generated images between On and 0=K An effective attribution algorithm should lead to leave- K Koout model generating images that deviate more from the original images, resulting a larger AGo(c, c) value:
AGole, dGGo (c, c)2 (0==(s,)))
where d can be any distance function, such L2 or CLIP [55]. Georgiev et al. [12] also adopt AGo(e, c) for evaluation. While evaluating loss increases and seed consistency is specific to diffusion models, the overarching idea of retraining and evaluating if synthesized image is still in the model applies across generative models.
Choice of loss L(z, 0). We focus on DDPM loss introduced by Ho et al. [56], the standard loss used to train diffusion models. Diffusion models learn to predict the noise added to a noisy image X =v + /1 are, where is the original image, is the Gaussian noise, ã, and / controls the noise strength. is an integer timestep sampled between I to T. where T typically set to 1000. A larger t implies more noise added. DDPM loss optimizes for the noise prediction task: El]lc c,t) - ], where cg(-) is the denoiser for noise prediction, and e denotes text condition.
4 Attribution by Unlearning
In this section, we introduce our attribution approach for text-to-image model 0, = A(D), trained on dataset D. We aim to find the highly influential images on a given synthetic image 2 in dataset D.
If we had infinite computes and a fixed set of K images, we could search for every possible subset of K images and train models from scratch. The subset whose removal leads to the most "forgetting" of the synthesized image would be considered the most influential. Of course, such a combinatoric search impractical, so we simplify the problem by estimating the influence score of each training point individually and selecting the top K influential images based on their scores.

---

Formally, we define a data attribution algorithm т. which given access to the training data, model, and learning algorithm, estimates the influence of each training point, denoted by T D, 6, A) (r ),, (2, x2). ..... ( zN)] T represents the factorized attribution function that estimates influence based on the synthesized sample and a training point. Although т can access all training data, parameters, and the learning algorithm, we omit them for notational simplicity.
One potential algorithm for 7(2, z) is to remove z from the training set, retrain a model and check its performance on 2. However, this method is infeasible due to the size of the training set. To overcome this, we have introduced two key ideas. First, rather than training from scratch, we use model
tuning pretrained model to remove a data point. Second, rather than unlearning training points, we apply unlearning to the synthesized image and assess how effectively each training image is forgotten. To measure the degree of removal, we track the training loss changes for each training image after unlearning, and we find this effective for data attribution.
Unlearning the synthesized image. naive approach to unlearn a synthesized image & is to solely maximize its loss [(2, 0). However, only optimizing for this leads to catastrophic forgetting [23]. where the model can no longer represent other concepts.
Instead, we propose to retain the information from the original dataset while "removing" the syn- thesized image, as though it had been part of training. Given model trained on the original dataset 8g =AD) and synthesized image 2, compute Here, we use the set removal notation \ to specify a "negative" datapoint in the dataset. Concretely, solve for the following objective function, using elastic weight consolidation (EWC) loss 23 as an approximation:
model 0 = A(D)) 2)), with 2 removed.
C unlearm (0)=- C(2, 0) Σ C(z,0)
xED
(4)
N
-((,
0o) TO)TF((( -0)). F(8-O)).
2
where is the Fisher information matrix, which is approximated
diagonal form for computational
efficiency. F approximates the training data loss to the second-order [57], technique widely used in
continual learning [23 [58) This enables us to solve for the new model parameters 0_2 efficiently, by
initializing from the pretrained model tg. We optimize this loss with Newton updates:
0 - 0 VC N
(5)
where a controls the step size, F-1 the inverse of the Fisher information matrix. In practice, Newton updates allow us to achieve effective attribution with few iterations and, in some cases, as few as one step. We denote the unlearned model as 0 0_ We provide details of the EWC loss, and Newton update in Appendix
Attribution using the unlearned model. After we obtain the unlearned model 0_2, we define our attribution function T by tracking the training loss changes for each training sample z:
T(2 2) L(z, L(2, 0u).
(6)
The value r(2, z) is expected to be close to zero for most unrelated training images since the EWC loss used in obtaining 0 acts as a regularization to preserve the original training dataset. A higher value of 7(2, z) indicates that unlearned model 0_ no longer represents the training sample z.
Relation with influence functions. Our method draws parallel with the influence function, which aims estimate the loss change of 2 by removing a training point However, training leave-one-out models for every training point is generally infeasible. Instead, the influence function relies on heavier approximation to estimate the effect of perturbing single training point, rather than actually forgetting the training samples. In contrast, our approach only requires running the unlearning algorithm once for a synthesized image query. This allows us to use milder approximation and obtain a model that forgets the synthesized sample. Guo et al. 200 and Tanno el al. [24] explore similar formulation for unlearning training images and draw a connection between their unlearning algorithms and influence function. Our approach aims to unlearn the synthesized image instead,
5

---

O
D-T344
Genarated
"A dus hovvling on frotwry other bifffle."
DNO
Ours
D-TTAK
Genorated Semple
"A man in Iiidd conl shtling
througle
JoureeyTTK DND
anauy fidul""
Figure 2: Attribution results on MSCOCO models. We show generated samples used as query on the left, with training images being identified by different methods on the right. Qualitatively, our method retrieves images with more similar visual attributes. Notably, our method better matches the poses of the buses (considering random flips during training) and the poses and enumeration of skiers.
which connects to influence function in similar fashion. We discuss our method's connection to influence function in more detail Appendix A.3
Optimizing subset of weights. To further regularize the unlearning process, we optimize a small subset of weights, specifically 4 and W" the key and value projection matrices cross-attention layers 559 60). In text-to-imeen -mage models, cross-attention facilitates text-to-image binding, where W identifies which features match each text token, while T determines how to modify the features for the matched patches. We observe that performing unlearning W and W is effective for attribution. Prior works also select the same set of parameters to improve fine-tuning [19 and unlearning [40].
Implementation details. We conduct our studies on text-conditioned latent diffusion models 2 Since diffusion models are typically trained with 1000 steps, evaluating the loss for all timesteps is costly. Therefore, we speed up computation by calculating the loss L(z, f) with strided timesteps; we find that using stride of 50 100 leads to good attribution performance. For calculating the loss change AC 0) during evaluation, we take a finer stride of 5 steps to ensure more accurate estimation of the DDPM loss. Additional details our method, including hyperparameter choices, are provided in Appendix B
5 Experiments
We validate our method in two ways. The first a reliable, "gold-standard". but intensive retraining model from scratch without influential images identified by the algorithm. In Section 5.1 we
perform this evaluation on a medium-sized dataset of 100k MSCOCO images (25). Secondly, in
Section 5.2] we evaluate our method on the Customized Model Benchmark [18), which measures
attribution through customization on Stable Diffusion models This tests how well our method
apply to large-scale text-to-image models.

---

AC(2, 0) (x10
AGole, c) (MSE) f (x10")
AGu(e, e) (CLIP) 1 (x10
Family
Method
K=500
K=1000
K=4000
K=500
K=1000
K=4000
K=500
K=1000
K=4000
Random
Random
3,5
3,5,003
3,5,0.00
4.1s0.05
4. 4,1,0.06
4,0,006
7.9.0:33
7.8
7.9.0.03
Pixel
Pixel
3.6
3.6
4.0
4.3
4,3
4.9.221
7.9.0.10
7,8.0.9
7.7
Text
Text
3.84612
4.28074
5.5
4,160:0
4.3.0
".6.661"
7.80008
7.740.00
7.4.0.08"
DINOV2
3,920112
4:32075
6,30003
4:320:20
3,620:30
"5.12011"
7.770008
77.72000
7,000009
Image
CLIP
4.2+0.14
18
6,4+0,33
4.4.0.20
4.6-0.21
5.2+0
7.6.0.09
7,5+0.08
6.8-0.08
DINO
1.8
5.6
8.
4.5
5,3
5,9
7.4anoo
7.1
6,3.0.10
CLIP (ABC)
4.40015
499817
6,9
4.6.0:5
5.0
3.644135
7.54nio
7.3600.00
6.5 6.5-0
AbC
DINO (AbC)
4.8
5.5
8. 1an.35
4.8
4,9.0.20
5. 5.8
7.5a0.08
7.2at1.0
6.3,9.9
Datalif
3.7
3.7a011
3.9.1..13
4,130.15
4. 110.00
4,2s018
7.92mic0
7.9921.00
7. 1000000
Influence
TRAK
5.2+0.14
5.8+0.16
7.1+0.16
4.7-6.1
4.7-0.4
4.7-0.0
7.6+0:00
7,6+0.00
7.5-0.00
TRAK
4.4g0.11
4.9
5.7a0.15
4.8
5.4
5.4
7.7auos
7.5
2.5
D-TRAK
164
2.
9.6.0.33
8.9.0.44
6.4.0
7.8.0.0
7.3
7.1.0.09
6.4.0.00
Ours
Ours
5.621.16
6.720:33
9.8.0.33
3.720
G.
7.320.09
7.022.09
0.4
Table 1: Leave-K-ou baseline comparisons. Given synthesized image 2, we train leave-K-our models for each of the attribution methods and track AC(2, 0), the increase in loss change, and AGo(e, c), deviation of generation. We report results over 110 samples, and gray shows the standard error. Bolded and underlined are the best and second best performing method, respectively.
5.1 Leave- Kout counterfactual evaluation
Evaluation protocol. We select latent diffusion models [2] trained MSCOCO [25], as its moderate size (118,887 images) allows for repeated leave-K retraining. Specifically, we use the pre-trained model evaluated in Georgiev et al. [12). As outlined in Section 3 for each synthesized image 2, we measure the leave-K model's ' (1) loss change A 0) and (2) deviation of gener- ation AGø(e, c). The deviation is measured by mean square error (MSE) and CLIP similarity (55). We collect 10 synthesized images from the pre-trained model for evaluation, with different text prompts sourced from the MSCOCO validation set. We evaluate A 0) and AGo(e, c) for all synthesized images and report mean and standard error.
We compare method with several baselines:
Random: We train models with K random images removed, using 10 models per value of K.
Image similarity: pixel space, CLIP image features [55. DINO 17), and DINOv2 66]
Text similarity: CLIP text features
Attribution by Customization [18] (AbC): fine-tuned image features trained on the Customized Model benchmark, denoted as CLIP (AbC) and DINO (AbC)
DataInf 62]: an influence estimation method based on approximating matrix inverses.
TRAK 16 and JourneyTRAK 12 are influence dunction-based methods that match the
loss gradients of training and synthesized images, using random projection for efficiency. Both methods run the influence function on multiple models trained on the same dataset (20 in this test) and average the scores. The main difference is in the diffusion loss calculation: TRAK randomly samples and averages the loss over timesteps, while JounnyyTRA calculates it only at 400 for synthesized images during counterfactual evaluation.
D-TRAK [4] a concurrent work that extends TRAK by changing the denoising loss function into a square loss during influence computation. As mentioned by the authors, D-TRAK yields unexpectedly stronger performance even though the design choice not theoretically understood. Different from TRAK, D-TRAK yields competitive performance with a single model.
For attribution methods. we use K 500, 1000, and 4000, representing approximately 0.42%. 0.85%, and 3.4% of the MSCOCO dataset, respectively.
Visual comparison of attributed images. In Figure 2 we find that our method, along with other baselines, can attribute synthesized images to visually similar training images. However, our method more consistently attributes images with the same fine-gre meniindd attributes, such as object location, pose, and counts. We provide more results in Appendix [C.]] Next, we proceed with the counterfactual analysis, where we test whether these attributed images truly influential.
Tracking loss changes Lave-K-out models. First, we report the change in DDPM loss for leave- K Kout models in Table I Matching in plain pixel or text feature space yields weak performance,
7

---

Ours
D-TRAK
D-TRAK
Ger erated Sample
Generated Sample
"A bus eniviing ov freeway
DINO
muns Biue coot skiing through a Snawy fielel"
DINO
nox stier
bafffo."
Jaurmey VASK
Joumey TROK
Figure 3: Leave-K -out analysis for MSCOCO models. We compare images across method and baselines generated by leave-K-ou models, using different K values, all under the same random noise and prompt. significant deviation in regeneration indicates that critical, influential images were identified by the attribution algorithm. Our method leads to image generation that deviate significantly, even with as few as 500 influential images removed (~0.42% of the dataset).
while deep image features, particularly DINO, perform better. Interestingly, DINO outperforms most influence function methods K 4000, despite not being trained specifically for the attribution task. Fine-tuning image features with the Customized Model benchmark, such as CLIP (AbC), shows some improvement. However, in general, the improvement is limited, indicating that transferring from attributing customized models to general models remains challenging [18]
Among influence functions, DataInf performs poorly. According to the paper, the matrix inverse approximation Datalnf uses is more suitable for LoRA fine-tuned models rather than text-to-image models trained from scratch. This approximation leads to poor performance.
TRAK significantly outperforms JourneyTRAK We hypothesize that this is because JourneTRRKK TRAK collects gradients only for denoising loss at timestep = 400, making it less effective for identifying influential images that affect the DDPM training loss different noise levels. On the other hand, iisenneeffuntioon
D-TRAK is the best performing strong performance not well-understood [1].
method, although the reason behind its
Our method consistently performs best across all K values, outperforming both influence functions and feature-matching methods.
Deviation of generated output in leave-K-out models. Figure and Table T shows the deviation in generated outputs for Mavee-Kour models, where all images are generated using the same noise input and text prompt. Consistent with the loss change evaluation, D-TRAK, DINO, and our method yield the largest deviation with small budget. While the three methods perform similarly well in CLIP similarity, D-TRAK outperforms our method in MSE deviations. and JourneyTRe TRAK have formulations similar D-TRAK, but they perform subpar in this test. Interestingly, while D-TRAK yields significant performance gain by changing the loss function, we did not observe the same improvement when applying the same loss function changes method. We discuss this more Appendix C.T
contrast, TRAK
In addition, include analysis whether unlearned models and leave-K-our models forget only the specific concept, along with more qualitative results, in Appendix C.I
Ablation studies. We study two design choices: (1) the effect of EWC regularization and (2) different weight subsets for optimization. Table the results. We find that unlearning without EWC loss greatly hurts attribution performance, indicating the importance of regulating unlearning with Fisher information. We also find that using a subset of weights to unlearn leads better attribution in general. We test three weight subset selection schemes (Attn, Cross Attn, Cross Attn KV), of which outperform the version using all weights. Among them, updating Cross Attn KV performs the best, consistent with findings from model customization [9 63 and unlearning [40).

---

(x10
AGole, c) (MSE) (x10"))
AGole, cj (CLIP) J (x10"))
Variation
EWC?
K=500
K=1000
K=4000
K=500
K=1000
K=4000
K=500
K=1000
K=4000
SGD (1 step)
3,5
3.5
3.5
3.9
3.9:0.17
4.1,0.20
7.9.0009
7.8.0.09
7,8,0009
SGD (10 step)
3,4
3,5+0.10
3,5+0.10
3.740.16
3.9
3.9
7.9.0.0
7.9.0.0
7.8+0.00
Full
5.310.17
6:210:30
8.4.6.27
1.610.19
5:210:3
3.55...3
7.610.08
7.410.00
7.1+0.10
Attn
5.5
6.6e1.23
9.3
5.1.0.33
5.34n22
6.240.22
7.4
7.2+0.00
6.740000
Cross Altn
5.5+0.17
6.5+0.24
9.1-0.3
4.8+0.20
5.3+0.22
6.0en.23
7.3
7.240.10
6.6+0.00
Cross Attn KV
5.6
6,720.23
9.8
5.1
5.7an24
6.122.22
7.3,0.09
7.0g0.09
6.440.10
Table 2: Leave-K -utt ablation studies. We ablate our design choices and report AC(z, 0) and
AGa(e, c) as in Table We find that naive unlearning (SGD without EWC regularization) is not effective. We then compare four different sets of weights to unlearn and find that cross- aooss-attention W* W" (Cross Attn VV) outperforms other configurations. Bolded are the best performing method, and gray shows the standard error.
STOP
STOP
STOP
STOP
STOP
STO
STOP
STOP
ORIN
'A wootoroycle and stop sigu."
Cropped
Attributed training images
Query
Figure 4: Spatially-localizze attribution. Given synthesized image (left), crop regions containing specific objects using GroundingDINO [4]. We attribute each object separately by only running forgetting on the pixels within the cropped region. Our method can attribute different synthesized regions to different training images.
Spatially-Iocalizze attribution. While our formulation is written for whole images. we can run attribution on specific regions with little modification. We demonstrate this in Figure a generated image of a motorcycle and stop sign, using bounding boxes identified by GroundingDINO (64] For each detected object, we run our unlearning (using the same prompt) on that specific object by optimizing the objective only within the bounding box. By doing so, we attribute different training images for the stop sign and motorcycle.
5.2 Customized Model Benchmark
Wang el al. [8] focus on specialized form of attribution: attributing customized models trained on an individual or few exemplar images. This approach provides ground truth attribution, since the images generated by customized models are computationally influenced by exemplar images. While this evaluation has limited generalization to attribution performance with larger training sets, it is the only tractable evaluation for attributing large-scale text-to-image models to date.
Evaluation protocol. Since the Customized Model Benchmark has ground truth, the problem is evaluated as a retrieval task. We report Recall @ K and mAP, measuring the success of retrieving the exemplar images amongst a set including 100K LAION images. We compare with Wang et al. feature-matching approach that finetunes on the Customized Model dataset, referred to as DINO (AbC) and CLIP (AbC). We also compare with D-TRAK, the best-performing influence function method in our previous MSCOCO experiment. For our evaluation, we selected subset of the dataset comprising 20 models: 10 object-centric and 10 artist-style models. We select 20 synthesized images with different prompts for each model, resulting in 400 synthesized image queries.
Comparing with other methods. We report Recall @ 10 and mAP in Figure 6 Our method performs on par with baselines for object-centric models, while significantly outperforming them on artist-style models. Although CLIP (AbC) and DINO (AbC) are fine-tuned for this attribution task, the feature matching approach can sometimes confuse whether to attribute synthesized image to style-related or object-related images. contrast, our method, which has access to the model itself, traces influential training images more effectively. D-TRAK performs worse than our method despite also having model access, which suggests that the influence approximations could be less accurate for larger models. In Figure 5 we show qualitative example. While DINO (AbC) and CLIP (AbC)

---

Object Centre
Arssti Style
O-TIAK Dut
LINC Birnk
Gorerated Cample crevingrt wet hertig"
Gample paploter style
Figure 5: Qualitative examples on the Customized Model benchmark. The red boxes indicate ground truth exemplar images used for customizing the model. Both our method and AbC baselines successfully identify the exemplar images on object-centric models (left), while our method outper- forms the baselines with artistic style models (right).
Stylet Models (Pecalli:-0)
DND JAbC] CUP
DND (AbE)
CUP DIND
DNO
Artict Style
0.0
0.6
Object-Centric Models (Recalig
Cbjtci-Centric Models (MAP)
Figure 6: Customized Model benchmark [18]. We report Recall @ 10 (left) and mAP (right) and show performance on artist-style models (y-axis) vs. object-centric models (x-axis). On object-centric models, our method performs on par with AbC features, which were directly tuned on the benchmark, while significantly outperforming them on artist-style models. D-TRAK performs the second best on artist-style models but worse on object-centric models. We plot one standard error on both axes.
retrieve visually or semantically similar images, our method successfully identifies the exemplars in both cases. We include ablation studies in Appendix C.2
6 Discussion, Broader Impacts, and Limitations
Generative models have entered the public consciousness, spawning companies and ecosystems that are deeply impacting the creative industry. The technology raises high-stakes ethical and legal questions surrounding the authorship of generated content [65] 66
67 68). Data attribution is critical piece of understanding the behavior of generative models, with potential applications in informing compensation model for rewarding contributors for training data. In addition, data as a set of tools that allow end users interpret how and why a model behaves, enabling more trustworthy environment for machine learning models.
attribution can join other works (69 70 7
Our work proposes method for data attribution for text-to-image models, leveraging model unlearn- ing. We provide counterfactual validation, verifying that removing the identified influential images indeed destroys the target image. While our method empirically demonstrates that unlcarning can be effectively used, work remains to make this practical. Though our model unlearns efficiently, estimating the reconstruction loss on the training set remains bottleneck, several forward passes are required on each training estimate, as profiled in Appendix D While our evaluation showed that unlearning useful for attribution, direct evaluation of unlearning algorithms for large generative models remains an open research challenge. Furthermore, to find critical set of images, our method and baselines assign influence scores to individual images and sort them. However, groups of images may have interactions that are not captured in such system. Furthermore, our method and baselines explore attribution of the whole image, while finer attribution on individual aspects of the image, such style, structure, individual segments, are of further interest.
10

---

Acknowledgements. We thank Kristian Georgiev for answering all of our inquiries regarding JoureyTRAK implementation and evaluation, and providing us their models and an earlier version of JoureyTRA, code. We thank Nupur Kumari, Kangle Deng, Grace Su for feedback on the draft. This work partly supported by the Packard Fellowship, JPMC Faculty Research Award, and NSF IIS-2239076.
References
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, llya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint St1i::2112.1074.) 2021.
[2] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
[3] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 22040220. 2022.
[4] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong. Gunjan Baid, Zirui Wang. Vijay Vasudevan. Alexander Ku, Yinfei Yang, Bureu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu, Scaling autoregressive models for content-rich text-to-image generation. arXiv preprint arXiv:2206. 10789, 2022.
[5] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Seyed Kamyar Seyed Ghasemipour, Burcu Karagol Ayan, S Sara Mahdavi, Rapha Gontijo Lopes, Tim Salimans, Ho Jonathan, David J Fleet, and Mohammad Norouzi. Photorealistic text to-maaee diffusion models with deep language understanding. arXiv preprint arXiv:2205. 11487, 2022.
Oran Gafni, Adam Polyak, Oron Ashual, Shelly Sheynin, Devi Parikh, and Yaniv Taigman. Make-a-scene: Scene-based texl-to-image generation with human priors. arXiv preprint arXiv:2203.13131, 2022.
[7] Minguk Kang. Jun- Yan Zhu, Richard Zhang, Jaesik Park, Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-ine sotttto-mmage synthesis. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[8] Huiwen Chang, Han Zhang, Jarred Barber, Aaron Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Patrick Murphy, William T. Freeman, Michael Rubinstein. Yuanzhen Li, and Dilip Krishnan. Muse: Text-to-image generation via masked generative transformers. In International Conference on Machine Learning (ICML), 2023.
[9] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5h: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08-00 2022.
[10] Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramèr, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188, 2023.
[11] Gowthami Somepalli, Vasu Singla, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Diffusion art digital forgery? investigating data replication in diffusion models. arXiv preprint arXiv:2212.0388 2022.
[12] Kristian Georgiev, Joshua Vendrow, Hadi Salman, Sung Min Park, and Aleksander Madry. The journey, not the destination: How data guides diffusion models. arXiv preprint arXiv:2312.06200 2023.
[13] Pang Wei Koh and Percy Liang. Understanding black-box predictions via influence functions. In International conference on machine learning, pages 1885-1894, PMLR, 2017.
[14] Xiaosen Zheng, Tianyu Pang, Chao Du, Jing Jiang, and Min Lin. Intriguing properties of data attribution on diffusion models. In International Conference on Learning Representations (ICLR), 2024.
[15] Frank R. Hampel. The influence curve and its role in robust estimation. Journal of the American Statistical Association, 1974.
[16] Sung Min Park, Kristian Georgiev, Andrew llyas, Guillaume Leclere, and Aleksander Madry. Trak: Attributing model behavior scale. arXiv preprint arXiv:2303.144888 14186, 2023.
11

---

[17] Mathilde Caron. Hugo Touvron. Ishan Misra, Hervé Jégon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In IEEE International Conference on Computer Vision (ICCV). 2021.
[18] Sheng- Yu Wang, Alexei Efros, Yan Zhu, and Richard Zhang. Evaluating data attribution for text-to-image models. In IEEE International Conference on Computer Vision (ICCV), 2023.
[19] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Multi-conceptt customization of text-to-image diffusion. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[20] Chuan Guo, Tom Goldstein, Awni Hannun, and Laurens Van Der Maaten. Certified data removal from machine learning models. In International Conference on Machine Learning (ICML), 2020.
[21] Lucas Bourtoule, Varun Chandrasekaran, Christopher A Choquette- Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, and Nicolas Papernot. Machine unlcarning. In 202/ IEEE Symposium on Security and Privacy (SP), 2021.
[22] Aditya Golatkar, Alessandro Achille, and Stefano Soatto. Eternal sunshine of the spotless net: Selective forgetting deep networks. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[23] James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei Rusu, Kieran Milan, John Quan, Tiago Ramalho. Agnieszka Grabska- et al. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciences (PNAS), 114(13):3521-3222 2017.
[24] Ryutaro Tanno, Melanie Pradier, Aditya Nori, Yingzhen Li. Repairing neural networks by leaving the right past behind. In Conference on Neural Information Processing Systems (NeurIPS), 2022.
[25] Tsung-Yo Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision (ECCV), 2014.
[26] Andrea Schioppa, Polina Zablotskaia, David Vilar, and Artem Sokolov. Scaling up influence functions. In Proceedings of the AAAI Conference on Artificial Intelligence, 2022.
[27] James Martens and Roger Grosse. Optimizing neural networks with frottcker-factored approximate curvature. In International Conference on Machine Learning (ICML), 2015.
[28] Thomas George, César Laurent, Xavier Bouthillier, Nicolas Ballas, and Pascal Vincent. Fast approxi- mate natural gradient descent in a kronecker factored eigenbasis. In Conference Neural Information Processing Systems (NeurlPS), 2018
[29] Roger Grosse, Juhan Bae, Cem Anil, Nelson Elhage, Alex Tamkin, Amirhossein Tajdini, Benoit Steiner, Dustin Li, Esin Durmus, Ethan Perez, et al. Studying large language model generalization with inffuence 2023.
functions. arXiv preprint arXiv:2308.0322.
[30] Han Guo, Nazneen Fatema Rajani, Peter Hase, Mohit Bansal, and Caiming Xiong. Fastif: Scalable influence functions for efficient model interpretation and debugging. arXiv preprint 17777. 2020.
[31] Lloyd Shapley. value for n-person games. Princeton University Press Princeton. 1953.
[32] Amirata Ghorbani and James Zou. Data shapley: Equitable valuation data for machine learning. In International Conference on Machine Learning, pages 2242-2251. PMLR, 2019.
[33] Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas Spanos. Towards efficient data valuation based on the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pages 1167-1176. PMIR, 2019.
[34] Vitaly Feldman and Chiyuan Zhang. What neural networks memorize and why: Discovering the long tail via influence estimation. Conference on Neural Information Processing Systems (NeurIPS), 2020.
[35] Andrew llyas, Sung Min Park, Logan Engstrom, Guillaume Leclere, and Aleksander Madry. Datamodels: Predicting predictions from training data. In International Conference Machine Learning (ICML), 2022.
12

---

[36] Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Conferencee on Neural Information Processing Systems (NeurIPS), 33:19920- 19930, 2020.
[37] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Ruhinstein, and Kfir Aberman. Dream- booth: Fine tuning text-to-image diffusion models for subject-driven generation. In arXiv preprint arxiv:2208. 12242, 2022.
[38] Myeongseob Ko, Feiyang Kang, Weiyan Shi, Ming Jin, Zhou Yu, and Ruoxi Jia. The mirrored influence hypothesis: Efficient data influence estimation by harnessing forward passes. In IEEE Conference on Computer Vision and Puttern Recognition (CVPR). 2024.
[39] Masaru Isonuma and Ivan Titov. Unlearning reveals the influential training data of language models. In The Annual Meeting of the Association for Computational Linguistics (ACL.,, 2024.
[40] Nupur Kumari, Bingliang Zhang, Sheng- Yu Wang, Eli Shechtman, Richard Zhang, and Jun-Yan Zhu. Ablating concepts in text-to-image diffusion models. In IEEE International Conference Computer Vision (ICCV), 2023.
[41] Rohit Gandikota, Joanna Materzynska, Jaden KauttoKa and David Bau, Erasing concepts from diffusion models. In IEEE International Conference on Computer Vision (ICCV), 2023.
[42] Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúiska, and David Bau. Unified concept editing in diffusion models. In IEEE/CVF Winter Conference on Applications of Computer Vision (WACV). 2024.
[43] Chi-Pin Huang, Kai-Po Chang, Chung-Ting Tsai, Yung-Hsuan Lai, and Yu-Chiang Frank Wang. Re- celer: Reliable concept erasing of text-to-image diffusion models via lightweight erasers. arXiv preprint arXiv:311.17711,, 2023.
[44] Eric Zhang, Kai Wang, Xingqian Xu, Zhangyang Wang, and Humphrey Shi. Forget-me-not: Learning to forget in text-to-image diffusion models. arXiv preprint arXiv:2303. 17591, 2023.
[45] Laura Graves, Vineel Nagisetry, and Vijay Ganesh. Amnesiac machine learning. In Conference on Artificial Intelligence (AAAI), 2021.
[46] Ayush K Tarun, Vikram Chundawat, Murari Mandal, and Mohan Kankanhalli. Fast yet effective machine unlearning. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[47] Quae Phong Nguyen, Bryan Kian Hsiang Low, and Patrick Jaillet. Variational bayesian unlearning. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
[48] Aditya Golatkar, Alessandro Achille, Avinash Ravichandran, Marzia Polito, and Stefano Soatto. Mixed- privacy forgetting in deep networks. In JEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[49] Ayush Sekhari, Jayadey Acharya, Gautam Kamath, and Ananda Theertha Suresh. Remember what you want forget: Algorithms for machine unlearning. In Conference on Neural Information Processing Systems (NeurIPS), 2021.
[50] Yongjing Zhang. Zhaobo Lu, Feng Zhang, Hao Wang, and Shaojing Li. Machine unlearning by reversing the continual learning. Applied Sciences, 2023.
[51] Jack Foster, Stefan Schwepf, and Alexandra Brintrup. Fast machine unlearning without retraining through selective synaptic dampening. In Conference on Artificial Intelligence (AAAI), 2024.
[52] Xi Shen, Alexei A Efros, and Mathieu Aubry. Discovering visual patterns in art collections with spatially- consistent feature learning. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 9278-9287, 2019.
[53] Xuan Su, Jiaming Song, Chenlin Meng, and Stefano Ermon. Dual diffusion implicit bridges for image-to- image translation. In International Conference on Learning Representations (ICLR), 2023.
[54] Valentin Khrulkov, Gleb Ryzhakov, Andrei Chertkov, and Ivan Oseledets. Understanding ddpm latent codes through optimal transport. In International Conference on Learning Representations (ICLR), 2023.
[55] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning (ICML), 2021.
13

---

[56] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Conference on Neural Information Processing Systems (NeurIPS), 2020.
2020.
[57] Razvan Pascanu and Yoshua Bengio. Revisiting natural gradient for deep networks. In International Conference on Learning Representations (ICLR), 2014.
[58] Yijun Li. Richard Zhang, Jingwan Lu, and Eli Shechtman. Few-shot image generation with elastic weight consolidation. In Conference on Neural Information Processing Systems (NeurlPS), 2020.
[59] Damitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In International Conference Learning Representations (ICLR), 2015.
[60] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Conference on Neural Information Processing Systems (NeurIPS), 2017.
[61] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.077 2023.
[62] Yongchan Kwon, Eric Wu, Kevin Wu, and James Zou. Datainf: Efficiently estimating data influence in lora-tuned Iims and diffusion models. In International Conference on Learning Reprexentations (ICLR), 2024.
[63] Yoad Tewel, Rinon Gal, Gal Chechik, and Yuval Atzmon. Key-locked rank one editing for text-to-image personalization. ACM Transactions on Graphics (TOG), 2023.
[64] Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su. Jun Zhu, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection, arXiv preprint arXiv:2303.00 05449, 2023.
[65] Katherine Lee, A Feder Cooper, and James Grimmelmann. Talkin" ai generation: Copyright and the generative-ai supply chain. arXiv preprint urXiv:2309.08133. 2023.
[66] Niva Blkin-Koren, Uri Hacohen, Roi Livni, and Shay Moran. Can copyright be reduced to privacy? arXiv preprint 2r35 14822, 2023.
[67] Uri Hacohen, Adi Haviv, Shahar Sarfaty, Bruria Friedman, Niva Elkin-Koren Roi Livni. and Amit H Bermano. Not all similarities created equal: Leveraging data-driven biases to inform genai copyright disputes. urXiv preprint arXiv:2403. 176911 2024.
[68]
Ziv Epstein, Aaron Hertzmann. Hany Farid, Jessica Fjeld, Morgan R. Frank, Matthew Groh, Laura Herman, Neil Leach, Robert Mahari, Alex "Sandy" Pentland. Olga Russakovsky. Hope Schroeder, and Amy Smith. Art and the science of generative ai. Science, 310(6650):1110-111 111-11111,1 2023.
[69] David Bau, Jun Zhu, Hendrik Strobelt, Zhou Bolei, Joshua Tenenbaum, William T. Freeman, and Antonio Torralba. Gan dissection: Visualizing and understanding generative adversarial networks. In International Conference on Learning Representations (ICLR), 2019.
[70] Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. Locating and editing factual knowledge in gpt. arXiv preprint arXiv:2202.0522 2022.
[71] Alex Foote, Neel Nanda, Esben Kran, loannis Konstas, Shay Cohen, and Fazl Barez. Neuron to graph: Interpreting language model neurons at scale. arXiv preprint arXiv:2305. 19911, 2023.
[72] Amil Dravid, Yossi Gandelsman, Alexei A Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2023.
[73] James Martens. New insights and perspectives on the natural gradient method. Journal of Machine Learning Research. 21(166):1-66 2020.
[74] Dan Ruta, Saeid Motiian, Baldo Faieta, Zhe Lin, Hailin Jin, Alex Filipkowski, Andrew Gilbert, and John Collomosse. Aladin: All layer adaptive instance normalization for fine-grained style similarity. In IEEE International Conference on Computer Vision (ICCV), 2021.
[75] Mark Harden. Mark harden's 's artchive. https://www artchive.com 1998.
14

---

A Derivations for Unlearning
In Section we describe our unlearning method and its relationship to influence functions. Here, provide more detailed derivations. Let 0u be the pretrained model trained dataset 'D and loss EeED Ciz. 0). N the size of the dataset. Our goal is to obtain a model 8_2 that unlearns the synthesized image 2.
A.1 EWC Loss
We summarize EWC Loss [23]. which is the second order Taylor approximation of the data EXED C(z, around 8g. We denote the Hessian of the loss as Ho, where [Hoolis =
loss
N 80,00; EnED Liz, 00 We denote the remainder term as R(0).
1
N
Σ C(z, Σ C(z,00) Σ
0) 8= Vl(z,0)
0o) + 2 00)² (0)"11, Ha. (0 0o) + R(0)
xED
N
Oo)
(0 00)² 11.0 (0 0o).
2
We assume that pretrained 0u near the local minimum of the loss, resulting in near-zero gradient LEED VL(zz 00)10-9. We drop out higher order remainder term R(0) in our approximation. If model is trained with a negative log-likelihood loss, the Hessian Ha, is equivalent the Fisher information F [29]. leading to:
N (0 0o)T F(O 0o). E C(z,0) E(z,,00) + 2 RED
(8)
ZED
We note that we focus diffuusinn models, which are trained lower bound of the log-likelihood. this context, the Fisher information can be viewed as the Gauss-Newton approximation of the Hessian [73. The formulation satisfies the purpose of approximating the training loss the dataset and serves as an effective regularizer on our unlearning objective.
Diagonal approximation of Fisher. Since Fisher information is the covariance of the log- ligeliield gradient, its diagonal approximation equivalent taking the square of the gradients and averaging them across the training set. This diagonal approximation is adopted by EWC loss [23 (88. In the context of diffusion models, the Fisher information estimated by averaging across training data, random noise, and timesteps.
A.2 Updating step for unlearning
We then derive the Newton update of our unlearning objective in Equation 5 Below, we repeat our unlearning objective Equation
N (0 0 0o) L unleamm (0)z C(2,0) + 22222
(9)
The Newton step is a second-order update of the form below, where controls the step size;
+ 0 [H (0)
V
L Luteeam (
(10)
where H² unleamn (0) is the Hessian of L unteaam! (0). Now we derive V V unteamn (
V 2 Luntcarn
(0) VC(2,0) +N FFB 00)
(11)
VL(2,0) 0),
15

---

where we assume 0 is close to 8y, so the term N F(0 fb) can be omitted. Empirically, we also tried unlearning with this term added, but observed little change performance. Then, derive H Wunlaarn (0) as follows:
H unleamn
(0)= ~ H2( + N. R N F,
(12)
where (0) is the Hessian of L(2, 0). We assume the magnitude (in Forbenius norm) of (2) is bounded, and with large dataset size N, we can approximate the Hessian H untearm()) (0) as N F only. Incorporating Equation and 12 into Equation I0 we obtain our Newton update in Equation 5
0 - 0
F N
0).
(13)
A.3 Connection to influence functions
We note that special case of our formulation, running update step once, with small step size, is close to the formulation of influence functions. The difference mainly on the linear approximation error of the loss on the training point. Starting with pretrained model Og and taking an infinitesimal step yy:
0_ 0
+ F
(14)
When we evaluate the loss of the training point using the unlearned model 9_2, we can write the loss in linearized form around du. as we taking small step:
C(z, 22) == Olz,8)) + VL Bo) ( 0b) = Olz,0)) + VVLz - (
(15)
VL(2,
Now, we plug Equation 15 into our attribution function in Equation
T(2 2)
L(z, 0 L(z, 0o)
(16)
=
In this special case, our method equivalent to influence function V LL(z F-- approximations. Practically, the difference between our method and influence functions is that we taking larger, sometimes multiple steps (rather than a single, infinitesimal step), and are explicitly evaluating the loss (rather than with linear, closed-form approximation).
0). after
B Implementation Details
B.1 MSCOCO Models
Models trained from scratch. select our source model for attribution from Georgiev et al. [1]. which is a latent diffusion model where the CLIP text encoder and VAE are exactly the ones used in Stable Diffusion v2, but with a smaller U-Net. To retrain each MSCOCO model for leave-K-or evaluation, we follow the same training recipe as the source model, where each model is trained with 200 epochs, a learning rate 10-4, and a batch size of 128. We use the COCO 2017 training split as our training set.
Unlearning. To unlearn synthesized sample in MSCOCO models, we find that running with step already yields good attribution performance. We perform Newton unlearning updates with step sizes of 0.01 and update only cross-attention KV (W*, W*). We find that updating cross-attention KV yields the best performance, and we later provide ablation studies on the optimal subset of layers to update. We sample gradients 591,49 1,355 times to estimate the diagonal Fisher information, equivalent to 5 epochs of MSCOCO training set.
16

---

B.2 Customized Model Benchmark
Model collection. As described in Section 5.2 we selected subset of the dataset [88] comprising of 20 models: 10 object-centric and 10 artist-style models. For all object-centric models, we select models with distinct categories. For artist-style models, we select 5 models trained from BAM- FG [74 exemplars and 5 models trained from Artchive 7 exemplars. To speed up computation, calculate Fisher information Stable Diffusion v1.4, the base model of all the customized models, over the selected subset of LAION images. We then apply the same Fisher information to all customized models.
Unlearning. We find that running 100 unlearning steps yields a much better performance than running with 1 step for this task. Moreover, updating only cross-attention KV yields a significant boost performance in this test case. Appendix C.2 we show an ablation study on these design choices. We sample gradients 1,000,000 times to estimate the diagonal Fisher information, where the gradients are calculated from the 100k Laion subset using Stable Diffusion v1.4.
B.3 Baselines.
Pixel space. Following JounnyTRAK's implementation 12 we flatten the pixel intensities and use cosine similarity for attribution.
CLIP image and text features. We use the official VIT-B/32 model for image and text features.
DINO. We use the official ViT-B/16 model for image features.
DINOv2. We use the official ViT-LI4 model with registers for image features.
CLIP (AbC) and DINO (AbC). We use the official models trained on the combination of object- centric and style-centric customized images. CLIP (AbC) and DINO (AbC) are selected because they are the best-performing choices of features.
TRAK and Journey TRAK. We adopt the official implementation of TRAK and JouneyyTRA and use a random projection dimension of 4096, the same as what they use for MSCOCO experiments.
D-TRAK We follow the rest-performing hyperparameter reported in D-TRAK, using random projection dimension of 32768 and lambda of 500. We use a single model to compute the influence score.
B.4 Additional Details
Horizontal flips. Text-to-image models our experiments are all trained with horizontal flips. As a result, the models are effectively also trained with the flipped version of the dataset. Therefore, an attribution algorithm for each training image on its original and flipped version and obtain the final score by taking the max of the two. For fair comparison, We also find that taking the average instead of the max empirically yields similar performance.
adopt this approach for all methods.
Computational resources. We conduct all of our experiments A100 GPUs. It takes around 16 hours to train an MSCOCO model from scratch, 20 hours to evaluate all training image loss, and minutes to unlearn synthesized image from pretrained MSCOCO model. To finish all experiments on MSCOCO models, it takes around 77K GPU hours. For Customized Model Benchmark, it takes 2 hours to unlearn synthesized image and 16 hours track the training image loss. To finish all experiments on this benchmark, it takes around 36K GPU hours.
Licenses.
The source model from Georgiev et al. 12 JournyTRRKK is released under the MIT License. The MSCOCO dataset released under the Creative Commons Attribution 4.0 License. Stable Diffusion v2 is released under the CreativeMI. Open RAIL++M License. The CLIP model is released under the MIT License. The Customized Model Benchmark released under the Creative 4.0 International License.
Commons
17

---

Target Images forget)
Other Images (to retain)
MSE (t)
CLIP (1)
MSE (L)
CLIP (t)
SGD
0.081 =0.003
0.67=0.01
0.033+0.0004
0.83+0.002
Full weight Ours
0.086:0.005
0.70
0.009
0.86a0.002
0.09320.004
0.65
0.022:0.0004
0.89
Table 3: Effectiveness in Unlearning Synthesized Images. We compare different choices of unlearning algorithms and evaluate based on whether the method can forget the target images and retain other images. We measure the performance with regenerated images' deviations mean square error (MSE) and CLIP similarity. SGD refers to the naive baseline without EWC regularization, and full weight refers to updating on all of the weights instead of cross-attention KV.
Target
Other
Original Model
Ours
Full weight
SGD
Original Model
Ours
Full weight
SGD
Figure 7: Ablation for unlearning. We find that our unlearning method (Ours) outperforms other variants (Full Weight, SGD) in terms of forgetting the target (left) while retaining other concepts (right).
C Additional Analysis
C.1 MSCOCO Models
Additional plots.
We visualize the baseline comparisons Table a plots, where Figure [2 1
represents loss change, MSE deviation, and CLIP deviation, respectively. We also visualize ablations studies Table 2 as plots (Figure 16
Densely sweeping K is generally not feasible for attribution methods, each queried synthesized image requires retraining a different set of K images. However, the plots, we report more leave- K-out models for random baselines, sweeping through K = 500, 1000, 2000, 3000, 4000, and then 5000 to the full dataset size by increments of 2000. We use the densely swept random baselines as reference to provide better intuition of each method's 's performance. For example, in Figure 12 removing 500 images (just 0.42% the dataset) using our method is equivalent to removing 57.5k random images (48.6% of the dataset) vs. 55.8k images (47.1%) and 44.3k images (37.5%) images for the best-performing baselines, D-TRAK and DINO.
Ablation studies. We perform the following ablation studies and select hyperparameters for best performance each test case:
SGD (1 step): SGD step, step size 0.001
SGD (10 steps): SGD steps, step size 0.0001
Full weight: Newton steps, step size 0.0005
Attention: Newton steps, step size 0.005
Cross-attention: Newton steps, step size 0.005
Cross-attention KV: Newton steps, step size 0.01 (This is our final method)
The SGD step refers the baseline of directly maximizing synthesized image loss without EWC loss regularization, as described in Section
We also compare different subsets of weights optimize and report plots for loss change, deviation measured by MSE, and deviation measured by CLIP similarity in Figure 15
16 17 respectively.
Effectiveness in unlearning synthesized images. Our attribution method relies on unlearning synthesized images, making it crucial have unlearning algorithm that effectively removes these images without forgetting other concepts. We analyze the performance of our unlearning algorithm
18

---

Target Images (to forget)
Related Images (to retain)
Other Images (to retain)
MSE (1)
CLIP (4)
MSE (4)
CLIP (t)
MSE (4)
CLIP (t)
K=500
0.72+0.012 0.004
0.039
0.86#0.001
0.041 #0.0003
0.79
K=1000
0.058=0.004
0.68=0.014
0.041=0.0003
0.86=0.001
0.041 =0.0003
0.79=0.001
K=0000
0.060#0.004
0.61=0.014
0.046#0.0004
0.83+0.001
0.04 =0.0003
0.79=0.001
Table 4: Does leave-K-out models forget other images? We verify that leave-K-model forgets concepts specific to the target query. We report deviations (MSE, CLIP) from the target image, related images that are similar to the target, and other images of unrelated concepts. We find that target images deviate more than related and other images, while other images stay almost the same. Related images's errors increase with larger K, but they are much smaller than target images' deviations.
Target
Related images
Other images
Original Model
Leave-K-out Model
Figure 8: Does Seave-K-out models forget other images? We show that leave- K -out model forgets the specific target (left), while retaining generation on related images (middle) and images of other concepts (right).
itself and ablate our design choices. We construct experiments by unlearning a target synthesized image and evaluating:
Unlearning the target image: We measure the deviation of the regenerated image from the original model's output-thh the greater the deviation, the better.
Retaining other concepts: We generate 99 images using different text prompts and evaluate their deviations from the original model's output-the smaller the deviation, the better.
We measure these deviations using mean square error (MSE) and CLIP similarity. We evaluate across 40 target images, with text prompts sampled from the MSCOCO validation set. We compare to the following ablations:
SGD refers to swapping our method's Newton update steps (Eq. 5) to the naive baseline, where we run SGD steps to maximize the target loss without EWC regularization.
Full weight refers running Newton update steps on all of the weights instead of Cross Attn KV.
Table along with Figure 7, shows the comparison. Both our regularization and weight subset optimization help unlearn the target image more effectively, without forgetting other concepts.
Does leave-K-out models forget other images? In Sec. 5 we show that leave-K Morre-Koout models forget how to generate target synthesized image queries. Here we ask whether these models forget unrelated images, too. We study how much leave-K-out model's generation deviates from those of the original model in three categories:
Target images:
the attributed synthesized image. Leave- K -out models should forget these the greater the deviation, the better.
images synthesized by captions similar to the target prompt. We obtain the most similar 100 captions from the MSCOCO val set using CLIP's text encoder. Leave-K-our models should not forget all of them-the smaller the deviation, the better.
Related images:
Other images: images of unrelated concepts. Prompts are 99 different captions selected from the MSCOCO val set. Leave-K K out models should not forget these the smaller the deviation, the better.
19

---

Ca #) tx100
AGo(t, o) (MSE) 1 (x10
AGale, c) (CLIP) (x10
Method
Loss
K=500
K=1000
K=4000
K-500
K=1000
K=4000
K=500
K=1000
K=4000
(1)(();)) olf
5.2.0114
5.8
7.
4.7.0a1
4.7.004
4.7.000
7.6
7,6.009
7.5
D-TRAK
lee(x..01 01
5,4.016
6.6.0.22
9.6an:3
5.9.0
6.4.0
7,8
7.3
7.1,000
6.4.0.0
Our
3.6:0016
6.7.5
7,3in00
7.0.000
6.4.0.10
Ours (R-TRAK)
(ro(Xs,)))
4.4.0.1
5.1.000
7.640.6
4.7-033
4,94004
6.0
7.6
7.4-009
Table 5: Choice of loss function. We study different choices of loss. While D-TRAK 14] empirically
showed that changing the diffusion loss a square loss increases performance drastically for influence function, we find that this loss change does not grant performance gain for our formulation. We report ( 8) and AGy(c, c) as Table
Sttyt Modes (Recall@11)
1.0
Curs
Artist-Style Models (mAP)
Durs
0.8
Ours (no SM))
0.8
Durs (no V*]
I-dap
I-lep
waight update
upight
0.6
SGD
0.6
SGD
0.4
0.2
0.2
0.0
0.0
0.0
0.2
0.4
0.6 0.8
1.0
0.0
0.2
0.4
0.6
0.8
1.0
Object-Centric Models (Recall@10)
Object-Centric Models (MAP)
Figure Ablation studies for Customized Model Benchmark. We report evaluation on the Customized Model Benchmark in the same fashion as in Figure We find that training with multiple steps, updating a selected subset of weights, and regularizing unlearning via Fisher information is crucial to this task. Additionally, we test version where we apply our algorithm without the special token V". While reduces performance, it still performs well in overall.
In Fig find that the leave-K model "forgets" the query bus image specifically while retaining other buses and other concepts. In Table we quantitatively report deviations using mean square error (MSE) and CLIP similarity, where we evaluate 40 pairs of target images and leave-K-on K-out models. We observe that target images have larger MSE and lower CLIP similarity than related images and other images. Also, the number of removed influential images (K) increases, the target image error increases rapidly while other images stay almost the same. Interestingly, related images' errors increase with larger K, but the errors still much smaller than those of target images. As K increases, the group of influential images can start affecting other related concepts.
Applying the same loss function changes D-TRAK. As mentioned in Sec. 5.1 D-TRAK yields a significant performance gain by changing the loss function in TRAK from denoising loss (x c, < to square loss ( (x,, ))) In Table 5 same loss change to our method leads to worse performance.
we find that applying the
Additional results. We provide more attribution results in Figure 10 and more results on leave-K- out models Figure
C.2 Customized Model Benchmark
Ablation studies. We perform the following ablation studies and select hyperparameters for best performance in each test case:
Cross-attention KV (100 steps): 100 Newton steps, step size 0.1 (denoted as Ours in Figure
Cross-attention KV (1 step): 1 Newton step, step size 10
Full weight (100 steps): 100 Newton steps, step size x 10-5
Cross-attention KV, SGD step (100 steps): 100 SGD step, step size 0.01
Again, the SGD step refers to the baseline of directly maximizing synthesized image loss without EWC loss regularization, as described in Section 4
We report the result of our ablation studies in Figure 9 Our findings indicate that for this test case,
selecting small subset of weights (i.c., cross-attention KV) combined with multiple unlearning
20

---

steps (100 steps) is crucial for effective attribution. We hypothesize that stronger regularization is necessary for unlearning larger-scale models, and that such models benefit more from numerous smaller unlearning steps rather than fewer, larger steps to achieve a better optimization.
Customized models in this benchmark associate the exemplar with special token v', which is also used for generating synthesized images. Our method involves forgetting the synthesized image associated with its text prompt, so by default, we tested with V included. Mcanwhile, also evaluated our method without v" in the prompts. Figure 6 shows that removing V reduces performance, but the method still performs well overall.
D Efficiency Comparison with Other Methods
We compare our method's s efficiency with other methods. Below, we briefly overview each method:
Our unlearning method: we obtain a model that unlearns the synthesized image query and checks the loss increases for each training image after unlearning.
TRAK, JourneyTRAK, precompute mundolly-projecteo loss gradients for each training image. Given synthesized image query, the authors first obtain its random- then match it with the training gradients using the influence function. For good performance, both methods require running influence function on multiple pre-trained models (e.g., 20 for MSCOCO).
random-projected projected loss gradient and
D-TRAK: The runtime complexity is similar to that of TRAK, but it is running the influence function single pre-trained model.
Feature similarity: standard image retrieval pipeline. Precompute features from the training set. Given synthesized image query, we compute feature similarity for the entire database.
We compare the efficiency of each method.
Feature similarity is the most run-time efficient since obtaining features is faster than obtaining losses or gradients from a generative model. However, feature similarity does not leverage knowledge of the generative model. Our method outperforms various feature similarity methods (Table I
Our method more efficient than TRAK/Journey SAAEEIeeyTRAK/DT-R.TAAAA in precomputation. Our method's precomputation is much more efficient runtime and storage-wise compared to TRAK, JourneyTRA TRAK, and D-TRAK. Our method only requires computing and storing loss values of training images from a single model. On the other hand, TRAK-based methods require pre-training extra models (e.g., 20) from scratch, and precomputing/storinn loss gradients of training images from those models.
Our method less efficient when estimating attribution score. TRAK-based methods obtain attribution scores by taking dot product between synthesized image gradient and the stored training gradient features. The methods require calculating and averaging such dot product scores from the 20 pretrained models. On the other hand, as acknowledged in our limitations (Sec. o though our model unlearns efficiently (e.g., only 1-step update for MSCOCO), getting our attribution score involves estimating the loss on the training set, which is less efficient than dot product search. Tradeoff-wise. our method has a low storage requirement at the cost higher runtime.
Our main objective is to push the envelope on the difficult challenges of attribution performance. Improving computation efficiency of attribution is challenge shared across the community [6 29 and we leave it to future work.
21

---

Oue
Generated Sample
les
DNQ
oranspod space."
Our
D-TTAK
Generated Somple
rebme
liadfli the
green
DWO
Jour ney
Ous
D-TRAK
Generated Semple
"A ask lniinn en hes that G
DNO
naitmane"
IeMora
J.P.oor
Oue
"0 WiC
J.P.orr
IPMorga
D-TRAK
gan
RÍ AS
Generaled Sunt pla
"A tennds playo
I.MMnaa
BK.AMI
J.P.Morg
movsing to get
the bal..
JoumeyTAAK DIOO
Figure 10: Additional attribution results for MSCOCO models. This is extension Figure
in the main paper. Our method identifies images with similar poses and visual attributions to the query image. Importantly, in Figure [] we verify that leaving these images out of training corrupts the ability of the model to synthesize the queried images.
22

---

K-500
K-1000
K-4000
K-500
K-1000
K-4000
D-TRAK
D-TRAK
Ganerated Sempla
Ganerated Somple
"A small alosol tokket in
DIN
zebia all by Ckodif in Lhe forest
DINO
spare"
TRAK
moy
K--00
K=1000
K=4000
K-500
K=1000
K-4000
OD wio
D-TRAK
D-TRAK
Gar erated Semple
Generated Sample
ane laging on clarkes that
DINO
"A tenaii player
DINO
manning to
ave
the batl."
JourneyTRAK
Journey
Figure 11: Additional leave-K-out model results for MSCOCO models. This is extension of Figure 3 in the main paper, showing the results from removing top-K influential images from different algorithms, retraining, and attempting to regenerate synthesized sample. The influential images for these examples are shown Figure 10 Our method consistently destroys the synthesized examples, verifying that method is identifying the critical influential images.
23

---

Loss Change Images Removed
0.03
DDPM Change
random
CUP
Datainf
JoUmeyTRAK
Pisel
DIN
CLIP (AbC)
DINO (AbC)
TRAK
D-TRAK
0.02
0.01
Iimagesromover removed
After removing images and retraining, the retrained model equivalent removing how many images?
Full dataset
100000
k=500
of random images
k=1000
75000
K-4000
50000
Pixel
CLIP Text
DINOx2
CUP
CUP (AbC)
DINO DINO (AbC) Datainf
TRAK JourneyTRAK D-TRAK
Ours
Figure 12: Evaluating loss changes in leave-K-one -uo models. Given a synthesized image 2, we train leave-K-our models for each of the attribution methods and track AC 0), the increase in loss change. (Top) We plot DDPM loss change versus K. The loss change becomes more significant with a larger K value, and our method (brown) yields the largest loss change for every K, indicating our method more successfully identifies the influential images. The purple curve represents models where random images are removed, serving as a reference. We show error bars representing standard error in this plot (which small and negligible). (Bottom) For each method, we show the equivalent number of random images needed to achieve loss change.
MSE Recorstruction Error
0.10
MSE error
random
CUP Text DINO v2
CLIP
DINO
Datainf TRAK
JoumeyTRAK
Ours
Pivel
CLIP (AbC)
DINO (AbC)
D-TRAK
0.06
0.04
1000
4000
10000
Full dataset
iimages
of random Images comoved
After removing K images and retraining, the retrained model equivalent to removing how many random images?
100000
75000
k-500
k=0000
25000
k-4000
Pisel
CUP Text
DINO v2
CUP
CLIP (AbC)
DIND
DINO (AbC) Datahif
TRAK JoumeyTRAK D-TRAK
Durs
Figure 13: Deviation of generated output in leave-K-out models, in mean square error (MSE). We report the deviation of generated output in the same fashion as in Figure [2 The higher the MSE, the better since more deviation indicates that the K images removed are more influential.
24

---

CLIP
0.8
Reconstructonn
0.6
random
JoAK
Pixel
DINO v2
CLIP (AbC)
DINO (AbC)
TRAK
D-TRAK
0.4
10000
K kiaags removed
of rondom images removed
removing images retraining, retrained model equivalent removing many images?
Full dataset
100000
k-500
k=1000
75000
k=4000
50000
CUP Text
CLIP (ABC)
DINO
Datainf
TRAK
D-TRAK
Figure 14: Deviation of generated output in leave- K -out models, in CLIP similarity. We report the deviation of generated output in the same fashion as in Figure 12 A lower CLIP similarity in Leave-K-ou models indicates better attribution algorithm.
Loss Change vs. K Images Removed
0.03
DDPM Loss Change
random
Cross
step)
Attn
Cross
SGD (10 step)
0.01
0.00
500
1000
4000
10000
Full dataset
K Images removed
# random images temoved
removing images retraining,
equivalent removing many images?
Full datase!
100000
75000
50000
25000
k=1000
km4000
SGD
Full
Cross
Figure 15: Ablation studies for attributing MSCOCO models. We report the change synthesized image loss in leave-K K -out models in the same fashion as in Figure 12 in the main paper. We compare four different sets of weights
unlearn (full, attention layers, pross-attention layers, and cross- attention W W"), and we find that coss-attention W W outperforms other configurations. We also evaluate naive variation where we apply SGD to maximize the synthesized image's loss, and we find that updating with steps or 10 steps both perform only at chance (random baseline).
25

---

MSE errr
MSE Recorstructin
0.07
random
SGD (1 step)
SCD
Full
Ath
Cross Altn
Cross Alth KV
0.06
0.05
0.04
1000
4000
10000
Full dataset
Images removed
of random imags ramoved
After removing K images and retraining, the retrained model equivalent to removing how random images?
Full dataset
100000
k=500
k-1000
75000
k-4000
50000
SGD 11 step)
SGD (10 step)
Full
Altn
Cross Altn
Cress Attn KV
Figure 16: Ablation studies for attributing MSCOCO models. We report the deviation of generated output in leave- Koout models in mean square error (MSE) in the same fashion as in Figure [3] We find that the trend each ablation follows that of Figure 15
0.5
random
SGD (10
Attn
Cross
500
1000
4000
10800
Full data set
images removed
of rundom images nimozed
After removing K im agee and retraining, the retrained model equivalent to removing how many random images?
dataset
100000
1=00
75000
k<4000
50000
25000
step)
step)
Figure 17: Ablation studies for attributing MSCOCO models. We report the deviation of generated output leave-X, K-out models in CLIP similarity in the same fashion as Figure 14 We find that the trend of each ablation follows that of Figure T5
26

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In the abstract and introduction, the claims are made reflect the paper's contributions and scope accurately.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: See Section 6
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
27

---

Justification: We provide the derivation of our method, assumptions, and proofs in Section and Appendix A
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: We provide all the information for reproducing our main results in the main paper (Section 5) and the Appendix (Section B). We will also release the code upon publica- tion.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, a No answer this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing verifying the results.
5. Open access to data and code
28

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [No]
Justification:
We will release them upon publication.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ mripss.cc/ubliic/gui.eee CoodeSumeeieeeeee ssi iessonPolli mollicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We provided these details in the main paper (Section 5 and Appendix (Sec-
tion B
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We report I-standard-emme error bars in Figure 12 state that reporting error bars representing 1 standard
16
17
We
the captions.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
29

---

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables figures symmetric error bars that would yield results that are out of range (c.g. negative emror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We report compute resources for our experiments in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't 't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
We fully conform with the NeurlPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss societal impacts in Section
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
30

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Answer: [NA]
Justification: We conduct our experiments existing models and data. Our work does not require releasing data or models.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: In Appendix B
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.papersstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
31

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We will release the code upon publication. No assets provided for this submission.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/codeemoodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
32


Reviews:
[{"review_id": "Vbp9E927Tk", "reviewer": "Reviewer_ZhUS", "summary": "Authors propose a method for identifying training images that need to be removed from the training set of a generative model to prevent a single specific “bad” (undesired) output from occurring in its output. Authors propose to directly unlearn the synthetized “bad” image, evaluate how this unlearning changes the training loss across training examples, and retrain the model omitting highly effected training examples - instead of evaluating how including each training example affects the “bad” output. Authors found that the resulting approach worked best when combined with prior techniques for preventing catastrophic forgetting in classifier uncleaning (e.g. optimizing only a set of weights and performing unlearning steps using Newton-Raphson iterations on the unlearned example using the original Fisher matrix). Authors show that retraining generators on pruned training sets indeed prevents the retrained generator from generating the undesired image - more specifically, authors show that the loss on the undesired output increases (more so compared to baseline methods) and that rerunning the retrained generator with the same random seed and text conditioning does not produce the “bad” example indeed.", "questions": "In addition to the weaknesses listed above, here are some questions/concerns I had while reading the paper:\n\nFigure 1 - it is not said explicitly, but after several attempts, I figured out that the “bad” output that we want to remove in this figure is “bedrooms”, so it makes sense that the method assigned high loss (bottom orange bars) and “picked” bedroom images, the fact that only a single image (mountain) either “picked by the method” or “remains in the training set” is confusing; it might be worth to keep only two bedrooms and add a couple more images with low loss and somehow highlighting that “bedrooms are picked for removal”, not “only a single mountain is picked for keeping”.\n\nL130 vs L104 - it is not clear why you need to introduce \\tilda \\theta if you already have \\theta, especially given that in (2) RHS appears to not depend on \\theta (w\\o \\tilda) and L157 also uses \\theta (w\\o \\tilda), so the use of \\tilda there is somewhat inconsistent and confusing; looking at L483-499 I think I understand now that the implied difference between the two is “optimal” vs “updated”, but i think this distinction is somewhat confusing in the main paper\n\nL132 - \\epsilon is never introduced at this point, so this is a bit confusing\n\nL137-139 - trained from the same random noise or generated from the same random noise? worth rewording\n\nL168-172 - these lines appear to not follow from the previous paragraphs introducing the notation for the attribution algorithm since it already provides per-sample scores (and therefore does not require 2^K evaluations); or not, and each tau(\\hat z, z_i) is assumed to potentially involve iterating over all subsets of the training set (w\\ and w\\o z_i) - if so, please elaborate\n\nFigure 2 - “Notably, our method better matches the poses of the buses (considering random flips during training) and the poses and enumeration of skiers.” - I would not say that it is very apparent from the figure.", "limitations": "Authors do mention some limitations of their work.", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "While at first the result might seem trivial (since authors both optimize and measure the same metric - “the model loss”) the finding is actually in fact very much non-trivial since authors retrain the model while pruning images that were affected most by unlearning the problematic output, so it might have so happened that removing these images did not prevent the output from occurring - so reported findings are indeed both interesting and valuable for the community (potentially beyond unlearning work).\n\nAuthors provide a random baseline and all provided results are significantly different from random (confidence intervals seem unnecessary).\n\nAuthors provide qualitative evaluation (Fig 3) suggesting that the desired effect is indeed observed in practice and not only in metrics. \n\nOverall, the paper is well written and does a very good job of motivating both the problem and the proposed solution. Authors provide an extensive literature review that manages to be helpful even to a person without prior experience with unlearning (me).", "weaknesses": "While I enjoyed reading the first half of the paper, I have serious concerns regarding the quality of the results section. While authors provide many qualitative results in the main paper, the burden of making sense of quantitative results (e.g. comparing them to ablations) is not only entirely on the reader, but is made worse by complete lack of any raw numbers for cross-comparison (e.g. no tables - only barplots w\\o numbers), and putting the vast majority of quantitative results into supplementary without providing that much more context there. For example, Figure 4 (top) - colors are very similar and lines are very close, making results not really legible. Given that performance is evaluated only at three data points, I see no reason why this can not be a table - that would aid both reproducibility and comprehension (e.g. cross-comparison). I also do not quite understand the point of reporting the \"equivalent to X random points” metric - it is never even explicitly discussed in the main paper. All results for MSE and CLIP metric and all Ablations (also no tables, only bar plots without numbers) are in the supplementary, but even there not much additional context for interpreting these results is provided. \n\nSome claims are discussed and are just never qualitatively verified - e.g. authors claim that the proposed technique is specifically designed to not lead to catastrophic forgetting, but I could not find any evidence that would support that. Moreover, from Figure 3 it might seem like it not only forgot how to generate this specific bus, but might have forgotten how to generate all buses - I am not entirely sure if that is the intended behavior. I would appreciate a qualitative confirmation that the model did not just “forget everything”.\n\nI have minor concerns regarding the delta G metric. While Georgiev et al. [12] supposedly already showed in an accepted (workshop) submission that two models trained independently on the same or similar datasets generate similar images when primed with the same noise input - justifying the existence of metric (3) - I’d appreciate some results confirming that this is indeed the case for these models in these experiments. For example, if authors showed at least qualitatively that removing images most influential for generating a particular “bad” bedroom from the training set does not indeed in any way affect the generation of unrelated images (e.g. an image of mountains), that would help. Or, even better, by performing textual inversion on the “undesired” input after removing images that affect it the most, to show that it can no longer be represented by the model. Otherwise, there is no way to tell if a particular “bad generation” just “migrated” elsewhere.\n\nAnother minor concern: since authors performed multiple gradient update steps, I wonder whether other methods can also be used with multiple update steps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the thorough comments and suggestions.\n\n### **Results Presentation**\n\n**Tables.** Thanks for the suggestion. We include a table for baseline comparison in Tab. 1 of the response PDF, which corresponds to results in Figs. 4, 7, and 8 of the main text. We will include this, along with a similar table for ablation studies, in our revision.\n\n**More context for supplemental results.** We will discuss our ablation studies more in the main text in our revision. To elaborate on the findings as reported in Appendix C.1, we find that using a subset of weights to unlearn leads to better attribution in general. We test three weight subset selection schemes (Attn, Cross Attn, Cross Attn KV), all of which outperform the using all weights. Among them, updating Cross Attn KV performs the best, consistent with findings from model customization [1,2] and unlearning [3].\n\nIn addition, we discussed the deviation of generated output (delta G metric) in Section 5.1 of the main text (L271-278). Due to space limit, we included the corresponding figures in the Appendix. We are happy to move some related figures (Figs. 7 or 8) to the main text in the revision.\n\n**“Equivalent to X random points” metric.** We introduced this metric in L229 of the main text. We proposed this metric to convert from DDPM loss changes (which is not intuitive to understand) to a budget of images (which is more understandable). For example, removing 500 images (0.4% of the dataset) predicted by our approach causes the same performance drop as randomly removing around half of the dataset. We will clarify this in the revision.\n\n### **Evidence for preventing catastrophic forgetting**\n\nPlease see G1 in our general response, where we quantitatively verify that the target image is forgotten, while other information is retained.\n\n### **Did the leave-K-out model “forget everything” about buses?**\n\nThe leave-K-out model “forgets” the specific query bus, while retaining other buses and other concepts. Fig. 1 of the response PDF shows a qualitative study of this. We also report a quantitative study of this in the general response G2.\n\n### **Justification of delta G metric**\n\nIn general response G2, we follow the suggestion and study whether leave-K-out models affect the generation of unrelated images. \n\nIn general response G3, we also verify that Textual Inversion cannot faithfully reconstruct the images, therefore making it a less ideal choice to check whether a concept is represented by leave-K-out models.\n\n### **Can other methods use multiple update steps?**\n\nExisting influence function methods for text-to-image models are based on a local linear approximation. The approximation is based on a single, infinitesimal network update, so it is not compatible with multiple update steps.\n\nFeature similarity methods leverage fixed features, so network updates will not apply to these methods.\n\n### **Other clarifications**\n\n**Fig. 1 images.** We will include more non-bedroom images in Fig. 1 in the revision.\n\n**The use of $\\tilde{\\theta}$.** Yes, $\\tilde{\\theta}$ refers to the “optimal” pretrained model. Since it is a constant term in the EWC regularization loss (Eq. 4), we use a different term to separate it from $\\theta$, the “updated” parameters for unlearning. We will make the notation more consistent in the revision and rename $\\tilde{\\theta}$ to $\\theta_0$ for more clarity.\n\n**Introducing $\\epsilon$.** In L132, $\\epsilon$ is the noise map introduced in L105. We will clarify this by mentioning it is the noise map again in L132.\n\n**Generated from the same random noise.** In L137-139, the images are generated from the same random noise. We will rephrase the sentence as follows:\n\n“Georgiev et al. find that images generated from the same random noise $\\epsilon$ have little variations, even when they are generated by two independently trained diffusion models on the same dataset.”\n\n**Beginning of method section.** To resolve the confusion about L168-172, we clarify our formulation as follows:\n1. If we had infinite compute and a fixed budget of K images, we could search for every possible subset of K images and train models from scratch. If removing the set of K images leads to the most “forgetting” of the synthesized image, this set should be the most influential set.\n2. Of course, the above is impractical, so we simplify the problem by individually estimating the influence of each training point.\n3. One way to estimate the influence of a training image is to obtain a model that unlearns the training image. However, for attribution, it is expensive to run unlearning on every single training image.\n4. To resolve this issue, we instead apply unlearning to the *synthesized image* and then assess how effectively each training image is also forgotten as a result. It is much faster as we only need to run unlearning once. \n\nWe will reorganize the beginning of the method section to make this clear.\n\n**Citations**\n\n[1] Kumari et al. Multi-Concept Customization of Text-to-Image Diffusion.\n\n[2] Tewel et al. Key-Locked Rank One Editing for Text-to-Image Personalization.\n\n[3] Kumari et al. Ablating Concepts in Text-to-Image Diffusion Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "5MXTkcI4AY", "reviewer": "Reviewer_8wwb", "summary": "This paper proposes a novel method for data attribution in text-to-image diffusion models. The key idea is to unlearn a synthesized image by optimizing the model to increase its loss on that image, while using elastic weight consolidation to avoid catastrophic forgetting. The authors then identify influential training images by measuring which ones have the largest increase in loss after this unlearning process. The method is evaluated through rigorous counterfactual experiments on MSCOCO, where models are retrained after removing the identified influential images. It outperforms baselines like influence functions and feature matching approaches. The method is also tested on a benchmark for attributing customized models.", "questions": "1. How does the performance of the unlearning method vary with different sizes and types of datasets?\n2. How does the unlearning method perform in terms of efficiency compared with other methods?", "limitations": "Yes", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper addresses the issue of data attribution, which is important for understanding the behavior of generative models and the contribution of training data.\n2. The paper proposes an innovative and effective approach to data attribution that outperforms existing methods on counterfactual evaluations.\n3. The authors conduct extensive experiments, including computationally intensive retraining, to thoroughly validate the effectiveness of the proposed method.", "weaknesses": "1. The evaluations are limited to medium-scale datasets like MSCOCO. It's unclear how well the approach would scale to the massive datasets, such as LAION, which is used to train state-of-the-art text-to-image models.\n2. The method is computationally expensive, requiring many forward passes over the training set to estimate losses. This may limit scalability to large datasets.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful suggestions and feedback.\n### **Efficiency comparison with other methods**\n\nFollowing the reviewer’s suggestion, we compare our method’s efficiency with other methods. To proceed with the efficiency analysis, we provide a brief overview of each type of method.\n\n- **Our unlearning method**: we obtain a model that unlearns the synthesized image query and check the loss increases for each training image after unlearning.\n- **TRAK, JourneyTRAK**: precompute randomly-projected loss gradients for each training image. Given  a synthesized image query, the authors first obtain its random-projected loss gradient, and then match it with the training gradients using influence function. For good performance, both methods require running influence function on multiple pre-trained models (e.g., 20 for MSCOCO).\n- **Feature similarity**: standard image retrieval pipeline. Precompute features from the training set. Given  a synthesized image query, we compute feature similarity for the entire database.\n\nWe compare the efficiency of each method.\n\n**Feature similarity is the most run-time efficient** since obtaining features is faster than obtaining losses or gradients from a generative model. However, feature similarity doesn’t leverage knowledge of the generative model. Our method outperforms various feature similarity methods (Fig. 4 of the main paper).\n\n**Our method is more efficient than TRAK/JourneyTRAK in precomputation.**  Our method’s precomputation is much more efficient runtime and storage-wise compared to TRAK and JourneyTRAK. Our method only requires computing and storing loss values of training images from a single model. On the other hand, both TRAK and JourneyTRAK require pre-training extra models (e.g., 20) from scratch, and precomputing & storing loss gradients of training images from those models.\n\n**Our method is less efficient when estimating attribution score.** TRAK and JourneyTRAK obtain attribution scores by taking a dot product between synthesized image gradient and the stored training gradient features. The methods require calculating and averaging such dot product scores from the 20 pretrained models. On the other hand, as acknowledged in our limitations (Section 6 of the main text), though our model unlearns efficiently (e.g., only 1-step update for MSCOCO), getting our attribution score involves estimating the loss on the training set, which is less efficient than dot product search. Tradeoff-wise, our method has a low storage requirement at the cost of higher runtime.\n\nOur main objective is to push the envelope on the difficult challenges of attribution performance. Improving computation efficiency of attribution is a challenge shared across the community [1, 2], and we leave it to future work.  \n\n\n### **Scaling up data attribution to massive datasets**\n\nIn the community, scaling data attribution to massive datasets remains a huge challenge in terms of both algorithm and evaluation [1, 2]. Our solution to mitigate this issue is to:\n\n1. Provide the gold-standard leave-K-out evaluation on a moderately-sized dataset (MSCOCO)\n2. Test on a Customized Model Benchmark, focusing on attributing personalized/customized large-scale text-to-image models.\n\nWhile we don't have the tech-giant-level resources to run full LAION-scale experiments, the MSCOCO evaluation is already the most computationally extensive evaluation in this space. \n\n\n**Citations**\n\n[1] Park et al. TRAK: Attributing Model Behavior at Scale.\n\n[2] Grosse et al. Studying Large Language Model Generalization with Influence Functions.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZRmwH4Al4i", "reviewer": "Reviewer_UBLR", "summary": "The paper discusses how we can identify influential examples by unlearning the synthesized images in the text-to-image generation models. Unlearning synthesized images leads to increase in the loss for the most influential training examples for the generation of synthesized images. The paper relies on a well-known regularization for Fisher information in order to avoid catastrophic forgetting.\n\nThe paper evaluates image generation quality two ways:\n1) Retraining the model from scratch after removing the influential examples from the training set. A Poor synthetic image generation quality is seen as a proxy for identifying most influential training examples.\n2) Using ground truth attribution provided in a Customized Model Benchmark.", "questions": "1. Since \\hat{z} is not part of the training dataset, how do we know that its effect is unlearned appropriately from the model with the elastic weight consolidation loss ? \n2. How sensitive is \\hat{z} to subtle changes in the caption ? \n3. Have you considered comparing your work against DataInf: https://arxiv.org/abs/2310.00902 ?", "limitations": "The paper discusses the limitations of the work.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The authors did a great job telling the story, motivating the problem and discussing related work.\n+ The evaluation metrics and benchmarks are well-described. The authors show the effectiveness of their approach both qualitatively and quantitatively.", "weaknesses": "+ It is unclear how robust the synthetic image generation for a given caption is. It would be good to estimate the sensitivity to the subtle changes in the caption to the generation quality.\n+ Overall theoretical justification is not very clear. It is not clear how unlearning affects the overall utility of the model.\n+ Figure 2 mentions that their approach is qualitatively better than DINO and JourneyTRAK, but how can we tell that it is quantitatively better ? It might be that the proposed method is better for some examples and worse for others.\nIn the implementation details section, line 212, the paper suddenly starts mentioning DDPM loss without explaining what it is and how it is applied in their method. In the methods section there is no mention of it but instead the paper discusses a different loss, EWC loss. This is a bit confusing.\n+ On Figures 3 and 2 DINO is listed as a baseline against which the authors evaluated the proposed method. It is, however, not an influence function's approximation or unlearning method. It seems that the authors used it for Image similarity. At the same time they compare it also against Journey TRAK which is an influence approximation approach. This is a bit confusing.\n+ Figure 4: where exactly are the error bars in the plot ?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the helpful suggestions and comments.\n### **Sensitivity to subtle changes in captions**\nWe clarify that our work focuses on finding training images that influence a generated image, not how changes in captions affect generation quality. From our experience, the model is not very sensitive to subtle caption changes (e.g., misspell).\n\n### **Theoretical Justification**\nIn Appendix A, we show the theoretical connection between influence functions and our approach, both of which make approximations toward characterizing the effect of removing a training point and evaluating its effect on a synthesized (or vice versa). As described in L165-173 of the main text, a “perfect” attribution is intractable, since it requires combinatorially searching the set of influential images, and our methods and influence function both serve as “proxy” solutions to this. We validate our method through the leave-K-out evaluation, showing our \"proxy\" is more effective than baselines such as influence functions. We will clarify this in the text.\n\n### **How unlearning affects the overall utility of the model**\nWe would like to clarify that we use unlearning to attribute images **generated from the original model**. Hence, the unlearned model will not be used to generate images for end users but instead only serve as a part of the attribution algorithm.\n\nIn addition, in our general response G1, we analyze and report effectiveness of our unlearning method.\n\n### **How to say an attribution algorithm is better “quantitatively”?**\nWe provide quantitative evaluation in Figure 4 in the main text, and Figs. 7&8 in the supplementary, showing our method consistently outperforms various baselines.\n\n\n### **Mention of DDPM loss**\nDDPM loss is introduced by Ho et al. [3], which is the standard loss used to train diffusion models. This corresponds to $\\mathcal{L}(\\hat{z}, \\theta)$ in Eq. 4 of the main text. We will clarify this, include a detailed description of DDPM loss, and add a citation in the revision.\n\n\n### **Using DINO as a baseline**\nFor completeness, we indeed use an array of strong baselines, including feature similarity, as is standard practice [1, 2].  Our method outperforms similarity-based baselines in most cases (Fig. 4, 7, and 8 of the main text).\n\n### **Error bars in Figure 4**\nWe show standard errors in the top plot in Figure 4, but the error bars are very small and negligible. To clarify this, following reviewer ZhUS’s suggestion, we will report a table that includes the standard errors in our revision (see Tab. 1 in our response PDF, which summarizes Figs 4, 7, 8).\n\n### **How to know if synthetic content is unlearned appropriately?**\nPlease see G1 in our general response, where we quantitatively verify that the target image is forgotten, while other information is retained.\n\n### **Comparison to DataInf**\nAs suggested, below, we compare our work against DataInf, evaluating their performance with metrics proposed in Sec. 5.1 of the main text: leave-K-out model’s (1) Loss change and (2) deviation of generation. \n\n|  | Loss Change $\\uparrow$ (K=500) | Loss Change $\\uparrow$ (K=1000) | Loss Change $\\uparrow$ (K=4000) | MSE $\\uparrow$ (K=500) | MSE $\\uparrow$ (K=1000) | MSE $\\uparrow$ (K=4000) | CLIP $\\downarrow$ (K=500) | CLIP $\\downarrow$ (K=1000) | CLIP $\\downarrow$ (K=4000) |\n|---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n| DataInf | 0.0032±0.0001 | 0.0034±0.0002 | 0.0036±0.0002 | 0.038±0.004 | 0.039±0.005 | 0.045±0.005 | 0.78±0.02 | 0.78±0.02 | 0.77±0.02 |\n| Ours | **0.0051±0.0003** | **0.006±0.0005** | **0.0087±0.0006** | **0.054±0.006** | **0.055±0.004** | **0.059±0.005** | **0.75±0.02** | **0.69±0.02** | **0.62±0.02** |\n\nOur method significantly outperforms DataInf quantitatively. We show a qualitative example in Fig. 4 of the response PDF. DataInf often attributes images that are not visually similar, making them less likely to be influential to the synthesized images. We believe DataInf performs poorly because it is designed for LoRA fine-tuned models, rather than text-to-image models trained from scratch. According to DataInf’s paper, the method uses a matrix inverse approximation to compute the influence function efficiently, but this approach results in a pessimistic error bound. While DataInf’s authors mentioned that the error is more tolerable in LoRA fine-tuning, this error may not be acceptable for data attribution in text-to-image models trained from scratch.\n\nWe will add results and discussion in the revision. Thanks for the reference. Due to time constraints, we average over 20 synthesized image queries here, but will show full results with 110 queries in the revision.\n\n**Citations**\n\n[1] Wang et al. Evaluating Data Attribution for Text-to-Image Models.\n\n[2] Singla et al. A Simple and Efficient Baseline for Data Attribution on Images.\n\n[3] Ho et al. Denoising Diffusion Probabilistic Models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "YfysceV3jv", "reviewer": "Reviewer_YhLa", "summary": "The authors propose a new approach for data attribution for text-to-image models that utilizes machine unlearning. They then perform multiple experiments to demonstrate that the proposed method is competitive to other methods.", "questions": "- See weaknesses.", "limitations": "- The authors have adequately addressed the limitations.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is generally well-written, clearly structured, and easy to follow.\n- The problem of data attribution in text-to-image models is interesting.\n- Multiple experiments are conducted to verify the viewpoints.", "weaknesses": "- This is not really a weakness, but I wonder if the leave-K-out model can still generate a target image just under a different prompt. Perhaps a metric that see how hard it is to perform Textual Inversion [1] can be used?\n\nReferences:\n[1] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H. Bermano, Gal Chechik, Daniel Cohen-Or, “An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the feedback and suggestions.\n\n### **Can Textual Inversion be used for evaluation?**\n\nIn general response G3, we verify that Textual Inversion is not an ideal choice to check whether a concept is represented by leave-K-out models.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their helpful comments. We are happy that reviewers found  that our paper motivated the problem well (ZhUS, UBLR), provided an extensive literature review (ZhUS, UBLR), proposed interesting findings (ZhUS, YhLa), and conducted extensive experiments (8wwb, YhLa). We will provide clarifications to questions shared across reviewers here.

### **(G1) Effectiveness in Unlearning Synthesized Images (ZhUS, UBLR)**

Our attribution method relies on unlearning synthesized images, making it crucial to have an unlearning algorithm that effectively removes these images without forgetting other concepts. Following reviewers ZhUS and UBLR’s suggestions, we analyze the performance of our unlearning algorithm itself and ablate our design choices.

We construct experiments by unlearning a target synthesized image and evaluating:
- **unlearning the target image**: We measure the deviation of the regenerated image from the original model’s output—the greater the deviation, the better.
- **retaining other concepts**: We generate 99 images using different text prompts and evaluate their deviations from the original model’s output—the smaller the deviation, the better.

We measure these deviations using mean square error (MSE) and CLIP similarity. We evaluate across 40 target images, with text prompts sampled from the MSCOCO validation set.

We compare to the following ablations:
- **SGD** refers to swapping our method’s Newton update steps (Eq. 5 in main text) to the naive baseline mentioned in L174 of the main text, where we run SGD steps to maximize the target loss without EWC regularization.
- **Full weight** refers to running our Newton update steps on all of the weights instead of Cross Attn KV.

The following table, along with Fig. 2 of the response PDF, shows the comparison.

|  | Target MSE ($\uparrow$) | Target CLIP ($\downarrow$) | Other MSE ($\downarrow$) | Other CLIP ($\uparrow$) |
|---|---|---|---|---|
| SGD | 0.081±0.003 | 0.67±0.01 | 0.033±0.0004 | 0.83±0.002 |
| Full weight | 0.086±0.005 | 0.7±0.01 | 0.039±0.001 | 0.86±0.002 |
| Ours | **0.093±0.004** | **0.65±0.01** | **0.022±0.0004** | **0.89±0.002** |

As shown in the figure and the table, both our regularization and weight subset optimization help unlearn the target image more effectively, without forgetting other concepts.

### **(G2) Does leave-K-out models forget other images? (ZhUS)**

In Sec. 5 of the main paper, we show that leave-K-out models forget how to generate target synthesized image queries. Reviewer ZhUS raises an interesting question about whether these models forget unrelated images, too. Our findings show that the answer is a no: leave-K-out models forget only the specific concepts while retaining others.

Following reviewer ZhUS’s suggestion, we study how much leave-K-out model’s generation deviates from those of the original model in three categories:

- **Target images**: the attributed synthesized image. Leave-K-out models should forget these—the greater the deviation, the better.
- **Related images**: images synthesized by captions similar to the target prompt. We obtain the most similar 100 captions from the MSCOCO val set using CLIP’s text encoder. Leave-K-out models should not forget all of them—the smaller the deviation, the better.
- **Other images**: images of unrelated concepts. Prompts are 99 different captions selected from the MSCOCO val set. Leave-K-out models should **not** forget these—the smaller the deviation, the better.

In Fig. 1 of the response PDF, we find that the leave-K-out model “forgets” the query bus image specifically while retaining other buses and other concepts.

Similar to G1, we quantitatively measure deviations using mean square error (MSE) and CLIP similarity. We evaluate 40 pairs of target images and leave-K-out models. The following table shows the results.

|  | Target MSE ($\uparrow$) | Target CLIP ($\downarrow$) | Related MSE ($\downarrow$) | Related CLIP ($\uparrow$) | Other MSE ($\downarrow$) | Other CLIP ($\uparrow$) |
|---|---|---|---|---|---|---|
| K=500 | 0.054±0.004 | 0.719±0.0123 | 0.039±0.0003 | 0.862±0.0009 | 0.041±0.0003 | 0.788±0.0014 |
| K=1000 | 0.058±0.0043 | 0.675±0.0136 | 0.041±0.0003 | 0.855±0.0009 | 0.041±0.0003 | 0.788±0.0014 |
| K=4000 | 0.06±0.0036 | 0.612±0.0136 | 0.046±0.0004 | 0.831±0.0011 | 0.041±0.0003 | 0.787±0.0014 |


We find that target images have larger MSE and lower CLIP similarity than related images and other images. Also, as the number of removed influential images (K) increases, the target image error increases rapidly while other images stay almost the same. Interestingly, related images’ errors increase with larger K, but the errors are still much smaller than those of target images. This can be due to the fact that as K increases, the group of influential images can start affecting other related concepts.


### **(G3) Running Textual Inversion for Leave-K-Out Models? (ZhUS, YhLa)**

Both reviewers ZhUS and YhLa suggest using Textual Inversion to check whether leave-K-out models forget the target concept. In fact, we find that Textual Inversion cannot reconstruct a given image faithfully. 
In Fig. 3 of the response PDF, we run Textual Inversion on the original model, and the inverted results are variations of the reference images instead of faithful reconstructions. This finding coincides with the actual goal of Textual Inversion: generate variations of a user-provided image.

Hence, we believe that deviation of generation (delta G) is a suitable choice for evaluation. In the previous response (G2), we further analyze and validate this choice.

Meta-review: The paper received positive reviews with scores of 7, 6, 6, and 6. All reviewers praised its strengths and recommended acceptance, contingent on the authors implementing the changes outlined in the rebuttal. The meta-reviewers considered it for a spotlight presentation, but after further discussion and comparison with other submissions, we decided that a poster presentation is most appropriate. Congratulations on the strong work!
Decision: Accept (poster)