GMValuator: Similarity-based Data Valuation for Generative Models
Authors: Jiaxi Yang
Wenlong Deng
Benlin Liu
Yangsibo Huang
James Zou
Xiaoxiao Li
Venue: ICLR 2025 Poster (2025)
Keywords: Generative Model
Data Valuation
Primary area: interpretability and explainable AI

Abstract: Data valuation plays a crucial role in machine learning. Existing data valuation methods, mainly focused on discriminative models, overlook generative models that have gained attention recently. In generative models, data valuation measures the impact of training data on generated datasets. Very few existing attempts at data valuation methods designed for deep generative models either concentrate on specific models or lack robustness in their outcomes. Moreover, efficiency still reveals vulnerable shortcomings. We formulate the data valuation problem in generative models from a similarity matching perspective to bridge the gaps. Specifically, we introduce Generative Model Valuator (GMValuator), the first training-free and model-agnostic approach to providing data valuation for image generation tasks. It empowers efficient data valuation through our innovative similarity matching module, calibrates biased contributions by incorporating image quality assessment, and attributes credits to all training samples based on their contributions to the generated samples.  Additionally, we introduce four evaluation criteria for assessing data valuation methods in generative models. GMValuator is extensively evaluated on benchmark and high-resolution datasets and various mainstream generative architectures to demonstrate its effectiveness. Our code is available at: https://github.com/ubc-tea/GMValuator.

Published as a conference paper at ICLR 2025
GMVALUATOR: SIMILARITY-BASED DATA VALUATION FOR GENERATIVE MODELS
Jiaxi Yang Wenlong Dengl.s. Benlin Liu?, Yangsibo Huang', James Zou", Xiaoxiao Li
l University of British Columbia University of Washington *Princeton University Stannfor University Veccor Institute
ABSTRACT
Data valuation plays crucial role in machine learning. Existing data valuation methods, mainly focused on discriminative models, overlook generative models that have gained attention recently. In generative models, data valuation measures the impact of training data on generated datasets. Very few existing attempts at data valuation methods designed for deep generative models either concentrate on specific models or lack robustness in their outcomes. Moreover, efficiency still reveals vulnerable shortcomings. We formulate the data valuation problem in generative models from a similarity matching perspective to bridge the gaps. Specifically, we introduce Generative Model Valuator (GMVALUATOR), the first training-free free and model-agnostic approach to providing data valuation for image generation tasks. empowers efficient data valuation through our innovative similarity matching module, calibrates biased contributions by incorporating image quality assessment, and attributes credits to all training samples based on their contributions to the generated samples. Additionally, we introduce four evaluation criteria for assessing data valuation methods in generative models. GMVALUATOR is extensively evaluated on benchmark and high-resolution datasets and various mainstream generative architectures to demonstrate its effectiveness. Our code is available at: http://gihub.ccmmubbbeeeelll Valuutoor
INTRODUCTION
As the driving force behind modern AI, particularly deep learning Pei 2020). a substantial volume of data is indispensable for effective machine learning. On one hand, informative data samples relevant to the task at hand play critical role in the training process. On the other hand, due to data privacy concerns, personal data safeguarded by various regulations, including the General Data Protection Regulation (DDPR))Regulation (2018), and has become a valuable asset. Consequently, data valuation has garnered significant attention from academic and industrial sectors recently.
The intricate relationship between data and model parameters presents significant challenge in the contribution measurement of each training sample, thus making data valuation difficult task. Most existing data valuation studies focus supervised learning for discriminative models (e.g., classifica- tion and regression). These methods can be categorized as (1) Metric-based methods: Methods such as Shapley Value (SV) and Banzhaf Index (BI) Ghorbani & Zou (2019); Wang & Jia 2023) provide the assessment of data value by calculating marginal contribution on performance metrics (e.g., ac- curacy loss) through retraining the mode["] [[] (2) Influence- based methods: This line of methods measures data value by evaluating influence on model parameters of data points Jia et al. (2019b); Nohyun et al. (2022). (3) Data-driven methods: These techniques avoid retraining by leveraging data characteristics (like data diversity, generalization bound estimation, class-wise wise distance), though generally necessitate data labels. Xu et al. (202I); WW et al. 20022) Just et al. (2023).
Urgent needs for generative models. Data valuation in the context of generative models has NOT been well-investigated in the current literature. Moreover, there exist significant challenges in
'Equal contribution.
Work was done during visiting at the University of British Columbia 1 Corresponding to: Xiaoxiao Li <xiaoxiao. li ece.ubc.ca>
Exception for KNN-Shap Jia et al. (2019a)

---

Published as a conference paper at ICLR 2025
directly adapting the aforementioned data valuation methods for discriminative models listed to generative models: Firstly, the challenge of applying metric-based methods arises from lacking robust performance metries in generative models, in contrast the existence of commonly used metrics (e.g., accuracy or loss) in discriminative models Betzalel et al. (022). This could potentially result in inconsistent outcomes when employing various performance metrics [Terashita et al. (202T). In addition, the expensive cost of retraining requirements is another obstacle to using metric-based methods. Secondly, influence-based methods may not perform well on non-convex objective function of generative models Bae et al.] (2022). Besides, estimating the influence function in a deep generative model expensive, as it requires computing (or approximation) inverse Hessian. Thirdly, for data- driven methods, they mainly focus on supervised learning, which requires knowing data labels to quantify data values.
To the best of our knowledge, limited studies have explored model-dependent data evaluation using influence functions for specific generative models. IF4GAN Terashita et al. (202T) consider multiple evaluation metrics, such as log-likelihood, Inception Score (IS), and Frechet Inception Distance (FID) Heusel et al. (2017) to identify the most responsible training samples for the overall performance of Generative Ad Networks (GAN)-based models. However, selecting appropriate metrics is critical, as the results are inconsistent across metrics. VAE- Kong & Chaudhuri] (202T) finds the most significant contributors for generating a particular generated sample for Variational Autoencoders (VAF) using the influence function. However, viable Hessian estimation in influence function calculations incur high computational costs and this method cannot be easily generalized to other generative models.
Our motivation and goal. Considering the challenges posed by the selection of performance metrics and computational efficiency and the need for broader applicability across various generative models, we aim to propose a unified and efficient data valuation method. We expect the unified method to satisfy the following key properties: 1) Model Model-agnostic: the method should be versatile, capable of being employed across diverse generative model architectures and algorithms,, regardless of specific design choices; 2) Computation Efficiency: the method does not require retraining the model and minimize computational overhead while maintaining a satisfactory level of accuracy and reliability in evaluating the value of data points; 3) Plausibility: the method should evaluate the value of data based on its alignment with human prior knowledge on the task, enhancing its credibility and reliability; 4) Truthfulness: the method should strive to provide
unbiased and accurate assessment of the value associated with individual data points. In this work, we evaluate the contribution of (good) training data given a fixed set of generated data from a certain well-trained generative model. We refer to the value of training data as the contribution to the given generated data, taking into account the quality of the generated data. To meet the design purposes, we propose similarity- data valuation approach Rl for the generative model, called Generative Model Valuator (GMVALUATOR), which is model aggostic and efficient.
In summary, our work here provides the following specific novel contributions:
To the best of our knowledge, GMVALUATOR the first modal modal-agnostic and retraining- free data valuation method for image generative models.
We formulate data valuation for generative models as an efficient similarit-matchinn problem. We further eliminate the biased contribution measurement by introducing image quality assessment for calibration.
We propose four evaluation methods to assess the truthfulness of data valuation and evaluate GMVALUATOR on different datasets (including benchmark datasets and high-resolution large-scale datasets) and various deep generative models to verify GMVALUATOR's 's validity.
Related Work. Different from discriminative models for regression or classification tasks, generative models in various forms (i.e., Variational auto-encoders (VAEs) Rezende et al. (2014), Variational auto-encoders (VAEs) Rezende et al. (2014), Diffusion Model Ho et al. (2020); Rombach et al. (2022) aim to learn data distribution for generation tasks. Consequently, the approach to data valuation in our work, which focuses on generative models, is distinct and orthogonal to that for discriminative models. The primary limitation of both metric-based methods and influence- based methods is the expensive computation cost, a drawback that is further magnified in generative models. Conversely, while
We detail why the similarity-based approach is formed in See 2 and explain how it is implemented in Sec
2

---

Published as a conference paper at ICLR 2025
data-driven methods training-free free, they predominantly concentrate supervised learning and data Xu et al. (2021); Wu et al.] (2022); Just et al. (2023). Apart from the expensive computational cost, the limited existing influence-basee can not adapt to the diverse and evolving trends in generative model development [Terashita et al.
methods tor generative models are model-specific,, which
(202T): Kong & Chaudhuri (202T). A more detailed related work is discussed in Sec. [C of the appendix.
2
MOTIVATION AND PROBLEM FORMULATION
In this section, we present the background and motivation behind our proposed method GMVALU TOR, which formulates the data valuation for generative models as a similarity-matching problem.
2.1 MOTIVATIONS
Let us denote X {#1, In En| }} as the training dataset without duplicated data for generation task, and is regarded the size of the dataset. Denote the generated dataset as == {f. by well trained generative model G (e.g., Generative Adversarial Network (GAN), Variational Auto-encoder (VAE), Diffusion Model) on X.
The core of these generative models is intractable probability distribution learning process to train a generator G, that maps each 2 sampled from the noise distribution Z to a generated sample & in estimated distribution by maximizing the likelihood px J pe (=)==)==(=)dd This can be achieved by minimizing the distance d(.,-) (ie., Wasserstein Distance) between the estimated distribution and training data distribution:
G -arg (
G
Therefore, the data generated by an optimal generator, which closely approximates the training distribution, can be considered as drawing from a subdistribution of X.
2.1.1 THEORETICAL JUSTIFICATION
Let I' C X denote the K contributors found by data valuator (e.g., GMVALUATOR) for X, and is a subset of the training dataset X. We focus on the data type with describable attributes (e.g., image data with semantic attributes). We first provide definitions of data and contributors.
Definition 2.1 (Data with Describable Auributes.) We assume an image can be characterized by V attributes, and there is labeling function mapping S, T and X to the same attribute space A = 10,
Definition 2.2 (Contributors in Generative Models.) Let S denotes the real Kcontributors, defined as follows:
S =rg min dl (x)
(2)
3CX
where | = K and K' is reasonable number for generative model training. X(x(a) is the data
distribution of generated data X on attributes A E A. and XGG(S)) A) is the distribution of data with attribute that are generated by the optimal generator trained by contributors S. According the objective of generative models and Eq. equation 2 we have X ((X)A) X(G(S))AA-- We follow Just al. 2023) assumptions and lemmas that will be used to obtain the theorem.
Assumption 2.3 Assume that the function f is c-Lisschitz and the loss function L {0, 13V x [0, 1] + R+ is k-Lipschitz both inputs and attributes. We have labeling functions that are all bounded by V as [| V...
Then, we introduce the error bound between T and S*
Theorem 2.4 (Bounded Attributes Classification Error on S* to T.) Let fs. : A= (0. 13V be the model trained the optimal contributor dataset S Following Assumption 2.3 the contributors
3

---

Published as a conference paper at ICLR 2025
corresponding to the given generated data X, we have:
[c (f(()).fs (x)) Eawus- ( (s(1))).
Exwur
Ske [dw(2aan)) {(1n) (d ((nn)
As shown in the Theorem 2.4 given the optimal contributor S dw (Ms-in) is deter- ministic term that approaching zero. Then, approximate S can be achieved by reducing the distance term dw ((ris),xx11)) Please see Sec. M in the appendix for the proof.
2.1.2 EMPERICAL VALIDATION
According theoretical justification, the generated data should show more similarity to the data samples used for training. In this context, examine the value of training data being more valuable compared to data that is not used for training, despite originating from similar distribution. We support this by partitioning a class of CIFAR-10 (the class is plane here) into non-overlapped subsets, de- noted as X₂1 and X 3 Next, we keep Xu1 as non-training data and use X22 training data train a BigGAN Brock al. (2018) and generate dataset X. If our assumption holds, the generated data will be more similar to the train- ing data X,2. As presented in the T-SNE Van der Maaten & Hinton (2088) plot (Figure TT. the generated data demon- strate a more substantial overlap in the shared feature space with the training dataset X than the non-training ones X.1 This observation motivates to address the data valuation for the generative model from a Sallaarttt-mtthinng perspective.
generated
Figure Data distribution for X and X for CIFAR-10, and Xx1,X.2 both airplane dataset.
2.2 PROBLEM FORMULATION AND CHALLENGES
The primary objective of our research is to tackle the issue of data valuation in generative models using black-box access. In essence, given fixed set of data samples X with a size of m generated from the well-trained deep generative model G*, our aim is to determine the value dz(x;, X, G*) associated with each data point I; E X for i E [n] in the deduplicated training dataset, which contributes to the generated dataset. We denote the value for training data 2; di in the rest of the paper for simplicity.
Following the motivation stated in Sec. 2.1 points. We denote the distance between training data 2; and generated data i.y as dij. for i E [n] and j E [m].
Φi should be function of the distance between the data
Definition 2.5 (Primary Contribution Score.) The contribution score of x; to ł is denoted as
V(a, 2 xx d which is inversely proportional distance since maximizing the log-likelihood of generative model is equivalent to minimizing the dissimilarity of real and generated data distribution. To link the dissimilarity -d3 to likelihood, we choose exp(-d;;) to likelihood
exp exp (-di))
(t) = Σ, exp (-diz)
Therefore, an intuitive definition for the value of data sample 2; can be written as below.
IIn practice, we perform T-SNE first to randomly split the non-overlapped samples from embedding space.
Quantitative statistical testing is provided Appendix.
4

---

Published as a conference paper at ICLR 2025
Pz
Data Value
3.6
Training Samples
Value Calculation
Generative Model
1. Efficiont Similarity Matching
du
exp(-de) Mas eep(
=
(.a())
2. Image Quality Assesament
(gy Oor Pay)
Generated Samples
www.
Elficient
MfliclantSimiiariyy Similarity Matching
Image Quality Assessment
Training
Perank
Festure Map Concat
Attention
Transposedd
Data
Distance Table
Encodar
Vision Transformar
Conv
Codebook
Embedding Vector
PQ Coda
Generated
Data
images
Score
Figure 2: Overview of GMVALUATOR, unified and training-free free data valuation approach for any generative models. GMVALUATOR contains three important modules - (1) Efficient Similarity Matching (ESM), (2) Image Quality Assessment, and (3)Value Calculation. Each generated data ④. is matched with training data through ESM approach, resulting in the distances with its top k contributors. The normalized contribution score from training sample æ; to J. defined as exp( / Σ (xx((daa) ). is adjusted based on the quality associated generated samples q5. We compute the data value di of each training sample 2; by summing its contributions to the generated samples, where it ranks among the top & contributors.
Definition 2.6 (Data Value.) The contribution of each training data point I; E S* for the generation of dataset X equals to the sum of its contributions to each generated data point &; in X.
Φ. = Σ (xr. E EX,,, E.. 1=1
(5)
The high-value data will achieve a smaller distance to the target distributions, thus, a better approx- imation. Therefore, data valuation the above problem formulation contains two steps: Step calculating all of the score value V( R2); Step 2, mapping the contribution from training data to generative data based on the scores V.
However, there are several open questions and challenges in performing the above two steps for calculating Eq. equation [5]
Challenge 1: Efficiency. In step 1, considering n training samples and m generated samples, where O(C) represents the complexity of the selected pair-wise distance calculation, the total complexity of this step amounts to O(mnC'). In practical scenarios with large training datasets (e.g., n 10K), the computation cost becomes prohibitively expensive. Additionally, fitting such large collection of high-dimensional data for distance calculation can pose significant challenges in system memory. Challenge 2: Contribution plausibiliry. To ensure that training data point contributes more if it similar to high-quality generated data and less it is similar to low-quality generated data, the contribution scores should be adjusted based the quality of the generated data.
Challenge 3: Non-zero scores. In practical scenarios, the distance between training data and the least similar generated data not infinite, which may result in false non-zero contribution scores. With large dataset size, the accumulation of these noisy scores yields biased data valuation.
In light this, we present a novel and efficient data valuation approach suitable for agnostic generative models, termed as GM VALUATOR as elaborated in Section
3 THE PROPOSED DATA VALUATION METHODS
The crucial idea behind GMVALUATOR is to transform the data valuation problem into similarity- matching problem between generated and training data. The overview GMVALUATOR is presented in Figure 2 To tackle challenge 1, propose employ efficient similarity matching (ESM), where
5

---

Published as a conference paper at ICLR 2025
each generated data point can be linked to multiple contributors from the training dataset (Section 3.1 Each generated data firstly is linked with top contributors via recall phase and then re-ranked by refined similarity for effectiveness. After that, the image quality of the generated sample is assessed to weigh the valuation (Sec. (32)). Finally, the value computation function combines both the quality score and the image-space similarity score to measure the value of the training data (Sec. 3.3)
3.1 EFFICIENT SIMILARITY MATCHING
Considering the complexity of calculating V(=;; :3). we formulate it as an ESM problem between generative data and training data. Each generated data sample is matched several training data sam- ples based on their similarity. We denote P, = {01, 22; ... h4} f(X, 2,) as the subset of training data that contains the k < most similar data samples. Here, represents the similarity-matchiie strategy, encompassing the recall and re- re-ranking phases, which will be introduced subsequently.
The main aim of the recall phase is to rapidly identify a subset of training samples that are similar to generated sample. To achieve this, an initial step involves encoding all original training images and generated samples from the image space RC to a lower-dimensional embedding space Rº using an pre-trained encoder fe (such as CLIP Radford et al. (202)) to reduce computational complexity. Subsequently, the technique of Product Quantization (PQ) [Jeoou et al. (2010) is employed to further decrease the computational burden. Specifically, PQ divides embedding vectors into subvectors and independently quantizes each subvector through Q-means clustering. This process generates compact PQ codes that serve representations of the original vectors. This representation significantly reduces the vector sizes, allowing for efficient estimation of Euclidean Distance between two samples. By incorporating the recall method into the similarity matching process, the computational complexity is lowered from O(mmC) to O(mQD), where D < C and Q < n. Consequently, generated images can quickly identify their top-k most similar training data samples.
Recall Phase:
Re-Ranking Phase: Following the PQ-based efficient recall process in GMVALUATOR, we further improve the precision of the results by utilizing perceptual similarity Fu et al. (2023) for precision ranking. Once we have extracted perceptual features from the top recalled training samples, we proceed to calculate the distance for each pair of items. To obtain precise distance measurements based on their perceptual content, we propose to use Learned Perceptual Image Patch Similarity (LPIPS) Zhang et al. (2018) or DreamSim Fu et al. (023) as the distance measurement d to gain insights into the perceptual dissimilarity between the generated sample and different training samples. These metrics enable us to precisely measure the most significant contributors according to their perceived similarity, more importantly, the obtained distance d will be employed to compute data valuation in Eq. equation 6 We do not use Wasserstein distance it is more proper to measure dissimilarity between two probability distributions rather than a pair of image instances with semantic characteristics.
3.2 IMAGE QUALITY ASSESSMENT
To tackle challenge 2, we have to establish a connection between the quality of the generated sample that the training samples contribute and the contribution score. This is necessary before assigning contribution scores to the ranked training samples for the generated sample. For low-quality generated
samples dlow. we expect their total contribution score from contributors be lower compared to
high-quality samples &high. namely Σi(()) V(z flow) ) < Σie(H) V(z fighh) This motivation
arises from the observations in Figure 3] where the first and second rows show generated samples using from a normal distribution and uniform distribution, respectively. The values denoted in Figure directly employs Eq equation 4 so we have Ex.ep V(s,, 2) = for Vi E X. The real samples noticeably dissimilar to the generated data, yet they still assigned high value.
Therefore, we propose to calibrate the contribution scores with the generated data quality. Specifically, we obtain comprehensive quality score di E [0, for each generated image integrated using MANIQA Yang et al. (2022) model into our evaluation process. A higher q, indicates better data quality. This score of MANIQA considers various factors such as sharpness, color accuracy, composition, and overall visual appeal. We also provide the necessity of this calibration in the appendix [ Incorporating the image quality evaluation provided by MANIQA allows more

---

Published as a conference paper at ICLR 2025
Generated Image
Value: 0.2628
Value: 0.2483
Value: 0.2455
Valust 0.2425
Generated
Value: 0.2703
Value: 0.2487
Value: 0.2417
Value: 0.2383
Figure 3: The value without generated image quality calibration for high-quality image (top row) and a low-quality image (bottom row). Column 1: generated images. Column 2-5:their top contributors.
accurately assess the generated images and take into account their perceptual fidelity and aesthetic qualities.
3.3 VALUE CALCULATION
Finally, we utilize the image quality g to calibrate the contribution scores. Notably, during the recall phase of similarity matching, select the top le contributors P, for the generated data ãz. Consequently, we only consider the contribution score between Pj and ă; by setting V(r,, .3) 0 for v; Py, This strategy addresses challenge 3 by assigning effectively reducing bias and noise in value estimation when dealing with a large n. Hence, we define the contribution score of each training data point for a specific synthetic data point follows:
scores irrelevant samples,
{
9i
(-d,) Et exp (-d,,)
I3 E P
V(x,, &t, dey 9z)
(6)
Is $ P
which is used to give different scores to training data points according to their ranking distance. The final data value is obtained by plugging Eq. G|ino Eq.
EXPERIMENTS
4.1 EXPERIMENT SETUP
Datasets. Our experiment settings are listed in Table [ Partieularly, consider different types of generative models, including GAN based models, 3- Higgins et al. (2016), and diffusion models Ho & Salimans (2022); Wang al. (2022). The generation tasks are conducted on benchmark datasets (i.e., MNIST LeCun et al. (1998) and CIFAR Kriizhvsky et al. (2009), face recognition dataset (i.e., CelebA Liu al. (2018) 8)), high-resolution image dataset with size 512 x 512, and 1024 1024 (i.e., AFHQ Choi et al. (2020), FFHQ Karras et al. (2019)), the large-scale dataset with 1,000 classes and 14,197.12 122 images (i.e., ImageNet Deng et al. (2009)), and text-to-image dataset
(i.e., Naruto Cervenka (2022)).
Table 1: Evaluation Setup.
Dataset
Network
MNIST, CIFAR-10
GAN, Diffusion Models, 3-VAE
C1
ImageNet
Masked Diffusion Transformed Gao et al. 2023
CelebA
biffusion-SStleCi..
C2
AFHQ, FFHQ
StyleGAN
Naruto
Stable Diffusion
C3
MNIST
DOGAN
Hardware Evironment
GPU
One RTX 3080 (IUGB) GPU
12 vCPU Intel(R) Xeon(R),
CPU
Platinum 8255C CPU 2.50GlIz
7

---

Published as a conference paper at ICLR 2025
Baselines. Note that GMVALUATOR is model-agnostic method. To showease its efficacy, compare it with two bascline methods: VAE- TracTra TracIn/Kong & Chaudhuri (2021 and IF4GAN Terashita
et al.. (222I) VAE-Trachn identifies the most significant contributors for a generated sample, which is the same as the similarity matching process of GM
(Sec. (.1). Thus, we compare a.3 and Sec. 4.4 IF4GAN finds the high-valued training samples for generative model training, which we regard it as the baseline method in section 4.5 We also examine our approach using DreamSim, LPIPS and l2-distance in the re-ranking phase, referred to as GM (DreamSim), GMValuator (LPIPS) and GMValuator (lz-distance) respectively. while the GMValuator without the re-ranking phase is referred to as GM GMValuator (No-Rerank).
GMVALUATOO with VAF-TracIn in Sec. S.3.
Embedding Appoarch. In this experiment, we utilize CLIP as an example choice for pre-trained encoder fe the recall phase of efficient similarity matching. To demonstrate the generalizability of GMVALUATOR we also do experiments using more embedding approaches in Sec. G of the appendix.
4.2 METRICS
Given the absence of definitive benchmark to evaluate data valuation methods in the context of generative models, we propose to evaluate the truthfulness of data valuation outcomes through the examination of four criteria (C):
C1: Identical Class Test. Following the concept of an identical class Hanawa et al. (2020), we posit that the most significant contributors among the training samples should belong the same class as the generated sample produced by well-trained generative model, which is also used for evaluation in
Kong& & Chaudhuri (202T).
C2: Identical Attributes Test. Following C1, we extend the concept of identical class test and consider the attributes of samples as the ground truth to examine the most significant contributors by our approach on datasets without class labels. We examine the overlap level of attributes (i.e, the number of identical attributes) between the most significant training samples with the generated sample.
C3: Out of Distribution Detection. Under the assumption, the out of distribution training data samples (e.g., noisy samples) be identified as low-contribution training samples for the generated dataset. To evaluate the performance of data valuation approaches, we consider the contribution level of noisy data samples
the performance metric across various approaches.
C4: Efficiency. As an efficient training-free approach, GM VALUATOO should measure the data value in limited time and be much more efficient than the previous work while obtaining truthful results.
4.3 IDENTICAL CLASS TEST (CI)
Methodology. In this subsection, we follow setup of VAE-Trachn Kong & Chaudhuri (202T) by training separate 3-VAD models on KrizheeKii et al. 2009), CIFAR LeCun et al. (1998).
We then attribute the most significant contributors by VAE-TracIn and our approach over generated samples. By the concept of identical class test, we expect perfect data valuation approach should indicate training data in the same subclass of the generated sample I, contribute more to ēy. We examine GM GMVALUATOR (DreamSim), GMVALLUTOR (LPIPS), GMValuator (lz-distance) and GMVALUATOR (No-Rerank) separately. In addition to using VAE and comparing it with the baseline method, we also perform experiments using masked diffusion transformer on the large-scale dataset (ImageNet) and other various generative models (in the appendix) to demonstrate the model-agnostic property of GMVALUATOR.
Results. For a given generated data, we examine the class(es) of is top k contributors in the training data. We count the number of training Q samples in the top contributors, which have the identical class as the generated data. The identical class ratio = Q/ k. We report the averaged p over the generated datasets (the data size m=100) on different choices of le in Table 2 the most contributors that belong to the same class with the generated sample among on MNIST and CIFAR-10, while GMVALUATOR (DreamSim) has the best performance. The results for CIFAR- 10 using VAE-TracIn are extremely bad. This could be attributed to underfitting, as the authors mentioned in their study Kong & Chaudhuri (2021), ). training a good 3- VAE model on CIFAR-10 is challenging. In addition, GMVALUATOR (DreamSim) shows significant advantage on ImageNet since it considers semantic characteristics while 12 disttnee or LPIPS destroy the performance, while the experiments conducted ImageNet using baseline method (VAE-Trach) is impractical due to
GM VAVAAAAOO has
8

---

Published as a conference paper at ICLR 2025
its unacceptable computational overhead. In the appendix. further analysis and explanation for this phenomenon are discussed and the results from various generative models are presented as well.
Table 2: Performance comparison of Identical Class Test (CI).
MNIST (%)
CIFAR-10 (%)
ImageNet (%)
Top-k
k=
k=50
k=100
k=30
k=50
k=100
k=30
k=50
k=100
VAE-TracIn
72.00
71.11
68.58
6.28
3.77
GMValuator (No-Rerank)
86.41
85.13
85.95
72.66
72.25
70.71
56.18
53.60
48.17
GMValuator (lz-distacce)
87.76
86.95
85,92
72.66
72.25
70.71
42.44
40,51
39.00
GMValuator (LPIPS)
88.78
88.19
86.69
72.60
71.75
70.84
50.30
47.51
45.13
GMValuator (DreamSim)
88.78
88.05
86.84
77.94
76.41
73.47
77.27
70.90
58.89
4.4 IDENTICAL ATTRIBUTES TEST (C2)
Methodology. In extension to C1, we focus on certain image attributes instead of class labels as the ground truth. We posit that the most significant contributors a generated sample should share similar attributes with it. We train Diffusion-Styleds our approach to identify the most significant contributors in training samples and use some attributes including hat, gender, and eyeglasses the ground truth, to check the correctness of our approach for identifying the most significant contributors for a generated sample. We also examine this on AFHQ.
Wang et al. (022) on CelebA, and leverage
FFHQ dataset by StyleGAN Choi et al. (2020), and text-to- text-to-image dataset Naruto Cervenka (2022) by stable diffusion model Rombach et al. (022).
Results. The results in Table indicatee that GMVALUATOR can find the most significant contributors with the same attributes to the generated sample. Besides, using DreamSim in the similarity re- ranking phase obtains the best performance over these three attributes. We also visualize some visible results in Figure CelebA, and AFHQ, FFHQ in the appendix. As we can see, the most significant contributors have similar attributes (e.g., skin color, hair) to the generated sample. Similarly, the visualized results AFHQ and FFHQ shown in the appendix demonstrate that the top A contributors have similar attributes to the generated sample such the fur color of cats or dogs AFHQ or hair color in FFHQ. In addition, the results on the text-to sottto--mmge dataset Naruto Cervenka (202 corroborate the same conclusion, as shown the appendix N
Table 3: Performance of Identical Attributes Test (C2) of some attributes including Hat, Gender, and Eyeglasses on CelebA.
Attribute (%)
Hat
Gender
Eyeglasses
Top contributors:
k=5
k=10
k=15
k=10
k=15
k=5
k=10
k=15
GM Valuator (No-Rcrank)
92.52
92.12
91.92
75.15
75.25
75.82
91.52
91.52
91.45
GMValuator (ly-distance)
90.91
89.90
89.83
63.64
61.01
60.00
94.95
94.65
93.80
GMValuator (LPIPS)
96.77
96.57
96.90
97.98
97.48
97.37
94.95
94.65
93.80
GMValuator (DreamSim)
97.78
97.07
96.90
99.19
98.99
98.79
96.77
96.26
95.96
GHVVllstoriDreammiii GMValuater(LPIIS)
GHVallutorri2Distancee
GHValuatorNNon-eenaa rerankk)
IF4GAN
4000
Value
6000
8000
10000
Figure 5: The y-axis represents the ranking of values from high to low, with the top being the highest value and the bottom being the lowest value. The x-axis represents the index of each noisy data.
Figure 4: Visualization of Identical Attributes Test on CelebA. Left: generated samples. Right: top k contributors.

---

Published as a conference paper at ICLR 2025
4.5 OUT OF DISTRIBUTION DETECTION (C3)
Methodology In this experiment, we introduce 100 noisy images into the MNIST dataset create a new corrupted training dataset and each noisy image is generated by adding Gaussian noise to the clean data. These contaminated samples can diminish the performance of generative models, and as result, they are anticipated to possess lower values compared to the rest of the training dataset. Our objective to assess the performance of both GMVALUATOR and its baseline method, IF4GAN Terashita et al. (2021), by analyzing the value rankings of noisy samples. An effective approach should place noisy data in lower positions within the value ranking. We conducted this evaluation using 10,000 generated images and referred to the study Terashita et al. [202T) for the baseline method (IF4GAN).
Results. The results depicted in Figure 5 demonstrate that the values of all noisy training samples, as calculated by GMValuator, are lower compared to the values calculated by IF4GAN. This observation suggests that the performance of GM VALUATOO is significantly better than that of IF4GAN. Besides, the performance of GMVALUATOR with the re-ranking phase is better than GM VALUAATOR (No- rerank).
4.6 EFFICIENCY (C4)
Methodology. We thoroughly evaluate and compare the efficiency of our approach against existing data valuation methods for generative models discussed above: VAE-TracIn and IF4GAN. Specifically, we measure the attribute time for one generated sample on an average of 100 test samples using both VAE-TracIn and our proposed approach on MNIST and CIFAR-10 10. Furthermore, under the same setting in C3, we assess the data valuation time when utilizing IF4GAN and our approaches on noise MNIST which we mentioned in C3. In addition, we analyze our computational efficiency as the data scale grows in the appendix O
Results. The average time taken to attribute the most significant contributors for one generated sample is calculated and compared with the bascline method approaches are all greatly more efficient than the baseline methods on both datasets. When comes to data valuation time for GAN on C3, GMVALUATOR (No-rerank) and GM VALUATR (LPIPS) are significantly better than IF4GAN. The aforementioned phenomenon is attributed to the costly nature of the Hessian estimation process, despite the utilization of certain acceleration methods. is noticeable that GMVALUATOE (DreamSim) is lightly time-consuming when compared to IF4GAN. This lead to a trade-off problem due the GMVALUATOR using DreamSim in the re-ranking phase obtained the better performance from C1 to C3.
as demonstrated in Table 4 Our
Overall, through the above experiments, GMVALUATOR outperforms than baseline methods, while CAVAORATOR (DreamSim) obtains the best performance. This is due to the fact that DreamSim captures mid-level similarities in image semantic content and layout compared to LPIPS. And the selection of different GMVALUATOR can be combined with the consideration of their efficiency. Furthermore, the results shown in the appendix O also demonstrate that our method maintains consistent computational efficiency even as the data scale increases and presents its scalability on large-scale datasets.
Table 4: Efficiency Comparison
GM
Attribute for VAE
VAE-TracIn
No-Rerank
ly-distance
LPIPS
DreamSim
MNIST CI
47.945
0.250 0.755
0.339
0.477
1.709
66.178
1.226
2.412
15.491
Valuttoor
Data Valuation for GAN
IF4GAN
No-Rerank 2,137
L didittance
LPIPS 4,771
DreamSim
Noise MNIST
14,543
3,388
17,086
5 CONCLUSION
To measure the contribution of each training data sample, we propose an efficient approach, GM- VALUATOR, for generative model. As far as we are aware, there is no prior model-agnostic and training-free free data valuation approach for generative models util GMVALUATOR. Our approach is based on efficient similarity matching, and it enables us to calculate the final value of each training data point, aligning with plausible assumptions. The proposed method is validated through a series of comprehensive experiments to showcase its truthfulness and efficacy on four criteria. In the future, we will validate the proposed methods on other data modalities.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENT
This work is supported in part by Natural Science and Engineering Research Council of Canada (NSERC), Canada CIFAR AI Chairs program, Canada Research Chair program, MITACS-CIFAR Catalyst Grant Program, the Digital Research Alliance of Canada, Institute of Information & com- munications Technology Planning & Evaluation (IITP) grant funded by the Korea govermment (MSIT).
REFERENCES
Juhan Bae, Nathan Ng, Alston Lo, Marzych Ghassemi, and Roger B Grosse. If influence functions the answer, then what is the question? Advances in Neural Information Processing Systems, 35:1 1551733117 2022.
Eyal Betzalel, Coby Penso, Aviv Navon, and Ethan Fetaya. A study on the evaluation of generative models. arXiv preprint arXiv:2206.109335 2022.
Akhilan Boopathy, Kevin Liu, Jaedong Hwang, Shu Ge, Asaad Mohammedsaleh, and Ila Fiete. Model-agnostic measure of generalization difficulty. In International Conference Machine Learning, pp. 2857-2884. PMLR, 2023.
Andrew Brock, Jeff Donahue, and Karen Simonyan, Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: 1006, 2018.
Fole Cervenka. Naruto blip captions.
httss//wnginggaaacooooddevvvaaa
naruto- catpptoolip-captioon// 2022.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Wor Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8188-8197, 2020.
R Dennis Cook. Detection of influential observation in linear regression. Technometrics, 19(1):15-18, 1977.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. leee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8794, 2021.
Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data. arXiv preprint arXiv:2306. 003444..9994.. 2023.
Shanghua Gao, Pan Zhou, Ming-Ming Cheng, and Shuicheng Yan. Masked diffusion transformer is strong image synthesizer. arXiv preprint 2rXiv:2303.11389, 2023.
Amirata Ghorbani and James Zou. Data shapley: Equitable valuation of data for machine learning. In International Conference on Machine Learning, pp. 2242-2251. PMLR, 2019.
Amirata Ghorbani, Michael Kim, and James Zou. A distributional framework for data valuation. In International Conference Machine Learning, pp. 3535-3544, PMLR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley. Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):139-144, 2020.
Kazuaki Hanawa, Sho Yokoi, Satoshi Hara, and Kentaro Inui. Evaluation of similarity-based explanations. arXiv preprint arXiv:2006.04555 2020.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to a local nash equilibrium. Advances in neural information processing systems, 30, 2017.
11

---

Published as a conference paper at ICLR 2025
Irina Higgins, Loic Matthey, Arka Pal, Christopher Burgess, Xavier Glorot, Matthew Botvinick, Shakir Mohamed, and Alexander Lerchner. beta-vae: Learning basic visual concepts with a constrained variational framework. In International conference learning representations, 2016.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12590 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Herve Jegou, Matthijs Douze, and Cordelia Schmid. Product quantization for nearest neighbor search.
IEEE transactions pattern analysis and machine intelligence, 33(1):117-128, 2010.
Ruoxi Jia, David Dao, Boxin Wang. Frances Ann Hubis, Nezihe Merve Gurel, Bo Li, Ce Zhang, Costas J Spanos, and Dawn Song. Efficient task-specific data valuation for nearest neighbor algorithms. arXiv preprint arXiv: 096.08 2019a.
Ruoxi Jia, David Dao, Boxin Wang, Frances Ann Hubis, Nick Hynes, Nezihe Merve Gürel, Bo Li, Ce Zhang, Dawn Song, and Costas J Spanos. Towards efficient data valuation based the shapley value. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 1167-1176. PMLR, 2019b.
Hoang Anh Just, Feiyang Kang, Jiachen T Wang. Yi Zeng, Myeongseob Ko, Ming Jin, and Ruoxi Jia. Lava: Data valuation without pre-specified learning algorithms. arXiv preprint arXiv:2305.00054, 2023.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8110-8119, 2020.
Zhifeng Kong and Kamalika Chaudhuri. Understanding instance-bai based interpretability of variational auto-encoders. Advances in Neural Information Processing Systems, 34:2400-2412, 2021.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86( 36(11))2278-2324 1998.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Large-scale celebfaces attributes (celeba) dataset. Retrieved August, 15(201 (12201):11, 1,,
Ki Nohyun, Hoyong Choi, and Hye Won Chung. Data valuation without training of model. In The Eleventh International Conference on Learning Representations, 2022.
Jian Pei. A survey on data pricing: from economics to data science. IEEE Transactions on knowledge and Data Engineering. 34(10):4586 4608, 2020.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data
science. Foundations and Trends in Machine Learning, 11(5-6):355-677, 2019.
Garima Pruthi, Frederick Liu, Satyen Kale, and Mukund Sundararajan. Estimating training data influence by tracing gradient descent. Advances in Neural Information Processing Systems, 33: 19920-19930, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International conference on machine learning, pp. 8748-88 87633 PMLR, 2021.
Protection Regulation. General data protection regulation. Intouch, 25, 2018.
12

---

Published as a conference paper at ICLR 2025
Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. Stochastic backpropagation and approximate inference in deep generative models. In International conference on machine learning. pp. 1278-1286. PMLR, 2014.
Adam Richardson, Aris Ratss-Ratsikss, and Boi Faltings. Rewarding high-quality data via influence functions, arXiv preprint arXiv: 11598, 2019.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Nikunj Saunshi, Arushi Gupta, Mark Braverman, and Sanjeev Arora. Understanding influence functions and datamodels via harmonic analysis. arXiv preprint arXiv:2210.01072,200 2022.
Naoyuki Terashita, Hiroki Ohashi, Yuichi Nonaka, and Takashi Kanemaru. Influence estimation for generative adversarial networks. arXiv preprint arXiv:2101.08367, 2021.
Laurens Van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of machine
learning research, 911)), 2008.
Jiachen T Wang and Ruoxi Jia. Data banzhaf: A robust data valuation framework for machine learning. In International Conference on Artificial Intelligence and Statistics, pp. 6388-6421 PMLR. 2023.
Zhendong Wang, Huangjie Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint 0el:22266.02222.. 2022.
Zhaoxuan Wu, Yao Shu, and Bryan Kian Hsiang Low. Davinz: Data valuation using deep neural networks at initialization. International Conference on Machine Learning, pp. 24150-24176. PMLR, 2022.
Xinyi Xu, Zhaoxuan Wu, Chuan Sheng Foo, and Bryan Kian Hsiang Low. Validation free and replication robust volume-bass based data valuation. Advances in Neural Information Processing Systems, 34:18837-108888888 2021.
Sidi Yang. Tianhe Wu, Shuwei Shi. Shanshan Lao, Yuan Gong. Mingdeng Cao, Jiahao Wang, and Yujiu Yang. Maniqa: Multi-dimension attention network for no-reference image quality assessment. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 1191-1200, 2022.
Richard Zhang, Phillip Isola, Alexci A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 586-595, 2018.
13

---

Published as a conference paper at ICLR 2025
CONTENTS
Introduction
2
Motivation and Problem Formulation
Motivations
2.1.1
Theoretical Justification
2.1.2
Emperical Validation
2.2
Problem Formulation and Challenges
3
The Proposed Data Valuation Methods
Efficient Similarity Matching
3.2
Image Quality Assessment
Value Calculation
Experiments
Experiment Setup
Metries
4.3
Identical Class Test (CI)
Identical Attributes Test (C2)
4.5
Out of Distribution Detection (C3)
10
4.6
Efficiency (C4)
10
Conclusion
10
6
Acknowledgement
11
Appendix
14
Notation
16
Algorithm
16
Related Work
16
Data Valuation
16
Generative Model
17
D Statistical Results for Figure]
18
Additional Results on C1 and
19
E.1 (C1) Identical Class Test Other Generative Models
19
denticcll Attributes Test on CelebA
19
(C2) Identical Attributes Test Other Datasets
20
14

---

Published as a conference paper at ICLR 2025
F
Different Generated Data Sizes
21
Alternative Embedding Approaches
21
Alternative Distance Metric
Necessity for calibration
21
J
Additional Experimental Details
22
Justification of Experiment Setup
22
J.2 Datasets
23
J.3
Architecture of Generative models
23
K Discussion on the Possible Applications
24
Current Limitation and Future Directions
24
M Omitted proofs
24
Experiment on Text to Image generation Model (Stable Diffusion)
26
Efficiency with Increased Data Size
26
Impact of quantization recall
26
15

---

Published as a conference paper at ICLR 2025
Roadmap of Appendix. In this appendix, provide concise summary of key notations and Algorithm|] T. detailing the pipeline of GMValuator Sec. [A] and Sec. B A more comprehensive review of related work is also provided in Sec. C Additionally, Sec. D contains detailed statistical analysis for the results depicted in Figure T. Our main text focuses on experiments conducted on benchmark datasets such as MNIST and CIFAR-10, along with large-scale dataset, ImageNet, to evaluate the most significant contributors found by GMVALUATON are in the same class as the generated sample (C1). In Sec. E extend the evaluation of GMVALUATOR
other generative models for C1. Additionally, we expand our evaluation using C2 by considering the ground truth of multiple combined attributes. We operate under the assumption that the most significant contributors to generated sample should exhibit similar attributes in the images. For more thorough validation of the effectiveness of GMVALUATOR, high-resolution datasets like AFHQ and FFHQ are also employed for C2. The appendix includes additional experimental insights such ablation studies, alternative embedding approaches, alternative distance metrics, calibration and further experimental details in Sec. F
and J respectively. The potential applications, limitations and future directions are also provided in Sec. K and Sec. Lastly, we present the proof of our theorem and ensure the reproducibility of our experiments.
Table 5: Some important notations in Sec.
Notations
Description
the distribution of the subset of the training data
μT
subset of the contributors generative model
G
well-trained generative model noise distribution a latent sample in the latent space distance function training dataset, training sample generated dataset, generated sample a subset of the training data the real K contributors K ==
Z Z
-(..-)
X. x
X.
$
S*
K
K contributors found by a data valuator
A X
attribute space data distribution loss function
C
f. f'g-
labeling function, model trained on S*
A NOTATION To make motivation and problem formulation clear, we list some significant notations from Sec. in
B ALGORITHM
To better introduce GMVALUTOOR the pseudocode is shown in Algorithm i
C RELATED WORK
C.I DATA VALUATION
There are three lines of methods data valuation: metric-based methods, influence-based methods and data-driven methods. terms of metric-based methods, the commonly-used approach is to calculate its marginal contribution (MC) based performance metrics (e.g.. accuracy, loss). As the basic method depending on performance metrics for data valuation, LOO (Leave-One- Out) Cook (1977) is used evaluate the value of the training sample by observational change
16

---

Published as a conference paper at ICLR 2025
Algorithm 1 GMVALUATOR
Input:
Training dataset X {ar [xx]] ji= well-trained model G", random distribution Z.
Output: Generated dataset X, the value training data
points T- = ( $2.... pn)
1: X = =1;11 + G for 2j E 2
Generate the synthetic dataset
2: for 9 in do
3:
I/ Matching process (see Sec. 3.1
4:
P, = =((X, 2.)) 11 Including two phases
5:
for z; in P, do
6:
dij- < DreanSSim(zs,0)) or others end for
7:
8:
qj = MANIQA(2;) I/ Image Quality Assessment (see Sec. 3.2)
9:
Calculate score Vi dej, qÿ) using Eq. equation 6/ Contribution Score Calculation (see
Sec. 3.3
end for
11: // Calculation of data value and return the result &
12: for H; X do
13: Calculate x;'s 's value o; using Eq.equation
14: end for
15: return 4 = {01. 02...., On.
of model performance when leaving out that data point from the training dataset. To overcome inaccuracy and strict desirability of LOO, SV Ghorbani & Zou (2019) and BI Wang & Jia (2023) originated from Cooperative Game Theory are widely used to measure the contribution of data Jia eet al. (2019b); Ghorbani et al. (2020). Considering the joining sequence of each training data point, SV needs calculate the marginal performance of all possible subsets in which the time complexity is exponential. Despite the introduction of techniques such as Monte-Carlo and gradient- -bsed methods, well as others proposed in the literature, approximating data significance value (SV) is computationally expensive and it typically requires retraining Ghorbani & Zou (2019); (2019a). The computational cost and need for unconventional performance metries present difficulties in adapting the methods to generative models. As for influence-bused methods, they evaluate the influence of data points on model parameters by computing the inverse Hessian for data valuation Jia et al. (2019a); Richardson al. (2019); Saunshi et al. (2022). Due the high computational cost, some approximation methods have also been proposed Pruthi et al. (2020). In addition, the use of influence function for data valuation is not limited to discriminative models, but also be applied to specific generative models such as GAN and VAE Terashita et al. (202T); Kong & Chaudhuri (2021). training- free methods that focus on the data itself Xu et al. (202T): | Wu et al. (2022); y Just et al. (003)). Additionally, the Information- Theoretic Measures method Boopathy
; Jia et al.
When it comes
dita--drvve methods, most of them
(2023) evaluates the inherent generalization difficulty of tasks and shows potential for adaptation to data valuation. However, it is not applicable to our specific setting due to reliance on performance metries, training requirements, and focus on dataset-level trends.
C.2 GENERATIVE MODEL
Generative models are a type of unsupervised learning that can learn data distributions. Recently, there has been significant interest combining generative models with neural networks create Deep Generative Models, which are particularly useful for complex, nigh-dimensional data distri- butions. They can approximate the likelihood of each observation and generate new synthetic data by incorporating variations. Variational auto-encoders [VAEs))Reeeee Rezende et al.] (2014) optimize the log-likelihood data by maximizing the evidence lower bound (ELBO), while generative adversarial networks (GANs) Goodfellow et al. (2020); Karras et al. (2020) involves generator and discriminator that compete with each other, resulting in strong image generation. Recently proposed diffusion models Ho et al. (2020); Rombach et al. (2022) add Gaussian noise to training data and learn to recover the original data. These models use variational inference and have a fixed procedure with a high-dimensional latent space.
17

---

Published as a conference paper at ICLR 2025
Table 6: The statistic test of data values of X,1 versus x2 using different generative models. Xu1 is supposed to have higher value than Xo2, given the generated data.
Ho: oDD,, S. H1: o(X,, (4) < ,, E Noi E X
BigGAN
Classifier-free Guidance Diffusion
Average value (v1)
0.319654
0.030434
Average value (v2)
1,632352
0.369565
P-value
6. 6.937027 x10 58
8.053195 x x10-
T-statistic
17.924512
15. 75.947860
Significance level
0.01
0.01
p-value less than 0.01, reject Ho, value of v2 less than v1 averagely
p-value less than 0.01, reject Ho. value of v2 less than v1 averagely
Result
D STATISTICAL RESULTS FOR FIGURE I
evident by visualization Figure I that the data points in X,2 (used for training) are more overlapped with generated data than data points in X21 (not used for training). We perform statistic testing on data values obtained by GMVALUTOR to examine data points X,2 (used for training) have significantly higher values than those of the data points in X.. (not used for training).
To this end, we use t-test with the null hypothesis that data values X,1 should not be smaller than those of X22 We compute p-value, which is the probability of getting difference large as we observed, or larger, under the null hypothesis. If the p-value is very low, we reject the null hypothesis and consider our approach, GMVALUATOR, to be verified with high level of confidence (1-p). Typically, a p-value smaller than significance level 0.01 is used as a threshold for rejecting the null hypothesis. T the outcomes of X,, and X in CIFAR-10 10 with < 0.01 for both BigGAN and diffusion model, indicating that the data points in X,2 have significantly more value than those in X.1- Consequently, these findings align with the presumption that the trained dataset Xu2 has a higher value than the untrained dataset Xx1 and verify our approach.
Figure 6: Visualization of Identical Attributes Test on AFHQ and FFHQ The results shown in the first and second subfigures on the left are conducted on AFHQ-Cat and AFHQ-Dog, respectively. The subfigure on the right presents the results FFHQ. each subfigure, the generated samples are on the left. and the top & contributors in the training dataset on the right.
18

---

Published as a conference paper at ICLR 2025
Table 7: Performance comparison of Identical Class Test.
MNIST
GAN (%)
k=30
k=50
k=100
GMValuator (No-Rerank)
96.27
96.26
95.86
GMValuator (l2-distance)
97.73
97.58
96.03
GMValuator (LPIPS)
97.77
97.72
97.38
GMValuator (DreamSim)
97.43
97.44
97.40
Diffusion (%)
k=30
k=50
k=100
GMValuator (No-Rerank)
92.40
91.82
91.26
GMValuator (2-distancee
92.90
92.66
91.88
GMValuator (LPIPS)
93.73
97.72
92.42
GMValuator (DreamSim)
93.90
93.44
92.55
CIFAR-10
BigGAN (%)
k=30
k=
k= 100
GMValuator (No-Rerank)
64.70
63.80
62. 14
GMValuator (l2-distance)
64.70
63.80
62.14
GMValuator (LPIPS)
63.67
62.80
61.51
GMValuator (DreamSim)
70.33
68.74
65.18
Class-free Guidance Diffusion (%)
k=30
k=50
k=100
VMVuutoor (No-Rerank)
72.67
72.00
71.00
GMValuator (ly-distance)
72.67
72.00
71.00
GMValuator (LPIPS)
72.53
72.28
71.06
GMValuator (DreamSim)
79.37
78.08
74.61
E ADDITIONAL RESULTS ON C1 AND C2
E.I (CI) IDENTICAL CLASS TEST ON OTHER GENERATIVE MODELS
We have presented Identical Class Test (C1) on 3- V-V and MNIST LeCun (1998), CIFAR 10 Krizhevsky et al. (2009) in Sec. 4.3 in our main context. Since GM VALUATOR is agdelsagnostic, we further validate our method of Cl on other generative models.
Here, we conduct the experiments using a GAN and a Diffusion Model on MNIST. The architectural details of the used generative models are described in Sec. in the appendix. We also conduct the experiment on BigGAN Brook et al. (2018) and Class-free Guidance Diffusion Ho & Salimans (2022) with CIFAR-10. We used the same number of generated samples m 100 as the experiments presented in Sec. 4
Following the similar settings in Sec. 4.3 (C1), we examine the class(es) of is top & contributors for given generated data in the training data. We calculate the number of training samples, denoted as Q, from the top l contributors that have the same class as the generated data. The identical class ratio, denoted p. is calculated as p = 2/k. We report the average value of p across the generated datasets for different choices of Table 7 GMVALUATOR (DreamSim) has the highest ratio of contributors that belong to the same class the generated sample among most of the models evaluated on MNIST and CIFAR-10 datasets for different values of k. And the ratio improves as the value of decreases, which is consistent with the top assumption and validates our method.
E.2 (C2) IDENTICAL ATTRIBUTES TEST ON CELEBA
We extend C1 to focus the attributes present in the images, treating them ground truth rather than class labels, as discussed in Sec. 4.4 Our experiments now incorporate multiple attributes simultaneously, rather than just single attribute, to verify the performance of GMVALUATOR For instance, when evaluating a generated image with both a hat and eyeglasses, the most significant contributors identified should also include both of these attributes. Table 8 showcases some outcomes of identical attributes test when regarding multiple combined attributes as ground truth, which validate the effectiveness of our methods.
19

---

Published as a conference paper at ICLR 2025
Table 8: Performance of Identical Attributes Test (C2) of multiple combined attributes.
Top K contributors:
k=5
k=10
k=15
Attribute: Eyeglasses & Gender (%)
GM (No-Rerank)
50.10
48.48
48.42
GMValuator (l2-distance)
59.19
56.67
56.23
GMValuator (LPIPS)
78.18
74.24
73.87
GMValuator (DreamSim)
92.53
90.61
90.30
Attribute: Eyeglasses & Hat (%)
Valuutro (No-Rerank)
78.79
78.38
85.86
Valuutoo (l-ddistance)
84.44
82.93
83.23
GMValuator (LPIPS) 86.87
86.16
86.33
GMVauttor (DreamSim)
89.49
87.58
87.41
Attribute: Gender & Hat (%)
GMValuator (No-Rerank)
58.59
57.47
57.71
GM GMValutoor (l--distance)
61.62
60.51
60.40
GMValuator (LPIPS)
63.84
63.23
62.96
GM Valuator (DreamSim)
65.25
64.44
64.18
E.3 (C2) IDENTICAL ATTRIBUTES TEST ON OTHER DATASETS
30
28
04
615
012
C50
028
018
034
ICRAB SOIE)
JO (4200
Figure 7: The change of with the different number of generated samples on MNIST and CIFAR- 10 by diverse generative models.
20

---

Published as a conference paper at ICLR 2025
To further validate GMVALUATOR, also conducted experiments high-resolution datasets: AFHQ|Chor et al. (020) and FFHQ et al. (2019), following the same settings in Sec[4.4] The results shown Figure 6 which demonstrates the effectiveness of our methods in C2. The results show that the top & contributors have similar attributes with the generated sample such as fur color of cats dogs in AFHQ. For the experiment conducted FFHQ, human faces attributes of the most significant contributors are also similar to the attributes in generated images.
F DIFFERENT GENERATED DATA SIZES
Since our value function Φ; (Eq, equation 5 for training data 2; is computed by averaging over generated samples, it is expected that the sensitivity of o; is connected to the size of the in- vestigated generated sample. To explore the influence of generated data size m on the utilization of GMMALLATOR
perform sensitivity testing MNIST, CIFARIO using generative models GAN Goodfellow et al. (2020), and Diffusion models Dhariwal & Nichol (202 as depicted below, The dataset, model and used varying number of samples from the same class. Specifically, we consider four different sample sizes, denoted by m, which are given by 10, 30, and 50. Next, we evaluate the GMVALUATON using parameter C1, and this evaluation is performed for each of the aforementioned values of m. Subsequently, we conduct the experiment times using EEMVALUATOO (No-Rerank), each time with different m-sized generated data samples from the same class. The results are presented as the mean and standard deviation (p) for accuracy, taken over these 10 runs. The results shown in Figure imply that varying does not yield notable differences in mean accuracy and increasing the number of generated samples m leads to more stable and consistent results.
denoted under each subfigure of Figure 7 First, we generate
G ALTERNATIVE EMBEDDING APPROACHES
In the step of efficient similarity matching, the embedding fe is exclusively used in the recall phase for retaining n samples with non-minimal contributions. Apart from using CLIP for embedding, we also investigate the influence for the results when using other embedding methods. in Table using more embedding methods, showing similar performance with CLIP with "Rerank". The reason for this results is that we implement re-ranking that relies on the image space rather than the embedding space, which allows to derive accurate top l contributors from samples >> k. Thus, the embedding could tolerate some noise in the recall phase.
present results
Table Comparison of different embedding methods.
MNIST (%)
No-Rerank
12-distance
LPIPS
DreamSim
CLIP
86.41
87.76
88.78
88.78
Alexnet
79.77
84.82
86.51
88.72
Densenet 80.47
86.77
87.97
89.35
H ALTERNATIVE DISTANCE METRIC
We suggest utilizing Learned Perceptual Image Patch Similarity (LPIPS) Zhang al. (2018)
DreamSim Fu et al. 2023) as the distance metric d during the re-ranking phase. This enables to understand the perceptual dissimilarity between generated and real data points. These metrics applied the image embedding space derived from pre-trained models. Alternatively. distance measurement can be based on pixel space, such as employing (2-distance. Notably, we find that distances calculated in the input pixel space yield comparable outcomes as shown in Table This could be flexible the different choices of distance metries
implies that our method GMVALUATU and the selection can depend on data prior.
I NECESSITY FOR CALIBRATION
For challenges and 3, we utilize image quality assessment and establish a non zero scores rule for calibration in GMVALUATOR. To better understand the impact and necessity of calibration,
21

---

Published as a conference paper at ICLR 2025
compare the distribution of the top 1, 2, and contributors' scores with (w) and without (w/o) calibration conducted on MNIST in Figure g It is evident that scores without calibration are generally higher than with calibration. We also extend C3 and perform ablation study in Figure 9 on scenarios where the generated samples include low-quality outputs. The results show that the OOD) (out-of-distribution)) training samples' value rank by GM without (w/o) calibration is smaller, indicating significant bias and poor performance.
Contribution Score (MNIST)
1.0
imae quality
was image quality moscastmnnt
022
2000
Comptbuttonn
0.8
0.4
6000
1hp1
Thp?
Topp3
quality calbrarion
quality calbrracion
Figure 8: Sore Distribution
Figure Value Rank
J ADDITIONAL EXPERIMENTAL DETAILS
J.1 JUSTIFICATION OF EXPERIMENT SETUP
We provide detailed justification in Table I0 for our experiment setup from C1 to C4.
C1. In Identical Class Test, the baseline method is VAE- TracIn Kong & Chaudhuri 2021 which can find the most influenced instances the training dataset. Since VAF-Trachn the model-specific method, we only need to compare it with GMVALUATOR when VAE model is used. Besides, all the datasets used in C1 should have the class labels. Considering the computational demands detailed in VAE-TracIn Kong & Chaudhuri (2021), the runtime complexity of VAE- TracIn correlates with the number of network parameters and the size of the dataset. Therefore, MNIST and CIFAR-10 for comparative evaluations with
analysis primarily utilizes simpler benchmark datasets such as
C2. In extending C1. we use image attributes to supplant the concept of class for datasets lacking class labels. For datasets with attribute labels, quantified experiments are feasible, as demonstrated Table and Table For datasets without attribute labels, we employ visualized experiments.
C3. IF4GAN, a model-specific method for GAN, serves as the baseline for measuring data value. Adhering the settings outlined in Terashita al. (20T)) and considering the impractical computational costs, we conduct experiments on the same dataset (MNIST) in Terashita et al. (202T) using DCGAN.
Table 10: Justification of Experiment Setup. The selection of datasets and models was based on three critical factors that guided the process. Firstly, attribute labels were required to evaluate C2 effectively. Secondly, benchmark datasets were meticulously chosen to ensure a fair comparison with baselines while also taking into account computational costs (C3 and C4). Finally, the selected generative models are powerful enough to generate good-quality data for the datasets.
Datasel
Model
Baseline
Label Requirements
C1
MNIST. CIFAR-10
Diffusion, GAN
VAE
VAE Tracln
Chanthur 2021
Class labels
Class labels
ImageNet
Masked Diffusion Transformer Gao al. 2023
Class labels
CelobA
Diffusion-StylGGAY
Attribute labels
AFHO, FFHQ
StyleGAN
MNIST
IXCANN
IrGGAN
Tabels
MNIST, I. CIFAR-TO
VAE
TAE-Tracla Xong & Chaniii 2021
Class labels
C4
MNIST
DOGAN
IF-GAN Terabila
labels
22

---

Published as a conference paper at ICLR 2025
C4. We present comparison of the efficiency of GMVALUATOR against baseline methods. In accordance with the settings used for VAE- TracIN and IF4GAN in in/Knn & Chaudhuni 2 we report the efficiency results for C1 on the MNIST and CIFAR- 10 datasets, and for C3 on MNIST.
J.2 DATASETS
We conduct the generation tasks in the experiments on benchmark datasets (i.e. MNIST LeCun et al. (T998) and CIFAR Krizhevsky al. (2009)), face recognition dataset (ie. CelebA Liu al. (2018). high-resolution image dataset AFHQ Choi et al. (2020) and FFHQ Karras et al. (2019), large-scale
image dataset ImageNet Deng et al. (2009).
MNIST. The MNIST dataset consists of a collection of grayscale images of handwritten digits (0-9) with a resolution of 28x28 pixels. The dataset contains 60,000 training images and 10,000 testing images.
CIFAR-I0. CIFAR-10 10 dataset consists of 60,000 color images in 10 different classes, with 6,000 images per class. The classes include objects such airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each image in the CIFAR-10 10 dataset has resolution of 32x32 pixels.
CelebA. The CelebA dataset is widely used face recognition and attribute analysis dataset, which contains a large collection of celebrity images with various facial attributes and annotations. The dataset consists of more than 200,000 celebrity images, with each image labeled with 40 binary attribute annotations such as gender, age, facial hair, and presence of eyeglasses.
AFHQ. The AFHQ dataset is a high-resolution image dataset that focuses on animal faces (e.g., dogs, cat), and it consists of high-resolution images with 512 x 512 pixels.
FFHQ. The FFHQ dataset is high-resolution face dataset that contains high-quality images (1024x 1024 pixels) of human faces.
ImageNet. ImageNet is a large-scale image dataset, which contains over 14 million images and is categorized into more than 20,000 classes.
J.3 ARCHITECTURE OF GENERATIVE MODELS
In our experiments, we leverage different generative models in the class of GAN, VAE and diffusion models. We utilize 3-VAE for both MNIST and CIFAR-I0 datasets while simple GAN is conducted on MNIST. BigGAN and B-V are also conducted on CIFAR-10. 10. We list the architecture details for these generative models from Table 1T Table [3 StyleGAN is used for high-resolution datasets AFHQ and FFHQ. CelebA uses Diffusion-S SttuuinnnnSylGGANNN Wang et al. (2022), for which we use the exact architecture in their open-sourced code. In addition, Masked Diffusion Transformer, as introduced by Gao et al. Gao et al. (2023). is applied to the ImageNet.
Table 11: The architecture of GAN for MNIST.
Generator
Discriminator
FC(100, 8192), BN(32), ReLU Conv2D(128, 64, 4, 2, 1), BN(64), ReLU
Conv2D(1, 128, 4, 2, 1), BN(128), LeakyReLU FC(8192, 1024), BN(1024), LeakyReLU
Table 12: The architecture of BigGAN.
Input
x8x28x1 (MNIST) & 32x32x3 (CIFAR-10).
Conv 32x4x4 (stride 2). 32x4x4 (stride 2),
Encoder
64 44x4 (stride 2), 64x4x4 (stride 2),
FC 256. ReLU activation.
Latents
32
Decony reverse of encoder. ReLu acitvation.
Decoder
Gaussian.
23

---

Published as a conference paper at ICLR 2025
Table 13: The architecture of 3-VAE.
8-VAE
Generator
Discriminator
E R2 (0,
RGB image æ R23233233
Embed(()) 1292
Linear (20 + 128) x 16ch
ResBlock down ch + 2ch
ResBlock up 16ch + 16ch
Non-Local Block (64 x 64)
ResBlock up 16ch + 8ch
ResBlock down 2ch - 4ch
ResBlock up 8ch + Ach
ResBlock down 4ch - 8ch
ResBlock up 4ch + 2ch
ResBlock down 8ch + 16ch
Non-Loocal Block (16 x 16)
ResBlock down 16ch + 16ch
ResBlock up 2ch + ch
ResBlock 16ch + 16ch
BN, ReLU. 3 Conv ch + 3
ReLU, Global sum pooling
Tanh
Emhed (y) h+ ( Ilincar
K DISCUSSION ON THE POSSIBLE APPLICATIONS
The application of data valuation within generative models offers a wide range of opportunities. A potential use case is quantify privacy risks associated with generative model training using specific datasets, since the matching mechanism GMVALUATOR can help re-identify the training samples given the generated data. By doing so, organizations and individuals will be able to audit the usage of their data more effectively and make informed decisions regarding its use.
Another promising application is material pricing and finding in content creation. For example, when training generative models for various purposes, such as content recommendation or personalized advertising, data evaluation can be used to measure the value of reference content.
In addition, GMVALUATOR can play an important role in the development of ensuring the responsi- bility of using synthetic data in safe-sensitive fields, such as healthcare or finance. By assessing the value of the data used in generative model training, researchers can ensure that the generated data are robust and reliable.
Last but not least, the applications of GM can promote the recognition of intellectual property rights. Determining the value of the intellectual property being generated by generative models is critical. By evaluating the data employed in training generative models, we can develop more comprchensive understanding of copyright that may emerge from the generative models. In essence, such insights help advance licensing agreements for the utilization of the generative model and its outputs.
L CURRENT LIMITATION AND FUTURE DIRECTIONS
The limitation of this work is that it only measures data value for vision-related generative models and conducts experiments exclusively within the field of computer vision. However, this does not mean that GM VALUATO cannot be easily adapted to Natural Language Processing (NIP) fields, given its core idea of similarity matching. In the future, should extend GM to NLPP and assess the data value for language-related generative models, such as large language models (LLMs).
M OMITTED PROOFS
We follow Just et al. (202)) to prove the theorem. Firstly, we give several assumptions that will be used in later proof.
Assumption M.1 Following Assumption 2.3 given distance function d(..-) between, defined the coupling between X(T(s) and X(s-1f) as π
π ==
arg inf xeT())))
E Don
24

---

Published as a conference paper at ICLR 2025
It is easy to that all joint distributions defined above couplings between the corresponding distribution pairs. Then, following Just et al. (2023) we prove the main Theorem.
Theorem M.2 (Restated of Theorem 2.4 Let fs- optimal contributor dataset S Following Assumption 2.3
= {0,1}' if the contributors corresponding to
1 be the model trained on the
the given generated data X, we have:
{ (1()).
R Ramus-
(8)
S ke
Proof M.3
[c (1())
C (f()).
(x)
Ezwar
= Exerr
Egupps [ [f(()))
E E
(1(6))).s. (+)
Eawps- (<(())) (x)(
Eewer [c (+))) (9)
[ (f(x) - EENNr [c f())
We bound Beous
follows:
E Bawps- [c (())) Exepr (f(x)
1re c (1sss)) (
(zx) dr (xr, xs-)
1fx c (xs-)))
(((Is:)))s-(TT)) - (f(Ts) (f (.s c (f(TT) (s-( di (Er,
/ (I(0s:))
[ & (I(Es)) du (Fr, (FT)
x2
+ f c (fs [ c (f(or)) do (x))
(10)
Then due to k-Lipschitzness of L and e-Lipschitzness of f. we can obtain:
RHS of Eq.equation
10
I1f's (xs.) - fs- (IT))ldT" (xT, x2
+ k
[1f(xs)) - f(xr) (ldm)) (XT,Ns))
x2
/ 20(IT, xs. )da (IT,
ke xx2
kedw (X(ri)), X(s-1));
where the last step is due the definition of 1- I-Wasserstein distance. Then, according the triangle inequality of Wasserstein distance Peyré et al. (2019), we can obtain:
dw KKrrs,,, S dw (N X(xnn) + dw (K(8))) X(s-11)
(11)
Combining Eq. equation Dand Eq. equation 77 we finished the proof and obtained the Theorem 2.4 By reducing the distance term dw X(rinxx(&1) n) we have X Xis- As result, the expected distance
"1 +εT((T.S*)
min )e 0,
with randomly sampling S and T with K elements.
25

---

Published as a conference paper at ICLR 2025
N EXPERIMENT ON TEXT TO IMAGE GENERATION MODEL (STABLE DIFFUSION)
In this section, we conducted experiments using stable diffusion model Rombach al. (2022) fine-tuned with LoRA on Naruto dataset Cervenka 2022).
Identical Attributes Test (C2). To evaluate the effectiveness of GMVALUATOR on text-to-image dataset, we use the attributes in the images as ground truth and employ the Identical Attributes Test (C2) assess its performance. The results presented in Table [4 show that the top & contributors share similar attributes with the generated sample, such as gender, hair, and hat, demonstrating the effectiveness of our approach in evaluating data contributions.
Table 14: Performance of Identical Attributes Test (C2) of some attributes including Gender, Hat, and Hair on Naruto.
Attribute (%)
Gender
Hat
Hair
Top contributors:
==
& 5
==
k 5
==
& 5
GMValuator (12 distance)
94.33
93.50
92.80
86.30
85.50
84,40
88.00
87.75
88.40
GMValuator (LPIPS)
98.67
98.50
98.80
87.00
86.25
85,40
96.67
96.25
96.00
GMValuator (DreamSim)
99.00
98.50
98.80
86.33
86.75
85.80
98.30
97.75
96.00
Efficiency. To evaluate computational efficiency, we report the time taken to identify significant contributors for a single generated sample in Table [5] The results emphasize the effectiveness and efficiency of our approach for text-image models such as Stable Diffusion. The experiments were conducted in environment equipped with an RTX A6000 (48GB) GPU and 12-vCPU Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz.
Table 15: Efficiency using Stable Diffusion Naruto.
Method
GMValuator (12, distance) GMValuator (LPIPS) GMValuator (DreamSim)
Time (s)
1,27888
3.09580
6,28144
O EFFICIENCY WITH INCREASED DATA SIZE
As training-free and efficient data valuation approach for generative models, GMVALUATOR is also highly scalable to large-scale and complex datasets. To evaluate this, we assess the efficiency of GMVALUATOR on datasets of varying sizes, illustrating the trend in computational efficiency as the data size increases. In Fig TO we present the running time versus the dataset size for GMVAUTTO variants (dashed lines, with colors representing different similarity metrics) using the same settings as in Sec This result demonstrates that average running time for various variants of GMValuator on a single generated image, with the recalled number set to 200, is dominated by the re-ranking process. As a result, the running time is nearly independent of the dataset size. Noting that the number of recalled images during the re-rank phase fixed, thus time cost reveal minimal variation in runtime with increasing dataset size, demonstrating that our method maintains consistent computational efficiency even as the data scale grows.
In Fig 11 we present the time cost for one generated image during the recall phase (No-Rerank), with dataset sizes ranging from 100 K to 50000 K. The results show that the running time remains small across different dataset size. Benefiting from the efficiency of our recall phase, the time overhead increases only marginally. In contrast, as noted in Kong & Chaudhuri (202T). "the run-time complexity of VAE- TracIn is linear with respect to the number of samples, checkpoints, and network parameters". Similarly, IF4GAN Terashita et al. (202I) is time-intensive, with complexity increasing alongside model size and the number of training epochs, as outlined in Algorithm 2 of their work.
P IMPACT OF QUANTIZATION RECALL
In this section, we evaluate the effect of quantization recall by empirically compare quantization recall with nearest neighbor (NN) search using varying recalled number K during the recall phase. To be specific, We conducted our experiments on the MNIST dataset using GAN model with recalled number K increasing from 100 to 700. For consistency, we leverage L2 -istttaee in the reranking phase.
26

---

Published as a conference paper at ICLR 2025
0.030
No Rerank 12Distance
0.045
(0)
LPIPS
8-
0.035
Time
2-
0.025
Time
0.020
100
500
1000
1500
2000
10000
40000
Figure 10: Running time vs. dataset size for variants of GMValuator.
Dataset Size
Figure 11: Running time vs. dataset size for recall phase (No-Rerank) of GMValtttr..
Dataset Size (K)
33
00
00-
Number Recal Samples
Figure 12: Running Time vs. Number of Recall Samples
Efficiency. GMVALUATOR significantly outperforms nearest neighbor search in speed, taking an average of 0.25 seconds attribute the most significant contributors for generated sample, which is more than 50 times faster compared to 13.75 seconds for NN search. Whats more, the time cost for NN search will increase linearly with dataset size, while using PQ recall shows only slight increased time in larger dataset size as shown in Fig I1 The experiments in this section are run in an environment equipped with an RTX A6000 GPU (48GB) and a 12-vCPU Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz.
Efffective. As illustrated in Fig. under different recalled numbers (from 100 700), the results of GMValuator are similar the exhaustive KNN search, demonstrating that method maintains high performance while ensuring computational efficiency.
27


Reviews:
[{"review_id": "Cm0fGQBAnF", "reviewer": "Reviewer_eyah", "summary": "The paper introduces GMVALUATOR to solve the data valuation problem for vision generative models, which is important yet has been overlooked by existing studies. GMVALUATOR formulates data valuation as a similarity-matching problem and incorporates image quality assessment to calibrate the contributions of data samples. Despite high computational complexity with large datasets, GMVALUATOR demonstrates effectiveness across multiple datasets and generative models, positioning it as a promising tool for data valuation in the field. Although the technical contribution is obvious, there is still much room for improvement in the author's presentation and content arrangement.", "questions": "1.\tAs stated by the authors: However, this does not mean that GMVALUATOR cannot be easily adapted to Natural Language Processing (NLP) fields, given its core idea of similarity matching. So,  how to apply the proposed method to generative models for text-based data or data from multiple modalities?\n2.\tIf data valuation is not used in training generative models, what will be the limitations? Can the proposed method or evaluation metrics measure the limitations?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1.\tThe authors proposed GMVALUATOR to tackle the data valuation issue for generative models. GMVALUATOR is innovative, and model-agnostic, enabling broad applicability and adaptability across various generative models. Besides,  GMVALUATOR does not require retraining of models, offering the advantage in R&D scenarios with limited computational resources.\n2.\tThe authors provide detailed theoretical justification for formulating data valuation for generative models as a similarity-matching problem. They also provide empirical validation for this motivation. This makes the GMVALUATOR being very reasonable.\n3.\tThe authors demonstrated through extensive experimental results that the data valuation method proposed can achieve good results in multiple aspects and over multiple generative models. Besides, they also provided validation of the experimental settings and the open-source implementation, which further increased the credibility of the results.", "weaknesses": "1.\tThe manuscript is not well written. For example, in the Introduction, before talking about the existing work, it is suggested to generally define/introduce the data valuation problem (including the input and the objective). Moreover, the authors didn't highlight the urgent need for data valuation in existing generative models; this poses a challenge to the motivation of this paper. Most importantly, instead of briefly introducing the principle of the proposed GMValuator (such as why and how to formulate data valuation for generative models as an efficient similarity-matching problem), the author only introduces the goal to be achieved. These places make the content difficult to read.\n2.\tAlthough the authors claim that they proposed a versatile data valuation method for generative models, in both the problem formulation, method introduction, and subsequent experiments, they only evaluate it on image samples. Therefore, the statement \"for generative models\" may be inappropriate. Even though the authors mentioned in the Current Limitation \"However, this does not mean that GMVALUATOR cannot be easily adapted to Natural Language Processing (NLP) fields, given its core idea of similarity matching.\" However, the reason for this statement may need further explanation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "beGufiTy8y", "reviewer": "Reviewer_smoK", "summary": "The paper focuses on the data valuation of generative models. Existing data valuation methods designed for discriminative models cannot adapt to generative models due to 1) lack of robust performance metrics; 2) the large size of generative models; and 3) lack of data labels. In order to mitigate this gap, the authors propose GMValuator. GMValuator is based on similarity matching between training data and generated data. If a training sample is similar to a generated sample, it is considered to have contributed to the generated sample. The value of a training sample is computed by the quality of its contributed generations. Four evaluation criteria are introduced to assess data valuation. Experiments demonstrate the effectiveness of the proposed GMValuator.", "questions": "Please refer ti weaknesses.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- This is the first paper on data valuation on generative models. Previous data valuation methods focus on discriminative models and cannot adapt to generative models.\n- Compared to the retraining-based and influence-based methods, GMValuator is efficient. It does not require any retraining or computation of hessian.\n- GMValuator is effective on the proposed metrics. GMValuator has significantly improved compared to baseline methods.", "weaknesses": "- For SOTA text-to-image models like stable diffusion, the image domain is much wider than the test models. As a result, a large number of generated images may be required for accurate data valuation. Meanwhile, generation with these models is slow. More results and ablation on stable diffusion on the SOTA text-to-image models would be helpful.\n- While the proposed metrics are intuitively reasonable, it is coarse-grained and may not be able to reflect the effectiveness of data evaluation methods.\n\nThe authors addressed my concerns.\nI raise my score to 6.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "sImsfR5CEq", "reviewer": "Reviewer_35uE", "summary": "This paper proposes a novel data valuation method for generative models. The authors introduce a model-agnostic, training-free data valuation framework, addressing a significant challenge in the field: existing methods typically require retraining or Hessian calculations, which are computationally intractable. In this approach, the contribution of a training data point to a generated data point is defined as inversely proportional to the distance between the two data points. The author performed extensive experiments against two baselines and showed effectivenss of the propsoed method.", "questions": "- What is the effect of the quantization step in the recall phase? \n- Empirically, how different is the proposed score from a nearest neighbor search with the rerank metric in the embedding space?\n- How well does the proposed method generalize to other generative models, such as diffusion models?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel and intuitive idea for data valuation in generative models, and the results are promising.\n- The experiments are well-designed, exploring multiple distance functions and encoders to validate the approach. Also, multiple test scenarios were covered, all showing good supporting results for the proposed method.\n- The paper is well-written and easy to follow, effectively conveying the methodology and findings.\n- The paper covers relative literature well.", "weaknesses": "- The impact of the quantization step on the final results is not explored. Understanding this effect would provide a clearer picture of the method’s performance.\n- While section 2 introduces some underlying assumptions and a theoretical motivation for using a similarity-guided data valuation score (illustrated in Figure 1), the framework would benefit from a more rigorous theoretical foundation. Further studies on theoretical support could strengthen the framework’s conceptual grounding and its reliability across different applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "l3JSlXRYbL", "reviewer": "Reviewer_LEU4", "summary": "In this paper, the authors introduce Generative Model Valuator (GMVALUATOR), the first training-free and model-agnostic approach to providing data valuation for generation tasks. The authors formulate data valuation for generative models as an efficient similarity-matching problem. The paper further eliminates the biased contribution measurement by introducing image quality assessment for calibration. Also, the paper introduced four evaluation criteria for assessing data valuation methods in generative models.", "questions": "GMVALUATOR is claimed as the first modal-agnostic and retraining-free data valuation method for generative models. How about the comparison with other approaches such as Information-Theoretic Measures (Akhilan Boopathy et al., ICML 2023, \"Model-agnostic Measure of Generalization Difficulty\"), etc.?\nPlease illustrate whether other approaches are applicable to generative models in a model-agnostic and retraining-free manner, which is important to your claim.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The strengths of this paper are listed as the following:\n\n1. GMVALUATOR is claimed to be the first modal-agnostic and retraining-free data valuation method for generative models.\n\n2. The authors formulate data valuation for generative models as an efficient similarity-matching\n problem. The paper further eliminates the biased contribution measurement by introducing image\n quality assessment for calibration.\n\n 3. The authors propose four evaluation methods to assess the truthfulness of data valuation and evaluate GMVALUATOR on different datasets, such as benchmark datasets and high-resolution\nlarge-scale datasets, and various deep generative models to verify GMVALUATOR’s validity.", "weaknesses": "The weaknesses of the paper are listed as follows:\n\n1. The paper only mentioned 4 criteria for assessing data valuation methods. They are: C1: Identical Class Test; C2: identical attributes test; C3: Out of Distribution Detection; and C4: Efficiency.  \n\nHow about other criteria? \nWhy only use these 4? \n\nFor example, how about Cost-Benefit Analysis (i.e., the trade-offs between the costs of acquiring or processing data versus the performance gains from using it in model training)?\nPlease give some examples and formulas to measure the costs of data acquisition/processing. Then compare them with the performance gains of those data in the context of generative model training. This will be more practical for the proposed new approach. \n\n2. The paper shall discuss the important aspects of data evaluation, such as accuracy and complexity. For example, how accurate is it for the proposed framework? How much is the complexity while implementing the proposed approach? Also, the authors shall discuss the proposed framework on accurately capturing the contributions of individual data points in various different scenarios. \n\n3. The paper shall illustrate other aspects such as scalability, whether the proposed approach is useful for handling big data in real-time applications.  For example, the authors can provide the results of testing the proposed method on progressively larger datasets or measuring processing times for different data sizes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)