GMValuator: Similarity-based Data Valuation for Generative Models

## Abstract

Abstract Data valuation plays a crucial role in machine learning. Existing data valuation methods have primarily focused on discriminative models, neglecting generative models that have recently gained considerable attention.
A very few existing attempts of data valuation method designed for deep generative models either concentrates on specific models or lacks robustness in their outcomes. Moreover, efficiency still reveals vulnerable shortcomings. To bridge the gaps, we formulate the data valuation problem in generative models from a similarity-matching perspective. Specifically, we introduce Generative Model Valuator ( GMValuator ), the first training-free and model-agnostic approach to provide data valuation for generation tasks. It empowers efficient data valuation through our innovatively similarity matching module, calibrates biased contribution by incorporating image quality assessment, and attributes credits to all training samples based on their contributions to the generated samples. Additionally, we introduce four evaluation criteria for assessing data valuation methods in generative models, aligning with principles of plausibility and truthfulness. GMValuator is extensively evaluated on various datasets and generative architectures to demonstrate its effectiveness.

## I Introduction

As the driving force behind modern artificial intelligence, particularly deep learning [], a substantial volume of data is indispensable for effective machine learning. On one hand, informative data samples that are relevant to the task at hand play a critical role in the training process. On the other hand, due to data privacy concerns, personal data is safeguarded by various regulations, including the General Data Protection Regulation (GDPR) [], and has become a valuable asset. Consequently, data valuation has garnered significant attention from both academic and industrial sectors in recent times.

The intricate relationship between data and model parameters presents a significant challenge in contribution measurement of each training sample, thus making data valuation a difficult task. Most of the existing data valuation studies focus on supervised learning for discriminative models (e.g., classification and regression). These methods can be categorized as: (1) Metric-based methods: methods such as Shapley Value (SV) and Banzhaf Index(BI) [, ] provide the assessment of data value by calculating marginal contribution on performance metrics (e.g., accuracy or loss) through retraining the model(^1^11Exception for KNN-Shap []).
(2) Influence-based methods: this line of methods measures data value by evaluating influence on model parameters of data points [, ]. (3) Data-driven methods: These techniques avoid retraining by leveraging data characteristics (like data diversity, generalization bound estimation, class-wise distance), though generally necessitate data labels. [, , ].

Data valuation in the context of generative models has NOT been well-investigated in the current literature. Moreover, there exist significant challenges in directly adapting the aforementioned data valuation methods for discriminative models listed to generative models:
Firstly, the challenge of applying metric-based methods arises from lacking robust performance metrics in generative models, in contrast to the existance of commonly used metrics (e.g., accuracy or loss) in discriminative models. In addition, the expensive cost of retrain requirements is another obstacle to use metric-based methods.
Secondly, influence-based methods may not perform well on non-convex objective function of generative models []. Besides, estimating influence function in deep generative model is expensive, as it requires computing (or approximation) inverse Hessian.
Thirdly, for data-driven methods, they mainly focus on supervised learning, requires knowing data labels to quantify data values.

To the best of our knowledge, limited studies have explored model-dependent data evaluation using influence functions for specific generative models.
IF4GAN [] consider multiple evaluation metrics, such as log-likelihood, inception score (IS), and Frechet inception distance (FID) [] to identify the most responsible training samples for the overall performance of generative adversarial networks (GAN)-based models. However, the selection of appropriate metrics is critical, as the results are inconsistent across metrics.
VAE-TracIn [] finds the most significant contributors for generating a particular generated sample for Variational Autoencoders (VAE) using influence function. However, viable Hessian estimation in influence function calculations incur high computational costs and this method cannot be easily generalized to other generative models.

Considering the challenges posed by the selection of performance metrics and computational efficiency, as well as the need for broader applicability across various generative models, we aim to propose a unified and efficient data valuation method. We expect the unified method to satisfy the following key properties: 1) Model-agnostic: the method should be versatile, capable of being employed across diverse generative model architectures and algorithms, regardless of specific design choices; 2) Computation Efficiency: the method does not require retraining the model and minimize computational overhead while maintaining a satisfactory level of accuracy and reliability in evaluating the value of data points; 3) Plausibility: the method should evaluate the value of data based on its alignment with human prior knowledge on the task, enhancing its credibility and reliability; 4) Truthfulness: the method should strive to provide an unbiased and accurate assessment of the value associated with individual data points.
To meet the design purposes, we propose a similarity-based data valuation approach for the generative model, called GMValuator, which is *model-agnostic* and *efficient*.

In summary, our work here provides the following specific novel contributions:

- •
To the best of our knowledge, GMValuator is the first modal-agnostic and retraining-free data valuation method for generative models.
- •
We formulate data valuation for generative models as an efficient similarity-matching problem. We further eliminate the biased contribution measurement by introducing image quality assessment for calibration.
- •
We propose four evaluation methods to assess the truthfulness of data valuation and evaluate GMValuator on different datasets and deep generative models to verify its validity.

## II Related Work

### II-A Data Valuation

There are three lines of methods on data valuation: metric-based methods, influence-based methods and data-driven methods. In terms of metric-based methods, the commonly-used approach is to calculate its marginal contribution (MC) based on performance metrics (e.g., accuracy, loss).
As the basic method depending on performance metrics for data valuation, LOO (Leave-One-Out) [] is used to evaluate the value of training sample by observational change of model performance when leaving out that data point from the training dataset. To overcome inaccuracy and strict desirability of LOO, SV [] and BI [] originated from Cooperative Game Theory are widely used to measure the contribution of data [, ]. Considering the joining sequence of each training data point, SV needs to calculate the marginal performance of all possible subsets in which the time complexity is exponential. Despite the introduction of techniques such as Monte-Carlo and gradient-based methods, as well as others proposed in the literature, to approximate data significance value (SV) is computationally expensive and it typically require retraining [, ]. The computational cost and needs for unconventional performance metrics presents difficulties in adapting the methods to generative models.
As for influence-based methods, they evaluate the influence of data points on model parameters by computing the inverse Hessian for data valuation [, , ]. Due to the high computational cost, some approximation methods have also been proposed []. In addition, the use of influence function for data valuation is not limited to discriminative models, but can also be applied to specific generative models such as GAN and VAE [, ]. When it comes to data-driven methods, most of them are training-free methods that focus on the data itself [, , ].

### II-B Generative Model

Generative models are a type of unsupervised learning that can learn data distributions. Recently, there has been significant interest in combining generative models with neural networks to create Deep Generative Models, which are particularly useful for complex, high-dimensional data distributions. They can approximate the likelihood of each observation and generate new synthetic data by incorporating variations. Variational auto-encoders (VAEs) [] optimize the log-likelihood of data by maximizing the evidence lower bound (ELBO), while generative adversarial networks (GANs) [, ] involves a generator and discriminator that compete with each other, resulting in strong image generation. Recently proposed diffusion models [, ] add Gaussian noise to training data and learn to recover the original data. These models use variational inference and have a fixed procedure with a high-dimensional latent space.

## III Motivation and Problem Formulation

In this section, we present the background and motivation behind our proposed method GMValuator, which formulates the data valuation for generative models as a similarity matching problem.

Figure: Figure 1: Data distribution for $X_{v1}$,$X_{v2}$ and $\hat{X}$ for CIFAR-10, and $X_{v1}$,$X_{v2}$ are both airplane dataset.
Refer to caption: x1.png

### III-A Motivation

Let $X=\left\{x_{1},...,x_{n}\right\}$ denote the training dataset for generative model training, in which $x\sim\mathcal{X}$ and $n$ is regarded as the size of data. Let $\mathcal{A}$ present the learning algorithm of generative models (e.g., Generative Adversarial Network (GAN), Variational Autoencoder (VAE), Diffusion Model). The primary goal of generative models is to generate $m$ synthetic data samples $\hat{X}=\left\{\hat{x}_{1},...,\hat{x}_{m}\right\}$ that are closely similar to the real data. Thus, the core of the generative model is an intractable probability distribution learning process to train a generator $\mathcal{A}$, that maps each sample $z$ sampled from the noise distribution $\mathcal{Z}$ to a sample $\hat{x}$ in estimated distribution $\mathcal{\hat{X}}$ while minimizing the discrepancy ($D$) between the estimated distribution $\mathcal{\hat{X}}$ and real data distribution $\mathcal{{X}}$:

$$ $\mathcal{A}=\operatorname*{arg\,min}_{\mathcal{A}}D(\mathcal{A}_{z\sim\mathcal {Z}}(z)),\mathcal{{X}}),\quad\mathcal{A}:z\mapsto\hat{x}\sim\mathcal{\hat{X}}$ $$

It’s important to understand the significance of each training data point in relation to the generated data. The objective function can be calculated using likelihood $p_{\mathcal{A}}(x|z)$ in (1), where $p_{\mathcal{A}}(x)$ qualifies the similarity between $\mathcal{A}(\mathbf{z})$ and $x$. The value of $p_{\mathcal{A}}$ should be high as $\hat{x}$ looks like the sampling data $x$, by maximizing the likelihood $p_{\mathcal{A}}(x|z)$.

$$ $p_{\mathcal{X}}(x)\approx\int p_{\mathcal{A}}(x|z)p_{\mathcal{Z}}(z)dz$ (1) $$

Thus, generated data should exhibit a higher degree of similarity to the data points used to train the generator than the ones not used for training, despite originating from a similar distribution.

Therefore, we put forth the plausible assumption that the similarity between training data and generated data characterizes the contribution of the training data to the generated data for any given model. We support the assumption by partitioning a class of CIFAR-10 (the class is plane here) into two non-overlapped subsets, denoted as $X_{v1}$ and $X_{v2}$.(^2^22In practice, we perform T-SNE first to randomly split the non-overlapped samples from the embedding space.) Next, we keep $X_{v1}$ as non-training data and use $X_{v2}$ as training data to train a BigGAN [] to generate dataset $\hat{X}$. If our assumption holds, the generated data will be more similar to the training data $X_{v2}$.
As presented in the T-SNE plot (Figure 1), the generated data demonstrate a more substantial overlap in the shared feature space with the training dataset $X_{v2}$ than the non-training ones $X_{v1}$.(^3^33Quantitative statistical testing is provided in Appendix.)
This observation motivates us to address the data valuation for the generative model from a similarity-matching perspective.

It’s worth noting that concurrent to our research, LAVA [] also proposes the utilization of a similarity metric, specifically the class-wise Wasserstein distance between the training and validation sets, to evaluate data value. However, their emphasis lies in classification tasks and theoretical analysis, whereas our work contributes practical algorithms tailored for generative models.

Figure: Figure 2: Overview of GMValuator, a unified and training-free data valuation approach for any generative models. GMValuator contains three important modules – *(1) Efficient Similarity Matching (ESM), (2) Image Quality Assessment, and (3)Value Calculation*. Each generated data $\hat{x}_{j}$ is matched with training data through ESM approach, resulting in the distances with its top $k$ contributors. The normalized contribution score from training sample $x_{i}$ to $\hat{x}_{j}$, defined as $\exp(-d_{ij})/\sum_{i}^{k}\exp(-d_{ij})$, is adjusted based on the quality of the associated generated samples $q_{j}$. We compute the data value $\phi_{i}$ of each training sample $x_{i}$ by summing its contributions to the generated samples, where it ranks among the top $k$ contributors.
Refer to caption: extracted/5562797/figures/pipeline.png

### III-B Problem Formulation and Challenges

The primary objective of our research is to tackle the issue of data valuation in generative models using *black-box* access. In essence, given a fixed set of data samples $\hat{X}$ with a size of $m$ generated from the trained deep generative model $\mathcal{A}^{*}$, our aim is to determine the value $\phi_{i}(x_{i},\hat{X},\mathcal{A}^{*})$ associated with each data point $x_{i}\in X$ for $i\in[n]$ in the *deduplicated* training dataset, which contributes to the generated dataset. We denote the value for training data $x_{i}$ as $\phi_{i}$ in the rest of the paper for simplicity.

Following the motivation stated in Sec. III-A, $\phi_{i}$ should be a functions of the distance between the data points. We denote the distance between training data $x_{i}$ and generated data $\hat{x}_{j}$ as $d_{ij}$, for $i\in[n]$ and $j\in[m]$. The contribution score of $x_{i}$ to $\hat{x}_{j}$ is denoted as $\mathcal{V}(x_{i},\hat{x}_{j})\propto d_{ij}^{-1}$, which is inversely proportional to distance since maximizing the log-likelihood of a generative model is equivalent to minimizing the dissimilarity of real and generated data distribution. To link the dissmilarity $-d_{ij}$ to likelihood, we choose $exp(-d_{ij})$ to likelihood, and define $\mathcal{V}(x_{i},\hat{x}_{j})$ as follows:

$$ $\displaystyle\mathcal{V}(x_{i},\hat{x}_{j})=\frac{\exp{(-d_{ij})}}{\sum_{i} \exp{(-d_{ij})}}.$ (2) $$

Therefore, an intuitive definition for the value of data $i$ can be written as

$$ $\displaystyle\phi_{i}=\sum_{j=1}^{m}\mathcal{V}(x_{i},\hat{x}_{j}),x_{i}\in X, \hat{x}_{j}\in\hat{X}.$ (3) $$

The high-value data will achieve a smaller distance to the target distributions, thus, a better approximation. Therefore, data valuation in the above problem formulation contains two steps: *Step 1*, calculating all of the score value $\mathcal{V}(x_{i},\hat{x}_{j})$; *Step 2*, mapping the contribution from training data to generative data based on the scores $\mathcal{V}$.

However, there are several open questions and challenges in performing the above two steps for calculating Eq. (3):
Challenge 1: Efficiency. In step 1, considering $n$ training samples and $m$ generated samples, where $\mathcal{O}(C)$ represents the complexity of the selected pair-wise distance calculation, the total complexity of this step amounts to $\mathcal{O}(mnC)$. In practical scenarios with large training datasets (e.g., $n>10K$), the computation cost for becomes prohibitively expensive. Additionally, fitting
such a large collection of high-dimensional data for distance calculation can pose significance challenges in system memory.
Challenge 2: Contribution plausibility. To ensure that a training data point contributes more if it is similar to high-quality generated data and less if it is similar to low-quality generated data, the contribution scores should be adjusted based on the quality of the generated data.
Challenge 3: Non-zero scores. In practical scenarios, the distance between training data and the least similar generated data is not infinite, which may result in false non-zero contribution scores. With a large dataset size, the accumulation of these noisy scores yield biased data valuation.

In light of this, we present a novel and efficient data valuation approach suitable for agnostic generative models, termed as GMValuator
as elaborated in Section IV.

## IV The Proposed Data Valuation Methods

The crucial idea behind GMValuator is to transform the data valuation problem into a similarity matching problem between generated and training data. The overview of GMValuator is presented in Figure 2.
To tackle *challenge 1*, we propose to employ efficient similarity matching (ESM), where each generated data point can be linked to multiple contributors from the training dataset (Section IV-A), where each generated data firstly link with top $k$ contributors via recall phase and then re-rank by a refined similarity for effectiveness. After that, the image quality of the generated sample is assessed to weight the valuation (Sec. IV-B).
Finally, the value computation function combines both the quality score and the image-space similarity score to measure the value of the training data (Sec. IV-C).

### IV-A Efficient Similarity Matching

Considering the complexity of calculating $\mathcal{V}(x_{i},\hat{x}_{j})$, we formulate it as an ESM problem between generative data and training data. Each generative data point is matched to several training data points based on their similarity. We denote $\mathcal{P}_{j}=\left\{x_{1},x_{2},...,x_{k}\right\}=f(X,\hat{x}_{j})$ as the subset of training data that contains the $k\ll n$ most similar data points. Here, $f$ represents the similarity-matching strategy, encompassing the *recall* and *re-ranking* phases, which will be introduced subsequently.

Recall Phase:
The main aim of the *recall* phase is to rapidly identify a subset of training samples that are similar to a generated sample. To achieve this, an initial step involves encoding all original training images and generated samples from the image space $R^{C}$ to a lower-dimensional embedding space $R^{D}$ using a pre-trained encoder $f_{e}$ (such as VGG-16) to reduce computational complexity. Subsequently, the technique of Product Quantization (PQ)  [] is employed to further decrease the computational burden. Specifically, PQ divides embedding vectors into subvectors and independently quantizes each subvector through $Q$-means clustering. This process generates compact PQ codes that serve as representations of the original vectors. This representation significantly reduces the vector sizes, allowing for efficient estimation of Euclidean Distance between two samples. By incorporating the *recall* method into the similarity matching process, the computational complexity is lowered from $\mathcal{O}(mnC)$ to $\mathcal{O}(mQD)$, where $D\ll C$ and $Q\ll n$. Consequently, generated images can quickly identify their top-$k$ most similar training data points.

Re-Ranking Phase:
Following the PQ-based efficient recall process in GMValuator, we further improve the precision of the results by utilizing perceptual similarity [] for precision ranking. Once we have extracted perceptual features from the top $k$ recalled training samples, we proceed to calculate the distance for each pair of items. To obtain precise distance measurements based on their perceptual content, we propose to use Learned Perceptual Image Patch Similarity (LPIPS) [] or DreamSim [] as the distance measurement $d$ to gain insights into the perceptual dissimilarity between the generated sample and different training samples. These metric enable us to precisely measure the most significant contributors according to their perceived similarity, more importantly, the obtained distance $d$ will be employed to compute data valuation in Eq. (4). We do not use Wasserstein distance as it is more proper to measure dissimilarity between two probability distributions rather than a pair of image instances.

Figure: Figure 3: The value without generated image quality calibration for q high-quality image (top row) and a low-quality image (bottom row). Column 1: generated images. Column 2-5:their top 4 contributors.
Refer to caption: x2.png

### IV-B Image Quality Assessment

To tackle *challenge 2*, we have to establish a connection between the quality of the generated sample that the training samples contribute to and the contribution score. This is necessary before assigning contribution scores to the ranked training samples for the generated sample. For low-quality generated samples $\hat{x}_{\rm low}$, we expect their total contribution score from contributors to be lower compared to high-quality samples $\hat{x}_{\rm high}$, namely $\sum_{i\in[k]}\mathcal{V}(x_{i},\hat{x}_{\rm low})<\sum_{i\in[k]}\mathcal{V}(x
_{i},\hat{x}_{\rm high})$. This motivation arises from the observations in Figure 3, where the first and second rows show generated samples using $z$ from a normal distribution and uniform distribution, respectively. The values denoted in Figure 3 directly employs Eq (2), so we have $\sum_{x_{i}\in\mathcal{P}}\mathcal{V}(x_{i},\hat{x})=1$ for $\forall\hat{x}\in\hat{X}$. The real samples are noticeably dissimilar to the generated data, yet they are still assigned a high value.
Therefore, we propose to calibrate the contribution scores with generated data quality. Specifically, we obtain a comprehensive quality score $q_{j}\in[0,1]$ for each generated image integrate using MANIQA [] model into our evaluation process. A higher $q_{j}$ indicate better data quality. This score of MANIQA considers various factors such as sharpness, color accuracy, composition, and overall visual appeal. Incorporating the image quality evaluation provided by MANIQA allows us to more accurately assess the generated images and take into account their perceptual fidelity and aesthetic qualities.

Figure: Algorithm 1 GMValuator

### IV-C Value Calculation

Finally, we utilize the image quality $q$ to calibrate the contribution scores. Notably, during the recall phase of similarity matching, we select the top $k$ contributors $\mathcal{P}_{j}$ for the generated data $\hat{x}_{j}$. Consequently, we only consider the contribution score between $\mathcal{P}_{j}$ and $\hat{x}_{j}$ by setting $\mathcal{V}(x_{i},\hat{x}_{j})=0$ for $x_{i}\notin\mathcal{P}_{j}$. This strategy addresses challenge 3 by assigning zero scores to irrelevant samples, effectively reducing bias and noise in value estimation when dealing with a large $n$. Hence, we define the contribution score function of each training data point for a specific synthetic data point as follows:

$$ $\begin{split}\mathcal{V}(x_{i},\hat{x}_{j},d_{ij},q_{j})=\begin{cases}q_{j} \cdot\frac{\exp{(-d_{ij}})}{\sum_{i=1}^{k}\exp{(-d_{ij}})}&x_{i}\in\mathcal{P} _{j}\\ 0&x_{i}\notin\mathcal{P}_{j}\end{cases}\end{split}$ (4) $$

which is used to give different scores to training data points according to their ranking distance. The final data value is obtained by plugging Eq. (4) into Eq. (3).

## V Experiments

Given the absence of a definitive benchmark to evaluate data valuation methods in the context of generative models, we propose to evaluate the truthfulness of data valuation outcomes through the examination of four criteria (C):

- •
C1: Identical Class Test. Following the concept of an identical class [], we posit that the most significant contributors among the training samples should belong to the same class as the generated sample produced by a well-trained generative model, which is also used for evaluation in VAE-TrancIn [].
- •
C2: Identical Attributes Test. We consider the attributes of samples as the ground truth to examine the most significant contributors by our approach. We examine the overlap level of attributes (i.e., the number of identical attributes) between the most valuable training samples with the generated sample
- •
C3: Data Cleansing. Under the assumption, noisy training data samples can be identified as low-value training data. To evaluate the performance of data valuation approaches, we consider the value level of noisy data samples as the performance metric across various approaches.
- •
C4: Efficiency. As an efficient training-free approach, GMValuator should measure the data value in a limited time and be much more efficient than the previous work while obtaining truthful results.

**TABLE I: Experiment settings.**
|  | Dataset | Network |
| --- | --- | --- |
| C1 | MNIST, CIFAR-10 | GAN, Diffusion, $\beta$-VAE |
| C2 | CelebA | Diffusion-StyleGAN |
| AFHQ | StyleGAN |  |
| C3 | Noise MNIST | DCGAN |
| C4: Hardware Evironment |  |  |
| CPU | One RTX 3080 (10GB) GPU |  |
| GPU | 12 vCPU<br>Intel(R) Xeon(R),<br>Platinum 8255C CPU @ 2.50GHz |  |

Experiment Setup.
Our experiment settings are listed in in Table I. Particularly, we consider different type of generative models, including GAN based models, $\beta$-VAE [], and diffusion models [, ]. The generation tasks are conducted on benchmark datasets (i.e., MNIST [] and CIFAR []), face recognition dataset (i.e., CelebA []) and high-resolution image dataset with size 512 $\times$ 512, and 1024 $\times$ 1024 (i.e., AFHQ [], FFHQ []).

Note that GMValuator is a model agnostic method. To showcase its efficacy, we compare it with two baseline methods: VAE-TracIn [] and IF4GAN []. VAE-TracIn identifies the most significant contributors for a generated sample, which is same to the similarity matching process of GMValuator (Sec. IV-A). Thus, we compare GMValuator with VAE-TracIn in Sec. V-A and Sec. V-B. IF4GAN finds the high-valued training samples for generative model training, which we regard it as baseline method in section V-C. We also examine our approach using DreamSim, LPIPS and $l_{2}$-distance in the re-ranking phase, referred to as GMValuator (DreamSim), GMValuator (LPIPS) and GMValuator ($l_{2}$-distance) respectively, while the GMValuator without the re-ranking phase is referred to as GMValuator (No-Rerank).

### V-A Identical Class Test (C1)

Methodology.
In this subsection, we follow setup of VAE-TracIn [] by training separate $\beta$-VAE models on MNIST [], CIFAR []. We then attribute the most significant contributors by VAE-TracIn and our approach over generated samples. By the concept of identical class test, we expects a perfect data valuation approach should indicate training data in the same subclass of the generated sample $\hat{x}_{j}$ contribute more to $\hat{x}_{j}$. We examine GMValuator (DreamSim), GMValuator (LPIPS), GMValuator ($l_{2}$-distance) and GMValuator (No-Rerank) separately. Apart from that, due to our method is model-agnostic, we also valid our method on GAN, Diffusion model, which are other kinds of generative models.

Results.
For a given generated data, we examine the class(es) of is top $k$ contributors in the training data. We count the number of training $Q$ samples in the top $k$ contributors, which has the identical class as the generated data. The identical class ratio $\rho=Q/k$. We report the averaged $\rho$ over the generated datasets (the data size $m$=100) on different choices of $k$ in Table  II. GMValuator has the most contributors that belong to the same class with the generated sample among on MNIST and CIFAR-10, while GMValuator (DreamSim) has the best performance. The results for CIFAR-10 using VAE-TracIn is extremely bad. This could be attributed to underfitting, as the authors mentioned in their study [], training a good $\beta$-VAE model on CIFAR-10 is challenging. The results of other diverse generative models are show in Table VI in appendix.

**TABLE II: Performance comparison of Identical Class Test.**
| MNIST (%) | $k$=30 | $k$=50 | $k$=100 |
| --- | --- | --- | --- |
| VAE-TracIn | 72.00 | 71.11 | 68.58 |
| GMValuator (No-Rerank) | 86.41 | 85.13 | 85.95 |
| GMValuator ($l_{2}$-distance) | 87.76 | 86.95 | 85.92 |
| GMValuator (LPIPS) | 88.78 | 88.19 | 86.69 |
| GMValuator (DreamSim) | 88.78 | 88.05 | 86.84 |
| CIFAR-10 (%) | $k$=30 | $k$=50 | $k$=100 |
| VAE-TracIn | 6.28 | 3.77 | 1.88 |
| GMValuator (No-Rerank) | 72.66 | 72.25 | 70.71 |
| GMValuator ($l_{2}$-distance) | 72.66 | 72.25 | 70.71 |
| GMValuator (LPIPS) | 72.60 | 71.75 | 70.84 |
| GMValuator (DreamSim) | 77.94 | 76.41 | 73.47 |

Figure: Figure 4: Visualization for Identical Attributes Test on CelebA. Left: generated samples. Right: top $k$ contributors.
Refer to caption: extracted/5562797/figures/attributes_tack_visualization.png

### V-B Identical Attributes Test (C2)

Methodology.
By regarding some attributes of images as the ground-truth, we assume that the most significant contributors should have the similar attributes to the generated sample. We train Diffusion-StyleGAN [] on CelebA, and leverage our approach to identify the most significant contributors in training samples, and use some some attributes including glasses, hair, hat, sex as the ground-truth, to check the correctness of our approach for identifying the most significant contributors for a generated sample. We also examine this on AFHQ, FFHQ dataset by StyleGAN [].

Results.
The results in Table III indicates that GMValuator can find the most significant contributors with the same attributes to the generated sample. Besides, using DreamSim in similarity re-ranking phase obtains the best performance over these three attributes. We also visualize some visible results in Figure 4 on CelebA and Figure 6 on AFHQ, FFHQ in Appendix. As we can see, the most significant contributors have the similar attributes (e.g., skin color, hair) to the generated sample.

**TABLE III: Performance of Identical Attributes Test of some attributes including Hat, Sex, and Eyeglasses on CelebA.**
| Top K contributors: | $k$=5 | $k$=10 | $k$=15 |
| --- | --- | --- | --- |
| Attribute: Hat (%) |  |  |  |
| GMValuator (No-Rerank) | 92.52 | 92.12 | 91.92 |
| GMValuator ($l_{2}$-distance) | 90.91 | 89.90 | 89.83 |
| GMValuator (LPIPS) | 96.77 | 96.57 | 96.90 |
| GMValuator (DreamSim) | 97.78 | 97.07 | 96.90 |
| Attribute: Gender (%) |  |  |  |
| GMValuator (No-Rerank) | 75.15 | 75.25 | 75.82 |
| GMValuator ($l_{2}$-distance) | 63.64 | 61.01 | 60.00 |
| GMValuator (LPIPS) | 97.98 | 97.48 | 97.37 |
| GMValuator (DreamSim) | 99.19 | 98.99 | 98.79 |
| Attribute: Eyeglasses (%) |  |  |  |
| GMValuator (No-Rerank) | 91.52 | 91.52 | 91.45 |
| GMValuator ($l_{2}$-distance) | 94.95 | 94.65 | 93.80 |
| GMValuator (LPIPS) | 94.95 | 94.65 | 93.80 |
| GMValuator (DreamSim) | 96.77 | 96.26 | 95.96 |

Figure: Figure 5: The y-axis represents the ranking of values from high to low, with the top being the highest value and the bottom being the lowest value. The x-axis represents the index of each noisy data.
Refer to caption: extracted/5562797/figures/dirty_value.png

### V-C Data Cleaning (C3)

Methodology
In this experiment, we introduce 100 noisy images into the MNIST dataset to create a new corrupted training dataset and each noisy image is generated by adding Gaussian noise to the clean data. These contaminated samples can diminish the performance of generative models, and as a result, they are anticipated to possess lower values compared to the rest of the training dataset. Our objective is to assess the performance of both GMValuator and its baseline method, IF4GAN [], by analyzing the value rankings of noisy samples. An effective approach should place noisy data in lower positions within the value ranking. We conducted this evaluation using 10,000 generated images and referred to the study [] for the baseline method (IF4GAN).

Results.
The results are depicted in Figure 5 demonstrate that the values of all noisy training samples, as calculated by GMValuator, are lower compared to the values calculated by IF4GAN. This observation suggests that the performance of GMValuator is significantly better than that of IF4GAN. Besides, the performance of GMValuator with re-ranking phase is better than GMValuator (No-rerank).

### V-D Efficiency (C4)

Methodology.
We thoroughly evaluate and compare the efficiency of our approach against existing data valuation methods for generative models discussed above: VAE-TracIn and IF4GAN. Specifically, we measure the attribute time for one generated sample on an average of 100 test samples using both VAE-TracIn and our proposed approach on MNIST and CIFAR-10. Additionally, under the same setting in C3, we assess the data valuation time when utilizing IF4GAN and our approaches on noise MNIST which we mentioned in C3.

Results.
The average time taken to attribute the most significant contributors for one generated sample is calculated and compared with the baseline method (VAE-TracIn), as demonstrated in Table IV. Our approaches are all greatly more efficient than the baseline methods on both datasets. When it comes to data valuation for GAN, GMValuator (No-rerank) and GMValuator (LPIPS) are significantly better than IF4GAN. The aforementioned phenomenon is attributed to the costly nature of the Hessian estimation process, despite the utilization of certain acceleration methods. It is noticable that GMValuator (DreamSim) is lightly time-consuming when compare to IF4GAN. This lead to a trade off problem due to the GMValuator using DreamSim in re-ranking phase obtained the better performance from C1 to C3.

Overall, through the above experiments, GMValuator outperform than baseline methods, while GMValuator (DreamSim) obtains the best performance. This is due to the fact that DreamSim capture mid-level similarities in image semantic content and layout compare to LPIPS. And the selection of different GMValuators can be combined with the consideration of their efficiency.

**TABLE IV: Efficiency Comparison**
| Attribute for VAE | Time(s) |  |
| --- | --- | --- |
| MNIST | VAE-TracIn | 47.945 |
| GMValuator (No-Rerank) | 0.250 |  |
| GMValuator ($l_{2}$-distance) | 0.339 |  |
| GMValuator (LPIPS) | 0.477 |  |
|  | GMValuator (DreamSim) | 1.709 |
| CIFAR-10 | VAE-TracIn | 66.178 |
| GMValuator (No-Rerank) | 0.755 |  |
| GMValuator ($l_{2}$-distance) | 1.226 |  |
| GMValuator (LPIPS) | 2.412 |  |
|  | GMValuator (DreamSim) | 15.491 |
| Data Valuation for GAN | Time(s) |  |
| Noise MNIST | IF4GAN | 14,543 |
| GMValuator (No-Rerank) | 2,137 |  |
| GMValuator ($l_{2}$-distance) | 3,388 |  |
| GMValuator (LPIPS) | 4,771 |  |
|  | GMValuator (DreamSim) | 17,086 |

## VI Conclusion

To measure the contribution of each training data sample, we propose an efficient approach, GMValuator, for generative model. As far as we are aware, there is
no prior model-agnostic and training-free data valuation approach for generative models util GMValuator. Our approach is based on efficient similarity matching, and it enables us to calculate the final value of each training data point, aligning with plausible assumptions. The proposed method is validated through a series of comprehensive experiments to showcase its truthfulness and efficacy on four criteria. In the future, we will validate the proposed methods on other data modalities.

## Appendix A Statistical Results for Figure 1

It is evident by visualization in Figrue 1 that the data points in $X_{v2}$ (used for training) are more overlapped with generated data than data points in $X_{v1}$ (not used for training). We perform statistic testing on data values obtained by GMvaluator, to examine if data points $X_{v2}$ (used for training) have significant higher values than those of the data points in $X_{v1}$ (not used for training).

**TABLE V: The statistic test of data values of $X_{v1}$ versus $X_{v2}$ using different generative models. $X_{v2}$ is supposed to have higher value than $X_{v1}$, given the generated data.**
| $H_{0}$: $\phi(D_{i},S,\mu_{i})\geq\phi(D_{j},S,\mu_{i})$<br>$H_{1}$: $\phi(X_{i},S,\mu_{i})<\phi(X_{j},S,\mu_{i}),i\in X_{v1},j\in X_{v2}$ |  |
| --- | --- |
| BigGAN |  |
| Average value (v1) | 0.319654 |
| Average value (v2) | 1.632352 |
| P-value | 6.937027 $\times 10^{-68}$ |
| T-statistic | 17.924512 |
| Significance level | 0.01 |
| Result | p-value less than 0.01, reject $H_{0}$,<br>value of v2 less than v1 averagely |
| Classifier-free Guidance Diffusion |  |
| Average value (v1) | 0.030434 |
| Average value (v2) | 0.369565 |
| P-value | 8.053195 $\times 10^{-55}$ |
| T-statistic | 15.947860 |
| Significance level | 0.01 |
| Result | p-value less than 0.01, reject $H_{0}$ |

To this end, we use a t-test with null hypothesis that data values in $X_{v1}$ should not be samller than those of $X_{v12}$.
We compute a $p$-value, which is the probability of getting a difference as large as we observed, or larger, under the null hypothesis. If the $p$-value is very low, we reject the null hypothesis and consider our approach, GMValuator, to be verified with a high level of confidence (1-$p$). Typically, a $p$-value smaller than significance level 0.01 is used as a threshold for rejecting the null hypothesis.
Table V showcases the outcomes of $X_{v1}$ and $X_{v2}$ in CIFAR-10 with $p\ll 0.01$ for both BigGAN and diffusion model,
indicating that the data points in $X_{v2}$ have significantly more value than those in $X_{v1}$. Consequently, these findings align with the presumption that the trained dataset $X_{v2}$ has a higher value than the untrained dataset $X_{v1}$ and verify our approach.

Figure: Figure 6: Visualization of Identical Attributes Test on AFHQ and FFHQ. The results shown in the first and second subfigures on the left are conducted on AFHQ-Cat and AFHQ-Dog, respectively. The subfigure on the right presents the results on FFHQ. In each subfigure, the generated samples are on the left, and the top $k$ contributors in the training dataset are on the right.
Refer to caption: x3.png

## Appendix B Additional Results on C1 and C2

**TABLE VI: Performance comparison of Identical Class Test.**
| MNIST |  |  |  |
| --- | --- | --- | --- |
| GAN (%) | $k$=30 | $k$=50 | $k$=100 |
| GMValuator (No-Rerank) | 96.27 | 96.26 | 95.86 |
| GMValuator ($l_{2}$-distance) | 97.73 | 97.58 | 96.03 |
| GMValuator (LPIPS) | 97.77 | 97.72 | 97.38 |
| GMValuator (DreamSim) | 97.43 | 97.44 | 97.40 |
| Diffusion (%) | $k$=30 | $k$=50 | $k$=100 |
| GMValuator (No-Rerank) | 92.40 | 91.82 | 91.26 |
| GMValuator ($l_{2}$-distance) | 92.90 | 92.66 | 91.88 |
| GMValuator (LPIPS) | 93.73 | 97.72 | 92.42 |
| GMValuator (DreamSim) | 93.90 | 93.44 | 92.55 |
| CIFAR-10 |  |  |  |
| BigGAN (%) | $k$=30 | $k$=50 | $k$=100 |
| GMValuator (No-Rerank) | 64.70 | 63.80 | 62.14 |
| GMValuator ($l_{2}$-distance) | 64.70 | 63.80 | 62.14 |
| GMValuator (LPIPS) | 63.67 | 62.80 | 61.51 |
| GMValuator (DreamSim) | 70.33 | 68.74 | 65.18 |
| Class-free Guidance Diffusion (%) | $k$=30 | $k$=50 | $k$=100 |
| GMValuator (No-Rerank) | 72.67 | 72.00 | 71.00 |
| GMValuator ($l_{2}$-distance) | 72.67 | 72.00 | 71.00 |
| GMValuator (LPIPS) | 72.53 | 72.28 | 71.06 |
| GMValuator (DreamSim) | 79.37 | 78.08 | 74.61 |

### B-A Identical Class Test on Other Generative
Models

We have presented Indetical Class Test (C1) on $\beta$-VAE and MNIST [], CIFAR-10 [] in Sec V-A in our main context. Since GMValuator is model-agnostic, we further validate our method of C1 on other generative models.

Here, we conduct the experiments using a GAN and a Diffusion Model on MNIST.
The architectural details of the used generative models are described in Sec E-B in Appendix. We also conduct experiment on BigGAN [] and Class-free Guidance Diffusion [] with CIFAR-10. We used the same number of generated samples $m=100$ as the experiments presented in Sec V.

Following the similar settings in Sec. V-A (C1), we examine the class(es) of is top $k$ contributors for a given generated data in the training data.
We calculate the number of training samples, denoted as $Q$, from the top $k$ contributors that have the same class as the generated data. The identical calss ratio, denoted as $\rho$, is calculated as $\rho=Q/k$. We report the average value of $\rho$ across the generated datasets for different choices of $k$ in Table VI.
GMValuator (DreamSim) has the highest ratio of contributors that belong to the same class as the generated sample among most of the models evaluated on MNIST and CIFAR-10 datasets for different values of $k$. And the ratio improves as the value of $k$ decreases, which is consistent to the top $k$ assumption and validates our method.

### B-B Identical Attributes Test on Other Datasets

To further validate GMValuator, we also conducted experiments on high-resolution datasets: AFHQ [] and FFHQ [], following the same settings in Sec V-B.
The results are shown in Figure 6, which demonstrates the effectiveness of our methods in C2. The results show that the top $k$ contributors have the similar attributes with the generated sample such as the fur color of cats or dogs in AFHQ. For the experiment conducted on FFHQ, human faces attributes of the most significant contributors are also similar to the attributes in generated images.

Figure: Figure 7: The change of $\rho$ with the different number of generated samples $m$ on MNIST and CIFAR-10 by diverse generative models.
Refer to caption: extracted/5562797/figures/mnist_gan_30.png

## Appendix C Different Generated Data Sizes

Since our value function $\phi_{i}$ (Eq. (3)) for training data $x_{i}$ is computed by averaging over generated samples, it is expected that the sensitivity of $\phi_{i}$ is connected to the size $m$ of the investigated generated sample. To explore the influence of generated data size $m$ on the utilization of GMValuator, we perform sensitivity testing on MNIST, CIFAR10 using generative models GAN [], and Diffusion models [] as depicted below. The dataset, model and used $k$ are denoted under each subfigure of Figure 7. First, we generate a varying number of samples from the same class. Specifically, we consider four different sample sizes, denoted by $m$, which are given by 1, 10, 30, 50. Next, we evaluate the GMValuator using parameter C1, and this evaluation is performed for each of the aforementioned values of $m$. Subsequently, we conduct the experiment 10 times using GMValuator (No-Rerank), each time with different
$m$-sized generated data samples from the same class. The results are presented as the mean and standard deviation ($\rho$) for accuracy, taken over these 10 runs. The results shown in Figure 7 imply that varying $m$ does not yield notable differences in mean accuracy and increasing the number of generated samples $m$ leads to more stable and consistent results.

## Appendix D Alternative Distance Metric

We suggest utilizing Learned Perceptual Image Patch Similarity (LPIPS) [] or DreamSim [] as the distance metric $d$ during the re-ranking phase. This enables understanding the perceptual dissimilarity between generated and real data points. These metrics are applied in the image embedding space derived from pretrained models. Alternatively, distance measurement can be based on pixel space, such as employing $l2$-distance. Notably, we find that distances calculated in the input pixel space yield comparable outcomes as shown in Table VI. This implies that our method GMValuator could be flexible to the difference choices of distance metrics and the selection can depends on data prior.

## Appendix E Additional Experimental Details

### E-A Datasets

We conduct the generation tasks in the experiments on benchmark datasets (i.e., MNIST [] and CIFAR []), face recognition dataset (i.e., CelebA []), and high-resolution image dataset AFHQ [] and FFHQ [].

MNIST. The MNIST dataset consists of a collection of grayscale images of handwritten digits (0-9) with a resolution of 28x28 pixels. The dataset contains 60,000 training images and 10,000 testing images.

CIFAR-10.
CIFAR-10 dataset consists of 60,000 color images in 10 different classes, with 6,000 images per class. The classes include objects such as airplanes, cars, birds, cats, deer, dogs, frogs, horses, ships, and trucks. Each image in the CIFAR-10 dataset has a resolution of 32x32 pixels.

CelebA.
The CelebA dataset is a widely used face recognition and attribute analysis dataset, which contains a large collection of celebrity images with various facial attributes and annotations. The dataset consists of more than 200,000 celebrity images, with each image labeled with 40 binary attribute annotations such as gender, age, facial hair, and presence of eyeglasses.

AFHQ.
The AFHQ dataset is a high-resolution image dataset that focuses on animal faces (e.g., dogs, cat), and it consists of high-resolution images with 512 $\times$ 512 pixels.

FFHQ.
The FFHQ dataset is a high-resolution face dataset that contains high-quality images (1024x1024 pixels) of human faces.

### E-B Architecture of Generative models

In our experiments, we leverage different generative models in the class of GAN, VAE and diffusion model.
We utilize $\beta$-VAE for both MNIST and CIFAR-10 datasets while a simple GAN is conducted on MNIST.
BigGAN and $\beta$-VAE are also conducted on CIFAR-10. We list the architecture details for these generative models in Table VII. StyleGAN is used for high-resolution dataset AFHQ and FFHQ. CelebA uses Diffusion-StyleGAN [], for which we use the exact architecture in their open sourced code.

## Appendix F Discussion on the Possible Applications

The application of data valuation within generative models offers a wide range of opportunities. A potential use case is to quantify privacy risks associated with generative model training using specific datasets, since the matching mechanism GMValuator can help re-identify the training samples given the generated data. By doing so, organizations and individuals will be able to audit the usage of their data more effectively and make informed decisions regarding its use.

Another promising application is material pricing and finding in content creation. For example, when training generative models for various purposes, such as content recommendation or personalized advertising, data evaluation can be used to measure the value of reference content.

In addition, GMValuator can play an important role in the development of ensuring the responsibility of using synthetic data in safe-sensitive fields, such as healthcare or finance. By assessing the value of the data used in generative model training, researchers can ensure that the generated data are robust and reliable.

Last but not least, the applications of GMValuator can promote the recognition of intellectual property rights. Determining the value of the intellectual property being generated by generative models is critical.
By evaluating the data employed in training generative models, we can develop a more comprehensive understanding of copyright that may emerge from the generative models. In essence, such insights can help advance licensing agreements for the utilization of the generative model and its outputs.

**TABLE VII: The architecture of $\beta$-VAE and BigGAN.**
| GAN for MNIST |  |
| --- | --- |
| Generator | Discriminator |
| FC(100, 8192), BN(32), ReLU | Conv2D(1, 128, 4, 2, 1), BN(128), LeakyReLU |
| Conv2D(128, 64, 4, 2, 1), BN(64), ReLU | FC(8192, 1024), BN(1024), LeakyReLU |

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 2
questions: 1.	As stated by the authors: However, this does not mean that GMVALUATOR cannot be easily adapted to Natural Language Processing (NLP) fields, given its core idea of similarity matching. So,  how to apply the proposed method to generative models for text-based data or data from multiple modalities?
2.	If data valuation is not used in training generative models, what will be the limitations? Can the proposed method or evaluation metrics measure the limitations?
soundness: 3
strengths: 1.	The authors proposed GMVALUATOR to tackle the data valuation issue for generative models. GMVALUATOR is innovative, and model-agnostic, enabling broad applicability and adaptability across various generative models. Besides,  GMVALUATOR does not require retraining of models, offering the advantage in R&D scenarios with limited computational resources.
2.	The authors provide detailed theoretical justification for formulating data valuation for generative models as a similarity-matching problem. They also provide empirical validation for this motivation. This makes the GMVALUATOR being very reasonable.
3.	The authors demonstrated through extensive experimental results that the data valuation method proposed can achieve good results in multiple aspects and over multiple generative models. Besides, they also provided validation of the experimental settings and the open-source implementation, which further increased the credibility of the results.
summary: The paper introduces GMVALUATOR to solve the data valuation problem for vision generative models, which is important yet has been overlooked by existing studies. GMVALUATOR formulates data valuation as a similarity-matching problem and incorporates image quality assessment to calibrate the contributions of data samples. Despite high computational complexity with large datasets, GMVALUATOR demonstrates effectiveness across multiple datasets and generative models, positioning it as a promising tool for data valuation in the field. Although the technical contribution is obvious, there is still much room for improvement in the author's presentation and content arrangement.
weaknesses: 1.	The manuscript is not well written. For example, in the Introduction, before talking about the existing work, it is suggested to generally define/introduce the data valuation problem (including the input and the objective). Moreover, the authors didn't highlight the urgent need for data valuation in existing generative models; this poses a challenge to the motivation of this paper. Most importantly, instead of briefly introducing the principle of the proposed GMValuator (such as why and how to formulate data valuation for generative models as an efficient similarity-matching problem), the author only introduces the goal to be achieved. These places make the content difficult to read.
2.	Although the authors claim that they proposed a versatile data valuation method for generative models, in both the problem formulation, method introduction, and subsequent experiments, they only evaluate it on image samples. Therefore, the statement "for generative models" may be inappropriate. Even though the authors mentioned in the Current Limitation "However, this does not mean that GMVALUATOR cannot be easily adapted to Natural Language Processing (NLP) fields, given its core idea of similarity matching." However, the reason for this statement may need further explanation.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 4
questions: Please refer ti weaknesses.
soundness: 4
strengths: - This is the first paper on data valuation on generative models. Previous data valuation methods focus on discriminative models and cannot adapt to generative models.
- Compared to the retraining-based and influence-based methods, GMValuator is efficient. It does not require any retraining or computation of hessian.
- GMValuator is effective on the proposed metrics. GMValuator has significantly improved compared to baseline methods.
summary: The paper focuses on the data valuation of generative models. Existing data valuation methods designed for discriminative models cannot adapt to generative models due to 1) lack of robust performance metrics; 2) the large size of generative models; and 3) lack of data labels. In order to mitigate this gap, the authors propose GMValuator. GMValuator is based on similarity matching between training data and generated data. If a training sample is similar to a generated sample, it is considered to have contributed to the generated sample. The value of a training sample is computed by the quality of its contributed generations. Four evaluation criteria are introduced to assess data valuation. Experiments demonstrate the effectiveness of the proposed GMValuator.
weaknesses: - For SOTA text-to-image models like stable diffusion, the image domain is much wider than the test models. As a result, a large number of generated images may be required for accurate data valuation. Meanwhile, generation with these models is slow. More results and ablation on stable diffusion on the SOTA text-to-image models would be helpful.
- While the proposed metrics are intuitively reasonable, it is coarse-grained and may not be able to reflect the effectiveness of data evaluation methods.

The authors addressed my concerns.
I raise my score to 6.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 3
questions: - What is the effect of the quantization step in the recall phase? 
- Empirically, how different is the proposed score from a nearest neighbor search with the rerank metric in the embedding space?
- How well does the proposed method generalize to other generative models, such as diffusion models?
soundness: 4
strengths: - The paper introduces a novel and intuitive idea for data valuation in generative models, and the results are promising.
- The experiments are well-designed, exploring multiple distance functions and encoders to validate the approach. Also, multiple test scenarios were covered, all showing good supporting results for the proposed method.
- The paper is well-written and easy to follow, effectively conveying the methodology and findings.
- The paper covers relative literature well.
summary: This paper proposes a novel data valuation method for generative models. The authors introduce a model-agnostic, training-free data valuation framework, addressing a significant challenge in the field: existing methods typically require retraining or Hessian calculations, which are computationally intractable. In this approach, the contribution of a training data point to a generated data point is defined as inversely proportional to the distance between the two data points. The author performed extensive experiments against two baselines and showed effectivenss of the propsoed method.
weaknesses: - The impact of the quantization step on the final results is not explored. Understanding this effect would provide a clearer picture of the method’s performance.
- While section 2 introduces some underlying assumptions and a theoretical motivation for using a similarity-guided data valuation score (illustrated in Figure 1), the framework would benefit from a more rigorous theoretical foundation. Further studies on theoretical support could strengthen the framework’s conceptual grounding and its reliability across different applications.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: GMVALUATOR is claimed as the first modal-agnostic and retraining-free data valuation method for generative models. How about the comparison with other approaches such as Information-Theoretic Measures (Akhilan Boopathy et al., ICML 2023, "Model-agnostic Measure of Generalization Difficulty"), etc.?
Please illustrate whether other approaches are applicable to generative models in a model-agnostic and retraining-free manner, which is important to your claim.
soundness: 3
strengths: The strengths of this paper are listed as the following:

1. GMVALUATOR is claimed to be the first modal-agnostic and retraining-free data valuation method for generative models.

2. The authors formulate data valuation for generative models as an efficient similarity-matching
 problem. The paper further eliminates the biased contribution measurement by introducing image
 quality assessment for calibration.

 3. The authors propose four evaluation methods to assess the truthfulness of data valuation and evaluate GMVALUATOR on different datasets, such as benchmark datasets and high-resolution
large-scale datasets, and various deep generative models to verify GMVALUATOR’s validity.
summary: In this paper, the authors introduce Generative Model Valuator (GMVALUATOR), the first training-free and model-agnostic approach to providing data valuation for generation tasks. The authors formulate data valuation for generative models as an efficient similarity-matching problem. The paper further eliminates the biased contribution measurement by introducing image quality assessment for calibration. Also, the paper introduced four evaluation criteria for assessing data valuation methods in generative models.
weaknesses: The weaknesses of the paper are listed as follows:

1. The paper only mentioned 4 criteria for assessing data valuation methods. They are: C1: Identical Class Test; C2: identical attributes test; C3: Out of Distribution Detection; and C4: Efficiency.  

How about other criteria? 
Why only use these 4? 

For example, how about Cost-Benefit Analysis (i.e., the trade-offs between the costs of acquiring or processing data versus the performance gains from using it in model training)?
Please give some examples and formulas to measure the costs of data acquisition/processing. Then compare them with the performance gains of those data in the context of generative model training. This will be more practical for the proposed new approach. 

2. The paper shall discuss the important aspects of data evaluation, such as accuracy and complexity. For example, how accurate is it for the proposed framework? How much is the complexity while implementing the proposed approach? Also, the authors shall discuss the proposed framework on accurately capturing the contributions of individual data points in various different scenarios. 

3. The paper shall illustrate other aspects such as scalability, whether the proposed approach is useful for handling big data in real-time applications.  For example, the authors can provide the results of testing the proposed method on progressively larger datasets or measuring processing times for different data sizes.

Meta-review: This paper studies data valuation for generative models which is a topic not much explored in the research community. It proposes a training-free and model-agnostic approach based on similarity matching. It also introduces four evaluation criteria.

Major strengths:
- Data valuation for generative models is an important topic that has not yet received sufficient attention.
- The proposed method has the advantages of being training-free, model-agnostic, and efficient.
- Theoretical justification is provided for formulating the data valuation problem as similarity matching.
- Extensive experiments are conducted to support the claims.

Major weaknesses:
- Experimental validation is only conducted on image data but not other data modalities.

We thank the authors for addressing the comments raised by the reviewers in their reviews and revising their manuscript to address some of them.

Decision: Accept (Poster)