Title: Where Did I Come From? Origin Attribution of AI-Generated Images
Venue: NeurIPS 2023
Accepted: yes
Keywords: Origin attribution of generated images
Abstract: Image generation techniques have been gaining increasing attention recently, but concerns have been raised about the potential misuse and intellectual property (IP) infringement associated with image generation models. It is, therefore, necessary to analyze the origin of images by inferring if a specific image was generated by a particular model, i.e., origin attribution. Existing methods only focus on specific types of generative models and require additional procedures during the training phase or generation phase. This makes them unsuitable for pre-trained models that lack these specific operations and may impair generation quality. To address this problem, we first develop an alteration-free and model-agnostic origin attribution method via reverse-engineering on image generation models, i.e., inverting the input of a particular model for a specific image. Given a particular model, we first analyze the differences in the hardness of reverse-engineering tasks for generated samples of the given model and other images. Based on our analysis, we then propose a method that utilizes the reconstruction loss of reverse-engineering to infer the origin. Our proposed method effectively distinguishes between generated images of a specific generative model and other images, i.e., images generated by other models and real images.

Where Did I Come From? Origin Attribution of AI-Generated Images

Abstract

Image generation techniques have been gaining increasing attention recently, but concerns have been raised about the potential misuse and intellectual property (IP) infringement associated with image generation models. It is, therefore, necessary to analyze the origin of images by inferring if a specific image was generated by a particular model, i.e., origin attribution. Existing methods only focus on specific types of generative models and require additional procedures during the training phase or generation phase. This makes them unsuitable for pre-trained models that lack these specific operations and may impair generation quality. To address this problem, we first develop an alteration-free and model-agnostic origin attribution method via reverse-engineering on image generation models, i.e., inverting the input of a particular model for a specific image. Given a particular model, we first analyze the differences in the hardness of reverse-engineering tasks for generated samples of the given model and other images. Based on our analysis, we then propose a method that utilizes the reconstruction loss of reverse-engineering to infer the origin. Our proposed method effectively distinguishes between generated images of a specific generative model and other images, i.e., images generated by other models and real images.

1 Introduction

In recent years, there has been a rapid evolution in image generation techniques. With the advances in visual generative models, images can now be easily created with high quality and diversity [14]. There are three important milestones in the field of image generation and manipulation, i.e., Generative Adversarial Networks (GAN) [5], Variational AutoEncoders (VAE) [6], and diffusion models [7]. Various image generation models are built based on these three models [8-13] to make the AI-generated images more realistic.

With its wide adoption, the security and privacy of image generation models becomes critical [14-19]. One severe and important issue is the potential misuse and intellectual property (IP) infringement of image generation models [16, 18]. Users may generate malicious images containing inappropriate or biased content using these models and distribute them online. Furthermore, trained models may be used without authorization, violating the model owner's intellectual property. For example, malicious users may steal the model's parameters file and use it for commercial purposes. Others may create AI-generated images and falsely claim them as their own artwork (e.g., photos and paintings) to

gain recognition, which also violates the model's IP. Therefore, it is essential to track the origin of AI-generated images. The origin attribution problem is to identify whether a specific image is generated by a particular model. As shown in Fig. 1, assuming we have a model M 1 and its generated images, the origin attribution algorithm's objective is to flag an image as belonging to model M 1 if it was generated by that model. On the other hand, the algorithm should consider the image as non-belonging if it was created by other models (e.g., M 2 in Fig. 1) or if it is a real image.

One existing way to infer the source of specific images is image watermarking. It works by embedding ownership information in carrier images to verify the owner's identity and authenticity [2023]. The image watermarking-based method requires an additional modification to the generation results in a posthoc manner, which may impair the generation quality. Also, it might not necessarily reflect the use of a particular model in the process of generating an image, which can reduce its credibility as evidence in lawsuits. Furthermore, the watermark can also be stolen, and malicious users can engage in criminal activities and disguise their identities us-

of Model  M

of Model  M

Model A

Model B

Belonging

Non-Belonging

Non-Belonging

ing the stolen watermark. Another approach to identifying the source models of the generated samples [24-27] is injecting fingerprinting into the models (e.g., modifying the model architecture) and training a supervised classifier to detect the fingerprints presented in the image. While their goal is similar to ours, these methods have several limitations. Firstly, they require extra operations during the model training phase, and they cannot be applied on pre-trained models without additional operations, such as modifying the model architecture [26, 27]. Secondly, since these methods modify the training or inference process of generative models, the model's generation performance may be affected. Thirdly, these previous studies mainly focus on a particular kind of generative model, i.e., GAN [5]. In contrast, our goal is to develop an origin attribution approach for different types of generative models, including diffusion models [4, 3] (model-agnostic), without requiring any extra operations in the training phase and image generation phase (alteration-free). We summarize the differences between our method and existing methods in Table 1.

In this paper, we propose a method for origin attribution that is based on the input reverse-engineering task on generative models (i.e., inverting the input of a particular generative model for a specific image). The intuition behind our method is that the reverse-engineering task is easier for belonging images than non-belonging images. Therefore, we design our method

based on the differences in the reconstruction loss for the reverse-engineering between generated images of a given model and other images. The origin attribution method we propose starts by using the model to generate a set of images and calculate the reconstruction loss on these generated images. To eliminate the influence of the inherent complexities of the examined images, we also calibrate the reconstruction loss by considering the hardness of the reverse-engineering on any other model that has strong generation ability, but different architectures and training data. Afterwards, the algorithm computes the calibrated reconstruction loss for the examined image and uses statistical hypothesis testing to determine if the reconstruction loss falls within the distribution of reconstruction losses observed in the generated images. This allows us to distinguish images generated by the given model from other images in a model-agnostic and alteration-free manner. Based on our design, we implemented a prototype called RONAN ( R everse-engineering-based O rigi N A ttributio N ) in PyTorch and evaluated it on three different types of generative models (i.e., unconditional model, class-to-image model, and text-to-image model) including various GANs [1, 10, 28-30], VAEs [6], and diffusion models such as latest Consistency Model [4] and Stable Diffusion [3]. Results demonstrate our

method is effective for the 'alteration-free and model-agnostic origin attribution" task. On average, RONAN achieves 96.07% of true positive rate with a false positive rate around 5.00%.

Our contributions are summered as follows: ① We introduce a new task called 'alteration-free and model-agnostic origin attribution", which entails determining whether a specific image is generated by a particular model without requiring any additional operations during the training and generation phases. ② To accomplish this task, we analyze the differences in the reconstruction loss for reverseengineering between the generated images of a given model and other images. Based on our analysis, we design a novel method that involves conducting input reverse-engineering and checking whether the reconstruction loss of the examined sample falls within the distribution of reconstruction losses observed in the generated images. ③ We evaluate our method on various different image generation models. The results show that our method effectively distinguishes images generated by the given model from other images in a model-agnostic and alteration-free manner. Our code can be found in https://github.com/ZhentingWang/RONAN .

2 Related Work

Detection of AI-Generated Contents. Detecting AI-generated content has become extremely important with the growing concerns about the misuse of AIGC technology [31]. The detection of AI-generated content is a binary classification problem that involves distinguishing generated samples from real ones. In the CV field, existing research has found that visually imperceptible but machine-distinguishable patterns in generated images, such as noise patterns [16, 32], frequency signals [33-36] and texture representation [37] can be used as the clues of AI-generated images. Researchers also proposed methods to detect sentences generated by generative NLP models such as ChatGPT [38-41]. Although these methods achieve promising performance for distinguishing AI-generated content and real content, they cannot infer if a specific content is generated by a given generative model, which is a novel but more challenging task and is the main focus of this paper.

Tracking Origin of Generated Images. There are several ways to track the source of the generated images. Image watermarking that pastes ownership information in carrier images [20-23] can be adapted to discern whether a specific sample is from a specific source. Watermarks can take the form of specific signals within the images, such as frequency domain signals [22] or display-camera transformations [42]. However, it requires an additional modification to the generation results in a post-hoc manner, and it does not necessarily reflect whether the image was generated by a particular model when the judges use it as the evidence (different models can use the same watermark). Another way is to inject fingerprints into the model during training and train a supervised classifier on it to discern whether an image is from a fingerprinted GAN model [24-27]. For example, Yu et al. [27] modify the architecture of the convolutional filter to embed the fingerprint, and they train the generator alongside a fingerprinting classifier capable of identifying the fingerprint and its corresponding source GAN models. It requires a modified model architecture, altered training process, and an additional procedure to train a source classifier. There are several existing methods [43-45] track the origin of the generated images by exploiting the input reconstruction. However, the practicality of these approaches is limited due to their reliance on strong assumptions (see § 5.5 for more details).

3 Problem Formulation

We focus on serving as an inspector to infer if a specific sample is generated by a particular model in an alteration-free and model-agnostic manner . To the best of our knowledge, this paper is the first work focusing on this problem. To facilitate our discussion, we first define the belonging and non-belonging of the generative models.

Definition 3.1 (Belonging of Generative Models) . Given a generative model M : I ↦→ X M where I is the input space and X M is the output space. A sample x is a belonging of model M if and only if x ∈ X M . We call a sample x is a non-belonging if x / ∈ X M .

Inspector's Goal. Given a sample x and a generative model M : I ↦→ X M where I is the input space and X M is the output space of M , the inspector's goal is to infer if a given image x is a belonging of M . Formally, the goal can be written as constructing an inference algorithm B : ( M , x ) ↦→{ 0 , 1 } that receives a sample x and a model M as the input, and returns the inference result (i.e., 0 denotes belongings, and 1 denotes non-belongings). The algorithm B can distinguish not only the belongings of the given model M and that of the other models (e.g., trained on different training data, or have different model architectures), but also the belongings of M and the natural samples that are not generated by AI. The inspector also aims to achieve the following goals:

Alteration-free: The algorithm B does not require any extra operations/modifications in the training phase and image generation phase.

Model-agnostic: The algorithm B can be applied to different types of image generative models with different architectures.

Inspector's Capability. The inspector has white-box access to the provided model M , thus the inspector can get the intermediate output and calculate the gradient of the models. In addition, the inspector cannot control the development and training process of the provided models. Note that the white-box access to the inspected model is practical especially in the scenarios where the inspector is the owner of the inspected models.

Real-world Application. The inspection algorithm can be widely used in various scenarios where it is necessary to verify the authenticity of generated images. We provide three examples as follows:

Copyright protection of image generation models: A copyright is a kind of intellectual property (IP) that provides its owner the exclusive right to perform a creative work [46]. In this scenario, a party suspects that a specific image may have been generated by their generative model without authorization, such as if a malicious user has stolen the model and used it to generate images. The party can then request an inspector to use our proposed method to infer if the doubtful image was indeed generated by their particular model, and the resulting inference can be used as a reference in a lawsuit. It is important to clarify that infering an image as a belonging of a model does not imply the IP of this image is totally belonging to this model. In fact, determining the ownership of the IP related to the generated images remains an unresolved challenge in the field of law. This complexity arises due to the involvement of multiple entities (such as contributors of training data, model trainers, input/prompt providers, and the models themselves) throughout the image generation process. The infering results of our origin attribution method can serve as a valuable reference for addressing IP protection concerns, instead of a definitive conclusion.

Tracing the source of maliciously generated images: Assume a user creates malicious images containing inappropriate or biased content and distributes them on the internet. The cyber police can utilize our proposed method to infer if the image was generated by a model belonging to a specific user. The resulting inference can be used as a reference for criminal evidence in lawsuits.

Detecting AI-powered plagiarism: Our method can also be used for detecting AI-powered plagiarism. For example, imagine a scenario where an individual generates AI-created images (e.g., using Midjourney) and dishonestly presents them as their own original artwork (e.g., photographs and paintings) to gain recognition and reputation. In such cases, the model owner (e.g., Midjourney's owner) may suspect that the image is generated using their model (e.g., Midjourney). Our proposed method can then be employed to uncover such instances of AI-powered plagiarism.

4 Method

Our method is built on the input reverse-engineering for image generation models. In this section, we start by formulating the input reverse-engineering task, followed by an analysis of the disparities in reconstruction loss between images generated by a particular model and those from other sources. We then proceed to present a detailed algorithm for our method.

4.1 Reverse-engineering

Recent researches [47, 17] demonstrate that the input of the modern image generative models is reconstructable by using the generated images. Here, we view the reverse-engineering as an optimization problem. Formally, it can be defined as follows:

Definition 4.1 (Input Reverse-engineering) . Given a generative model M : I ↦→ X M , and an image x , an input reverse-engineering task is optimizing the input i to make the corresponding output image from the model M ( i ) as close as possible to the given image x .

The input reverse-engineering task is performed by a reverse-engineering algorithm A : ( M , x ) ↦→ R , which can be written as:

where L is a metric to measure the distance between different images, and i ⋆ is the reverse-engineered input. The given value for the reverse-engineering algorithm A is a specific image x and a particular model M . The returned value of the algorithm A is the distance between the given image and its

0.0000

0.0100

0.0200

0.0300

Reconstructed MSE Loss

0.00

0.10

0.20

0.30

0.40

Percentage

Generated Samples

Training Samples

(a) DCGAN [1]

0.0000

0.0050

0.0100

0.0150

Reconstructed MSE Loss

0.00

0.10

0.20

0.30

0.40

Percentage

Generated Samples

Training Samples

(b) VAE [6]

0.0000

0.0100

0.0200

0.0300

Reconstructed MSE Loss

0.00

0.10

0.20

0.30

0.40

Percentage

Generated Samples

Training Samples

reverse-engineered version, i.e., L ( M ( i ⋆ ) , x ) , which is called as the reconstruction loss . We use the reconstruction loss to measure the hardness of the input reverse-engineering task.

Based on the definitions and formulations, we have the following theorem:

Theorem 4.2. Given a generative model M : I ↦→ X M , and a reverse-engineering algorithm A , if the model is deterministic (i.e., it produces the same output given the same input) and the reverse-engineering algorithm is perfect (i.e., it can find the global minimum of the reconstruction loss for the reverse-engineering), then for any x ∈ X M (belonging) and x ′ / ∈ X M (non-belonging) we have A ( M , x ′ ) > A ( M , x ) .

The proof for Theorem 4.2 can be found in the Appendix A. The theorem demonstrates that the reconstruction loss of the images generated by a specific model will be lower than that of images that are not generated by the model. The theorem also establishes that the distribution of reconstruction loss values for belonging and non-belonging images is perfectly separable. Thus, we can use a threshold value to separate the belonging images and non-belonging images. In the real world, many image generation models incorporate random noises into their image generation procedures to enhance the variety of images they produce. However, these models can also be deemed deterministic since we can regard all the random noises utilized in the generation procedure as parts of the inputs. On the other hand, in reality, the reverse-engineering algorithm may get stuck at a local minimum, and it is hard to guarantee the achievement of the global minimum. This is where the formula P ( A ( M , x ′ ) > A ( M , x )) ≥ λ becomes relevant as it serves as a relaxation for Theorem 4.2, explaining the practical scenario. In this formula, λ (e.g., 90% ) acts as a separability level for distinguishing between the two distributions: belonging images and non-belonging images.

To investigate the practical scenario, we conduct experiments on the CIFAR-10 [48] dataset using DCGAN [1], VAE [6], and StyleGAN2-ADA [10] models. The results are depicted in Fig. 2, where the x-axis represents the reconstruction loss measured by the MSE (Mean Squared Error) [49] metric, and the y-axis indicates the percentage of images whose reconstruction loss value corresponds to the corresponding value on the x-axis. We use blue color to denote 100 generated images of the given model and orange to represent 100 real images randomly sampled from the training data of the model. The results indicate that the reconstruction losses of the generated images (belongings) and those not generated by this model (non-belongings) can be distinguished.

4.2 Calibration

Different images have different inherent complexities [50-52]. Some images may be harder to reverse-engineer due to their higher complexity (e.g., containing more objects, colors, and details). In that case, the reconstruction loss will also be influenced by the inherent complexities of the examined images. To increase the separability level of belonging images and others, we disentangle the influence of the inherent complexities and the belonging status by measuring the complexity of the image and use it to calibrate the reconstruction loss. By default, we use the reconstruction loss on a reference image generation model M r that is trained on a different dataset as the measurement of the complexity, i.e., Complexity ( x ) = A ( M r , x ) (we use the Consistency model [4] pre-trained on ImageNet dataset [53] as the reference model by default). We also discuss other implements to measure the complexity of the inspected images in Appendix F. The calibrated reconstruction loss A ′ ( M , x ) is defined as follows:

4.3 Belonging Inference via Hypothesis Testing

We use Grubbs' Hypothesis Testing [54] to infer if a specific sample x is a belonging of the particular given model M . We have a null hypothesis H 0 : x is a non-belonging of M , and the alternative hypothesis H 1 : x is a belonging of M . The null hypothesis H 0 is rejected (i.e., the alternative hypothesis H 1 is accepted) if the following inequality (Eq. 3) holds:

Here, µ and σ are the mean value and standard deviation for the calibrated reconstruction loss on belonging samples of model M . Since model M is given to the inspector, the inspector can calculate µ and σ by using M to generate multiple images with randomly sampled inputs. N is the number of generated belonging images. A ′ ( M , x ) is the calibrated reconstruction loss of the examined image x . t α/N,N -2 is the critical value of the t distribution with N -2 degrees of freedom and a significance level of α/N , where α is the significance level of the hypothesis testing (i.e., 0.05 by default in this paper). The critical value of the t distribution (i.e., t α/N,N -2 ) can be computed using the cumulative distribution function (See Appendix B for more details).

4.4 Algorithm

We propose Algorithm 1 to determine if a specific sample belongs to a given model. The input of Algorithm 1 is the examined data x and the given model M . The output of this algorithm is the inference results, i.e., belonging or non-belonging. In line 3, we use the given model to generate N (i.e., 100 by default in this paper) images with randomly sampled inputs and calculate the mean value ( µ ) and standard deviation ( σ ) for the calibrated reconstruction loss on the generated belonging samples. This step can be done offline, i.e., it only needs to be performed once for each model. In line 5, we calculate the calibrated reconstruction loss of the examined image (Eq. 2), the reconstruction loss is computed via gradient descent optimizer (Adam [55] by default in this paper). In line 7, we determine if the examined image x belongs to the model M or not by conducting the Grubbs' Hypothesis Testing [54] (§ 4.3). The given image is flagged as a belonging of the given model if the corresponding hypothesis is accepted.

5 Experiments and Results

In this section, we first introduce the setup of the experiments (§ 5.1). We evaluate the effectiveness of RONAN (§ 5.2) and provide a case study on Stable Diffusion v2 model [3] (§ 5.3). We then conduct ablation studies in § 5.4, and discuss the comparison to existing reconstruction based attribution methods in § 5.5. The discussion about the efficiency and robustness against image editing can be found in the Appendix.

5.1 Setup

Our method is implemented with Python 3.8 and PyTorch 1.11. We conducted all experiments on a Ubuntu 20.04 server equipped with six Quadro RTX 6000 GPUs.

Models. Eleven different models are included in the experiments: DCGAN [1], VAE [6], StyleGAN2ADA [10], DDIM [2], DDPM [7], TransGAN [56], StyleGAN XL [30], Consistency Diffusion

Model [4], Control-GAN [28], StackGAN-v2 [29], and Stable Diffusion v2 [3]. These models are representative image generation models.

Performance Metrics. The effectiveness of the method is measured by collecting the detection accuracy (Acc). For a particular model, given a set of belonging images and non-belonging images, the Acc is the ratio between the correctly classified images and all images. We also show a detailed number of True Positives (TP, i.e., correctly detected belongings), False Positives (FP, i.e., non-belongings classified as belongings), False Negatives (FN, i.e., belongings classified as non-belongings) and True Negatives (TN, i.e., correctly classified non-belongings).

5.2 Effectiveness

In this section, we evaluate the effectiveness of RONAN from two perspectives: (1) its effectiveness in distinguishing between belongings of a particular model and real images; (2) its effectiveness in differentiating between belongings of a particular model and those generated by other models.

Distinguishing Belonging Images and Real Images. To investigate RONAN's effectiveness in distinguishing between belonging images of a particular model and real images, given an image generation model, we start by differentiating between the generated images of the given model and the training data of the model. The investigated models include DCGAN [1], VAE [6], StyleGAN2ADA [10] trained on the CIFAR-10 [48] dataset, Consistency Model [4] trained on the ImageNet [53] dataset, and ControlGAN [28] trained on the CUB-200-2011 [57] dataset. Among them, DCGAN and VAE are unconditional image generation models. StyleGAN2-ADA and the latest diffusion model Consistency Model, are class-conditional models. In addition to distinguishing belongings and the training data, we also conduct experiments to distinguish belongings from unseen data that has a similar distribution to the training data (i.e., the test data of the dataset). For each case, we evaluate the results on 100 randomly sampled belonging images and 100 randomly sampled non-belonging images. The results are demonstrated in Table 2. As can be observed, the detection accuracies (Acc) are above 85% in all cases. On average, the Acc is 94.2% for distinguishing belongings and training data, and is 95.9% for distinguishing belongings and unseen data. The results demonstrate that

RONAN achieves good performance in distinguishing between belonging images of a particular model and real images.

Distinguishing Belonging Images and Images Generated by Other Models. In this section, we study RONAN's effectiveness to distinguish between belonging images of a particular model and the images generated by other models. For a given model M 1 , we consider two different types of other models M 2 , i.e., the model trained on the same dataset but with different architectures and the model that has the same architecture but is trained on a different dataset.

Models with Different Architectures. We first evaluate RONAN's effectiveness in distinguishing between belonging images of a particular model and the images generated by other models with the same training data but different architectures. For the models trained on the CIFAR-10 [48] dataset, the used model architectures are DCGAN [1], VAE [6], StyleGAN2-ADA [10], DDIM [2], DDPM [7], TransGAN [56]. For the Imagenet [53] dataset, the involved models are the latest diffusion model Consistency Model [4] and StyleGAN XL [30]. For the CUB-200-2011 [57] dataset, we use text-to-image models ControlGAN [28] and StackGAN-v2 [29]. To measure the effectiveness of RONAN, we collect its results on 100 randomly sampled belonging images and 100 randomly sampled non-belonging images. The results are shown in Table 3, where Model M 1 denotes the examined model, and Model M 2 represents the other model that has the same training data but a different architecture. The results show that the average detection accuracy (Acc) of RONAN is 96.45%, confirming its good performance for distinguishing between belongings of a given model and the images generated by other models with different architectures.

Models Trained on Different Datasets. We also evaluate RONAN's effectiveness in distinguishing between belonging images of a particular model and the images generated by other models with the same model architecture but trained on different datasets. The model used here is the diffusion model Consistency Model [4]. We use a model trained on the ImageNet [53] dataset and a model trained on the LSUN [58] dataset. The results are demonstrated in Table 4, where Model M 1 denotes the examined model, and Model M 2 means the other model that has the same model architecture but is trained on a different dataset. The results show that RONAN can effectively distinguish between belonging images of a particular model and the images generated by other models with the same model architecture but different training data. On average, the detection accuracy of our method is 93.0%. In the Appendix, we also discuss the results when the training data of the

Table 4: Results for distinguishing belonging images and images generated by other models with different training data. Here, Model M 1 is the examined model, Model M 2 is the other model that has same architecture but different training data.

model M 2 and that of the model M 1 have overlaps. Empirical results demonstrate that RONAN is still effective even the training data of the examined model and that of the other model are similar (i.e., they have large overlaps).

5.3 Case Study on Stable Diffusion v2

In this section, we conduct a case study on the recent Stable Diffusion v2 [3] model. We first randomly collect 20 images of Shiba dogs from the internet and use these images as the nonbelonging images. We then use the prompt "A cute Shiba on the grass." and feed it into the Stable Diffusion v2 model to generate 20 belonging images. We apply RONAN on the model, and evaluate its performance in distinguishing the belonging images and non-belonging images. The results show that the detection accuracy of RONAN is 87.5%, with 18 TP, 3 FP, 2 FN, 17 TN. In Fig. 3, we show the visualization of a belonging image and a non-belonging image, as well as their corresponding reverse-engineered images. Note that the non-belonging image and the belonging image have similar inherent complexities (i.e., their reconstructed loss with the MSE metric on the reference model are 0.029 and 0.022, respectively). For the non-belonging image, the reverse-engineered image is more noisy and blurred, while the reverse-engineered image of the belonging image seems nearly identical to the original image. These results show the potential to apply our method on state-of-the-art models such as Stable Diffusion v2. More visualizations and examples can be found in Appendix J.

5.4 Ablation Study

In this section, we evaluate the impact of different metrics used in calculating reconstruction loss, and the impact of reconstruction loss calibration.

(a) A non-belonging image and its reverseengineered image for Stable Diffusion v2. The reconstructed loss is 0.0078 with MSE metric.

Fig. 3: Visualization of the belonging image and non-belonging image for Stable Diffusion v2 [3], and their corresponding reverse-engineered images. The non-belonging image and the belonging image have similar inherent complexities.

task (Eq. 1), we use a metric L to meaBy default, we select MSE [49] as the metric.

Table 5: Results on different distance metrics.

Different Metrics. In the reverse-engineering sure the distance between different images. In addition to MSE, we also evaluate the results on other image distance metrics, i.e., MAE (Mean Absolute Error) [59], SSIM (Structural Similarity Index Measure) [60], and LPIPS (Learned Perceptual Image Patch Similarity) [61]. The task is to distinguish between belonging images and real images (i.e., training images of the model here). The model used in this section is the StyleGAN2-ADA [10] trained on

CIFAR-10 [48] dataset. The results are shown in Table 5. As we can observe, the detection accuracy (Acc) under MAE, MSE, SSIM, and LPIPS are 94.5%, 97.0%, 85.5%, and 96.0%, respectively. Overall, the MSE metric achieves the highest performance in distinguishing belonging images and real images. Thus, we select MSE as our default metric.

Impacts of Reconstruction Loss Calibration. To eliminate the influence of images' inherent complexities, we calibrate the reconstruction loss by considering the hardness of the reverse-engineering on a reference model (§ 4.2). To measure the effects of the calibration step, we compare the detailed TP, FP, FN, TN, and Acc for the method with and without the calibration step. We use the Stable Diffusion v2 [3] model and follow the experiment settings

Table 6: Effects of reconstruction loss calibration.

described in § 5.3. The results in Table 6 demonstrate the detection accuracy for the method with and without the calibration step are 75.0% and 87.5%, respectively. These results show that the calibration step can effectively improve the performance of RONAN.

5.5 Comparison to Existing Reconstruction based Attribution Methods

Albright et al. [43] and Zhang et al. [44] are two existing attribution methods that also apply the reconstruction on the input. In this section, we compare our method to them. Given an inspected image, Albright et al. and Zhang et al. works by enumerting and conducting inversion on all models within a set of suspicious candidate models (referred to as the "candidate set" in this section), and attribute the model with the lowest reconstruction loss as the source of the image. Their methods rely on the assumptions that the inspector can have the white-box access to all models in the candidate set, and the examined image must be generated by one of the models in the candidate set. These assumptions diminish the practicability of their methods, whereas our method does not have such requirements. Thus, our threat model and experiment settings are fundamentally different to theirs. Our paper focus on the problem of determining if a given image is generated by a single given model or not. Consider a scenario where a model owner wants to verify if an image is generated by a model owned by him/her. While our method only needs to conduct the inversion on this specific model. Albright et al. and Zhang et al. need to compare the reconstruction loss of this particular model with a large number of suspicious models. There are several cases where Albright et al. and Zhang et al. are ineffective in addressing this problem: ① In cases where the inspector lacks white-box access

to some of the suspicious models, deriving reconstruction on them and getting inference results becomes infeasible. Notably, more and more state-of-the-art image generation models (e.g., such as Midjourney [62] and DALL-E2 [13]) are close-sourced and they only provide the black-box API to the users. ② Albright et al. and Zhang et al. are prone to making wrong predictions if the real source model is not included in the candidate set. This is attributed to their underlying assumption that the examined image must originate from one of the models within the candidate set. Ensuring the real source model is included within the candidate set is a very hard problem in practice. ③ Equally noteworthy, Albright et al. and Zhang et al. do not work when the inspected images are real images due to their strong assumption (i.e., the examined image must be generated by one of the model in the candidate set). Our method does not have the above problems.

Despite the threat models are different, we also conduct the comparison experiments. We consider the setting for distinguishing the belonging images of the inspected model M 1 and the generated images of other models M 2 , and the inspected model here is Stable Diffusion v2. For

our method, we assume the inspector can only access M 1 . For Albright et al. and Zhang et al., we assume the inspector can access both M 1 and M 2 . The results can be found in Table 7. While the average detection accuracy of Albright et al. and Zhang et al. are only 50%, our method achieves over 90% detection accuracy, meaning that our method outperforms Albright et al. and Zhang et al.

Laszkiewicz et al. [45] is a concurrent work focusing on the single-model attribution problem. Although the method proposed in Laszkiewicz et al. achieves promising performance, it is important to note that it is not model-agnostic. More specifically, it assumes the last layer of the inspected model is invertible. Therefore, it is not applicable to the models that use non-invertible activation functions (such as the commonly-used ReLU function) in the last layer. It also has assumptions on the architectures of the models, making it can not be used for many modern diffusion models [7, 63].

6 Discussion

Limitations. While our method can achieve origin attribution in a alteration-free and model-agnostic manner, the computation cost might be higher than that of watermark-based methods [20-23] and classifier-based methods [24-27]. More discussion about the efficiency of our method can be found in the Appendix. Speeding up the reverse-engineering will be our future work. This paper focuses on the image generation models. Expanding our method for origin attribution on the model in other fields, (e.g., vedio generation models such as Imagen Video [64], language generation models like ChatGPT [65], and graph generation models [66, 67]) will also be our future work.

Ethics. Research on security and privacy of machine learning potentially has ethical concerns [68-78]. This paper proposes a method to safeguard the intellectual property of image generation models and monitor their potential misuse. We believe that our approach will enhance the security of image generation models and be beneficial to generative AI.

7 Conclusion

In this paper, we take the first effort to introduce the 'alteration-free and model-agnostic origin attribution' task for AI-generated images. Our contributions for accomplishing this task involves defining the reverse-engineering task for generative models and analyzing the disparities in reconstruction loss between generated samples of a given model and other images. Based on our analysis, we devise a novel method for this task by conducting input reverse-engineering and calculating the corresponding reconstruction loss. Experiments conducted on different generative models under various settings demonstrate that our method is effective in tracing the origin of AI-generated images.

References

Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv:1511.06434 , 2015.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In International Conference on Learning Representations , 2020.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Yang Song, Prafulla Dhariwal, Mark Chen, and Ilya Sutskever. Consistency models. arXiv preprint arXiv:2303.01469 , 2023.

Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. Generative adversarial nets. In NIPS , 2014.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems , 33:6840-6851, 2020.

Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 4401-4410, 2019.

Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8110-8119, 2020.

Tero Karras, Miika Aittala, Janne Hellsten, Samuli Laine, Jaakko Lehtinen, and Timo Aila. Training generative adversarial networks with limited data. Advances in neural information processing systems , 33:12104-12114, 2020.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Jiapeng Zhu, Yujun Shen, Deli Zhao, and Bolei Zhou. In-domain gan inversion for real image editing. In European conference on computer vision , pages 592-608. Springer, 2020.

OpenAI. DALL-E 2. https://openai.com/product/dall-e-2 , .

Jamie Hayes, Luca Melis, George Danezis, and Emiliano De Cristofaro. Logan: Membership inference attacks against generative models. arXiv preprint arXiv:1705.07663 , 2017.

Nicholas Carlini, Jamie Hayes, Milad Nasr, Matthew Jagielski, Vikash Sehwag, Florian Tramer, Borja Balle, Daphne Ippolito, and Eric Wallace. Extracting training data from diffusion models. arXiv preprint arXiv:2301.13188 , 2023.

Hanqing Zhao, Wenbo Zhou, Dongdong Chen, Tianyi Wei, Weiming Zhang, and Nenghai Yu. Multi-attentional deepfake detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2185-2194, 2021.

Dingfan Chen, Ning Yu, Yang Zhang, and Mario Fritz. Gan-leaks: A taxonomy of membership inference attacks against generative models. In Proceedings of the 2020 ACM SIGSAC conference on computer and communications security , pages 343-362, 2020.

Ding Sheng Ong, Chee Seng Chan, Kam Woh Ng, Lixin Fan, and Qiang Yang. Protecting intellectual property of generative adversarial networks from ambiguity attacks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 3630-3639, 2021.

Zhenting Wang, Chen Chen, Yuchen Liu, Lingjuan Lyu, Dimitris Metaxas, and Shiqing Ma. How to detect unauthorized data usages in text-to-image diffusion models. arXiv preprint arXiv:2307.03108 , 2023.

Mitchell D Swanson, Bin Zhu, and Ahmed H Tewfik. Transparent robust image watermarking. In Proceedings of 3rd IEEE International Conference on Image Processing , volume 3, pages 211-214. IEEE, 1996.

Lixin Luo, Zhenyong Chen, Ming Chen, Xiao Zeng, and Zhang Xiong. Reversible image watermarking using interpolation technique. IEEE Transactions on information forensics and security , 5(1):187-193, 2009.

Shelby Pereira and Thierry Pun. Robust template matching for affine resistant image watermarks. IEEE transactions on image Processing , 9(6):1123-1129, 2000.

Matthew Tancik, Ben Mildenhall, and Ren Ng. Stegastamp: Invisible hyperlinks in physical photographs. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 2117-2126, 2020.

Ning Yu, Larry S Davis, and Mario Fritz. Attributing fake images to gans: Learning and analyzing gan fingerprints. In Proceedings of the IEEE/CVF international conference on computer vision , pages 7556-7566, 2019.

Yuzhen Ding, Nupur Thakur, and Baoxin Li. Does a gan leave distinct model-specific fingerprints. In Proceedings of the BMVC , 2021.

Ning Yu, Vladislav Skripniuk, Sahar Abdelnabi, and Mario Fritz. Artificial fingerprinting for generative models: Rooting deepfake attribution in training data. In Proceedings of the IEEE/CVF International conference on computer vision , pages 14448-14457, 2021.

Ning Yu, Vladislav Skripniuk, Dingfan Chen, Larry S Davis, and Mario Fritz. Responsible disclosure of generative models using scalable fingerprinting. In International Conference on Learning Representations , 2022.

Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Philip Torr. Controllable text-to-image generation. Advances in Neural Information Processing Systems , 32, 2019.

Han Zhang, Tao Xu, Hongsheng Li, Shaoting Zhang, Xiaogang Wang, Xiaolei Huang, and Dimitris N Metaxas. Stackgan++: Realistic image synthesis with stacked generative adversarial networks. IEEE transactions on pattern analysis and machine intelligence , 41(8):1947-1962, 2018.

Axel Sauer, Katja Schwarz, and Andreas Geiger. Stylegan-xl: Scaling stylegan to large diverse datasets. In ACM SIGGRAPH 2022 conference proceedings , pages 1-10, 2022.

Chen Chen, Jie Fu, and Lingjuan Lyu. A pathway towards responsible ai generated content. arXiv preprint arXiv:2303.01325 , 2023.

Francesco Marra, Diego Gragnaniello, Luisa Verdoliva, and Giovanni Poggi. Do gans leave artificial fingerprints? In 2019 IEEE conference on multimedia information processing and retrieval (MIPR) , pages 506-511. IEEE, 2019.

Joel Frank, Thorsten Eisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In International conference on machine learning , pages 3247-3258. PMLR, 2020.

Ricard Durall, Margret Keuper, Franz-Josef Pfreundt, and Janis Keuper. Unmasking deepfakes with simple features. arXiv preprint arXiv:1911.00686 , 2019.

Ricard Durall, Margret Keuper, and Janis Keuper. Watch your up-convolution: Cnn based generative deep neural networks are failing to reproduce spectral distributions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 7890-7899, 2020.

Yonghyun Jeong, Doyeon Kim, Youngmin Ro, and Jongwon Choi. Frepgan: robust deepfake detection using frequency-level perturbations. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 1060-1068, 2022.

Zhengzhe Liu, Xiaojuan Qi, and Philip HS Torr. Global texture enhancement for fake face detection in the wild. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8060-8069, 2020.

Xinlei He, Xinyue Shen, Zeyuan Chen, Michael Backes, and Yang Zhang. Mgtbench: Benchmarking machine-generated text detection. arXiv preprint arXiv:2303.14822 , 2023.

OpenAI. New AI classifier for indicating AI-written text. https://openai.com/blog/ new-ai-classifier-for-indicating-ai-written-text , .

Ganesh Jawahar, Muhammad Abdul-Mageed, and Laks VS Lakshmanan. Automatic detection of machine generated text: A critical survey. arXiv preprint arXiv:2011.01314 , 2020.

Matthias Gallé, Jos Rozen, Germán Kruszewski, and Hady Elsahar. Unsupervised and distributional detection of machine-generated text. arXiv preprint arXiv:2111.02878 , 2021.

Han Fang, Weiming Zhang, Hang Zhou, Hao Cui, and Nenghai Yu. Screen-shooting resilient watermarking. IEEE Transactions on Information Forensics and Security , 14(6):1403-1418, 2018.

Michael Albright and Scott McCloskey. Source generator attribution via inversion. In CVPR Workshops , volume 8, page 3, 2019.

Baiwu Zhang, Jin Peng Zhou, Ilia Shumailov, and Nicolas Papernot. On attribution of deepfakes. arXiv preprint arXiv:2008.09194 , 2020.

Mike Laszkiewicz, Jonas Ricker, Johannes Lederer, and Asja Fischer. Single-model attribution via final-layer inversion. arXiv preprint arXiv:2306.06210 , 2023.

Michael Spence. Intellectual property . OUP Oxford, 2007.

Benjamin Hilprecht, Martin Härterich, and Daniel Bernau. Monte carlo and reconstruction membership inference attacks against generative models. Proc. Priv. Enhancing Technol. , 2019 (4):232-249, 2019.

Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.

Zhou Wang and Alan C Bovik. Mean squared error: Love it or leave it? a new look at signal fidelity measures. IEEE signal processing magazine , 26(1):98-117, 2009.

Christoph Redies, Seyed Ali Amirshahi, Michael Koch, and Joachim Denzler. Phog-derived aesthetic measures applied to color photographs of artworks, natural scenes and objects. In Computer Vision-ECCV 2012. Workshops and Demonstrations: Florence, Italy, October 7-13, 2012, Proceedings, Part I 12 , pages 522-531. Springer, 2012.

Honghai Yu and Stefan Winkler. Image complexity and spatial information. In 2013 Fifth International Workshop on Quality of Multimedia Experience (QoMEX) , pages 12-17. IEEE, 2013.

I Mario, M Chacon, D Alma, and S Corral. Image complexity measure: a human criterion free approach. In NAFIPS 2005-2005 Annual Meeting of the North American Fuzzy Information Processing Society , pages 241-246. IEEE, 2005.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International journal of computer vision , 115(3):211-252, 2015.

Frank E Grubbs. Sample criteria for testing outlying observations. The Annals of Mathematical Statistics , pages 27-58, 1950.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Yifan Jiang, Shiyu Chang, and Zhangyang Wang. Transgan: Two pure transformers can make one strong gan, and that can scale up. Advances in Neural Information Processing Systems , 34: 14745-14758, 2021.

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. Caltech-ucsd birds-200-2011 (cub-200-2011). Technical Report CNS-TR-2011-001, California Institute of Technology, 2011.

Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of a large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:1506.03365 , 2015.

Tianfeng Chai and Roland R Draxler. Root mean square error (rmse) or mean absolute error (mae). Geoscientific model development discussions , 7(1):1525-1534, 2014.

Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing , 13(4): 600-612, 2004.

Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 586-595, 2018.

Midjourney. Midjourney. https://www.midjourney.com/ .

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion models. arXiv preprint arXiv:2210.02303 , 2022.

OpenAI. ChatGPT. https://openai.com/blog/chatgpt/ , .

Minkai Xu, Lantao Yu, Yang Song, Chence Shi, Stefano Ermon, and Jian Tang. Geodiff: A geometric diffusion model for molecular conformation generation. arXiv preprint arXiv:2203.02923 , 2022.

Clement Vignac, Igor Krawczuk, Antoine Siraudin, Bohan Wang, Volkan Cevher, and Pascal Frossard. Digress: Discrete denoising diffusion for graph generation. arXiv preprint arXiv:2209.14734 , 2022.

Florian Tramèr, Fan Zhang, Ari Juels, Michael K Reiter, and Thomas Ristenpart. Stealing machine learning models via prediction apis. In USENIX security symposium , volume 16, pages 601-618, 2016.

Zhenting Wang, Kai Mei, Juan Zhai, and Shiqing Ma. Unicorn: A unified backdoor trigger inversion framework. The Eleventh International Conference on Learning Representations , 2023.

Yingqi Liu, Guangyu Shen, Guanhong Tao, Zhenting Wang, Shiqing Ma, and Xiangyu Zhang. Complex backdoor detection by symmetric feature differencing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15003-15013, 2022.

Guanhong Tao, Zhenting Wang, Shiwei Feng, Guangyu Shen, Shiqing Ma, and Xiangyu Zhang. Distribution preserving backdoor attack in self-supervised learning. In 2024 IEEE Symposium on Security and Privacy (SP) , pages 29-29. IEEE Computer Society, 2023.

Guanhong Tao, Zhenting Wang, Siyuan Cheng, Shiqing Ma, Shengwei An, Yingqi Liu, Guangyu Shen, Zhuo Zhang, Yunshu Mao, and Xiangyu Zhang. Backdoor vulnerabilities in normally trained deep learning models. arXiv preprint arXiv:2211.15929 , 2022.

Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP) , pages 3-18. IEEE, 2017.

Zhenting Wang, Hailun Ding, Juan Zhai, and Shiqing Ma. Training with more confidence: Mitigating injected and natural backdoors during training. Advances in Neural Information Processing Systems , 35:36396-36410, 2022.

Minhao Cheng, Thong Le, Pin-Yu Chen, Jinfeng Yi, Huan Zhang, and Cho-Jui Hsieh. Query-efficient hard-label black-box attack: An optimization-based approach. arXiv preprint arXiv:1807.04457 , 2018.

Zhenting Wang, Kai Mei, Hailun Ding, Juan Zhai, and Shiqing Ma. Rethinking the reverseengineering of trojan triggers. Advances in Neural Information Processing Systems , 2022.

Zhenting Wang, Juan Zhai, and Shiqing Ma. Bppattack: Stealthy and efficient trojan attacks against deep neural networks via image quantization and contrastive adversarial learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 15074-15084, 2022.

Kai Mei, Zheng Li, Zhenting Wang, Yang Zhang, and Shiqing Ma. Notable: Transferable backdoor attacks against prompt-based nlp models. Annual Meeting of the Association for Computational Linguistics , 2023.

Yann LeCun, Corinna Cortes, and CJ Burges. Mnist handwritten digit database. ATT Labs [Online]. Available: http://yann.lecun.com/exdb/mnist , 2, 2010.

Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Cnngenerated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 8695-8704, 2020.

Riccardo Corvi, Davide Cozzolino, Giada Zingarini, Giovanni Poggi, Koki Nagano, and Luisa Verdoliva. On the detection of synthetic images generated by diffusion models. In ICASSP 20232023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) , pages 1-5. IEEE, 2023.

Gaurav Parmar, Krishna Kumar Singh, Richard Zhang, Yijun Li, Jingwan Lu, and Jun-Yan Zhu. Zero-shot image-to-image translation. In ACM SIGGRAPH 2023 Conference Proceedings , pages 1-11, 2023.

Robert B Ash. Information theory . Courier Corporation, 2012.

Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740 , 2017.

Roadmap: In this appendix, we show the proof for Theorem 4.2 (Appendix A), detailed process for calculating critical value of the t distribution used in § 4.3 (Appendix B), more results for distinguishing belonging images and images generated by other models (Appendix C), a thought experiment on VAE (Appendix D), results on different reverse-engineering methods (Appendix E), discussion on different methods for measuring the complexity of the images (Appendix F). RONAN's robustness to editing-based adaptive attack (Appendix G), the discussion for the efficiency of RONAN (Appendix H). Finally, we provide more results and demonstrate more visualizations on Stable Diffusion v2 [3] (Appendix I andAppendix J).

A Proof for Theorem 4.2

We start our analysis from ideal generative model and reconstruction algorithm, which we define as deterministic generative model and perfect reconstruction algorithm:

Definition A.1 (Deterministic Generative Model) . Given a generative model M : I ↦→ X M , it is deterministic if it always produce the same output x ∈ X M given the same input i ∈ I .

Definition A.2 (Perfect Reverse-engineering Algorithm) . Given a reverse-engineering algorithm A , if it is guaranteed that the returned reconstruction loss l is the global minima, then we say A is a perfect reverse-engineering algorithm.

Proof. Assume the given output sample x is generated by input i . Since the given model M is deterministic, we have:

In this case, the distance between the x and M ( i ) is 0, i.e., L ( M ( i ) , x ) = 0 . Based on Theorem 4.1, as A is a perfect reverse-engineering algorithm, it can find the input that can produce the minimal reconstruction loss. Therefore, we have:

Similarly, for sample x ′ / ∈ X M . There does not exist an input i ′ that can produce x ′ , meaning that there does not exist an input i ′ that have L ( M ( i ′ ) , x ′ ) = 0 . Thus, we have:

Finally, we have for any x ∈ X M and x ′ / ∈ X M we have A ( M , x ′ ) > A ( M , x

)

.

B Computing Critical Value of the t Distribution.

In § 4.3, we use the critical value of the t distribution to obtain the results of the hypothesis testing. In this section, we discuss the detailed process for calculating the critical value. For the t-distribution, we have the probability density function:

In Eq. 7, ν is the number of degrees of freedom and Γ is the gamma function. Based on Eq. 7, we have the cumulative distribution function:

where β denotes the incomplete beta function. Therefore, given a confidence level α and the number of degrees of freedom ν , we have can use Eq. 9 to obtain the value of the critical value t α,ν .

C More Results for Distinguishing Belonging Images and Images Generated by Other Models.

In this section, we provide more results for distinguishing belonging images of the model and the images generated by other models. We first consider distinguishing the belonging images and the images generated by the other models having the same architecture and overlapped training data. Given a model M 1 , we focus on the other model M 2 which shares the same architecture as M 1 and has training data that overlaps with M 2 's training data.

The model architecture used here is DCGAN [1]. We trained M 1 on the full CIFAR-10 [48] dataset, while M 2 is trained on the randomly sampled subsets of the CIFAR-10 dataset. The results are presented in Table 8, where the first column indicates the proportion of overlap between the training data of M 1 and M 2 . The second column of Table 8 displays the accuracy of RONAN in detecting the differences. Notably, even when 90% of M 2 's training data overlaps with that of M 1 , the detection accuracy remains above 95%. These results demonstrate that our method is still effective

when the training data of the inspected model and that of the other model are similar. We also consider distinguishing the belonging images and the images generated by the other models having the same training data and similar architectures. We use DCGAN [1] here and modifies the number of layers and the number of channels in the first Conv layer in the models to make M 1 and M 2 have similar architectures. Both M 1 and M 2 here are trained on CIFAR-10 [48] dataset. The results can be found in Table 9 and Table 10. Based on these results, our method is effective even the inspected model and other models have the same training dataset and similar architectures. We also provide a summary of the combinatorial setups used for distinguishing belonging images and images generated by other models in Table 11.

D A Thought Experiment on VAE.

If a image generation model perfectly fits the training samples, then the images generated by it will be indistinguishable to its training samples (which are real images). In this section, we investigate a question: 'Can image generation models perfectly fit the training samples?'. To study this question, we conduct a thought experiment on VAE [6]. In detail, we randomly sample 10 images from MNIST [79] dataset, and use VAE [6] with different numbers of neurons in the hidden layer to fit these 10 images. We set the epoch number to 5000 to ensure the training losses that measures the distance between the generated samples and the training samples of the model are converged. The detailed final training losses with different model sizes are shown in Table 12. There are two parts in the total training loss L total: the reconstruction part L reconstruction and the KL-divergence part L divergence , i.e., L total = L reconstruction + L divergence. In Table 12, Detailed values for each part are demonstrated.

With the increases in the model sizes, the final total training losses reduce at first. However, when the model is large enough, the final total training loss becomes stable at the region from 53.00-54.00. The results show that even the V AEs with enough large sizes can not fit all pixels perfectly. We also conduct the reverse-engineering of the training samples on the trained V AE with 1000 neurons in the hidden layer. The results show that we can not reconstruct the exact training samples.

As pointed out by existing works [33, 80, 81], morden generative models (e.g., VAEs, GANs, and Diffusion models) including the state-of-the-art models such as Stable Diffusion [3] and DALLE [13] will leave forensics traces in the frequency spaces of the generated images. This is caused by indispensable operations used in modern generative models (e.g., Upsampling operations and Convolutional Layers). Also, perfectly fitting the training samples means finding the global optimum in the optimization process. However, the essential gradient descent optimizers used in modern generative models (e.g., SGD and Adam) typically can not get the global optimum. Based on the above analysis and empirical results, at least we can conclude that the probabilities that the real images belong to the output space of generative models are very low.

E Discussion on Different Reverse-engineering Methods

In this section, we discuss the details of the reverse-engineering process on different models and the effects of different reverse-engineering methods. The details of the reverse-engineering for different

Inversion Method

Default

DDIM Inversion

Acc

87.5%

55.5%

62.5%

models are summarized in Table 13. We also investigate the effects of different reverse-engineering methods. For StyleGAN2-ADA [10] and StyleGAN-XL [30], we use the random noise space (i.e., Z space) to reconstruct the images by default. We also conducted experiments on using intermediate latent space (i.e., W + space) to reconstruct the images for distinguishing belonging images and images generated by other models (see setting for Table 3). The results on StyleGAN2-ADA with the CIFAR-10 [48] dataset and StyleGAN XL with ImageNet dataset are shown in Table 14. As can be observed, using the W + space has similar accuracy to using Z space, meaning that our method is not sensitive to the selection of the input spaces used for optimization.

We also conduct experiments for different reverse-engineering methods designed for diffusion models (i.e., DDIM inversion [2], Parmar et al. [82] and our method) on the Stable Diffusion v2 [3] model with the setting described in § 5.3. The results are shown in Table 15. DDIM inversion [2] is an inversion approach performing the reverse of the DDIM sampling. It is based on the assumption that the ODE process can be reversed in limited of small steps. It only has unsatisfying inversion performance on the conditional diffusion models (e.g., Stable Diffusion) because it will magnify the accumulated error in the inversion process (since it ignores the classifier-free guidance in the diffusion process). As can be seen in the above table, the accuracy of using DDIM inversion is only 55.5%. Parmar et al. [82] improve the DDIM inversion by using an approximated prompt as the conditional guidance in the inversion process. The approximated prompts are generated by a caption model (i.e., BLIP). This method's inversion quality is dependent on the captions used. Given a generated image of the inspected model, since the caption model can not get the accurate prompt used for generating this image, the inversion using this method is also not accurate, leading that the reconstruction losses of the belonging images and non-belonging images are not highly separable (the accuracy is 62.5%). Compared to DDIM inversion [2] and Parmar et al. [82], our method achieves higher accuracy.

F Discussion on Different Methods for Measuring the Complexity of the Images

In this section, we discuss the different methods for measuring the complexity of the images.

We first discuss the reference model based method. Here, we conducted experiments that using different models as reference models. The inspected model here is the StyleGAN2-ADA [10] model trained on CIFAR-10 [48] dataset and the setting here is identical to that used in the Table 2 (belongings vs training data). The results can be found in Table 16. As can be observed, using

Table 16: Results on using different models as reference models.

different reference models yields similar results, meaning that our method is not sensitive to the selection of the reference models. The measurement of the image complexity can also be implemented

by calculating the 2D entropy of the image [83]. Using this entropy-based method as the image complexity measurement also yields 97.0% final detection accuracy in our setting.

G Adaptive Attack

In this section, we evaluate the robustness of RONAN against the adaptive attack where the malicious user is aware of it and try to bypass the inspection of RONAN. For example, when the malicious user use a specific model to generate an image, he/she can make a slight modification on the image to bypass the inspection of the origin attribution algorithm. We consider the image editing as the adaptive attack because it can preserve most of the information in the original image while changing the image. To investigate if RONAN is robust to the image-editing-based adaptive attack, we use the _1977 instagram filter 3 and the box blur filter to conduct results. The model used here is the DCGAN [1] model trained on the CIFAR-10 [48] dataset. The results are shown in Table 17 and Table 18. For _1977 instagram filter, we can see that the detection accuracy of RONAN is still above 90% even under the adaptive attack. For box blur filter, we show the results under different box sizes of the filter. Besides the detection accuracy (Acc) of our method, we also demonstrate the Structural Similarity Index (i.e., SSIM [60]) between the original images and the edited images, which can measure the similarity between them. A higher SSIM value means the edited images are more similar to the original images. As can be observed, our method remains effective under relatively small box sizes. As the box sizes expand, however, the detection accuracy of our method diminishes. This outcome is understandable and acceptable, as it corresponds to a rapid reduction in the Structural Similarity Index (SSIM) between the edited images and their unaltered counterparts. When employing larger box sizes, it is conceivable that an adaptive attacker might find ways to elude our method's scrutiny, yet this comes at the cost of substantially compromising the quality of the edited images. Consequently, our method maintains its effectiveness even in the face of adaptive attacks that seek to maintain the quality of the edited images.

H Efficiency

In this section, we discuss the efficiency of RONAN. To study the efficiency, we measure the runtime of our method on different models (i.e., DCGAN [1], VAE [6], StyleGAN2-ADA [10], Consistency Model [4], and Stable Diffusion v2 [3]). For each model, we run 5 times on one Quadro RTX 6000 GPU and collect the average runtime and its standard deviation. The results can be found in Table 19. Our method can be accelerated by using mixed precision training [84]. Further approach for speeding up the

input reverse-engineering process will be our future work.

I More Results on Stable Diffusion

In this section, we provide more results on the Stable Diffusion. Here, we use the Stable Diffusion v2 model as the inspected model. The scenario considered here is distinguishing 1000 images generated by Stable Diffusion v2 model (the inspected model) and 1000 images generated by DeepFloyd-IF-IIL 4 model. Note that the images here are generated using the prompts randomly sampled from the

PromptHero 5 website. The results demonstrate that our method achieves 91.6% detection accuracy, meaning it is effective under this scenario.

J More Visualizations on Stable Diffusion

In this section, we provide more visualizations for our case study on Stable Diffusion v2 model [3] (§ 5.3). We show more visualizations for the belonging images and their corresponding reverseengineered images in Fig. 4. In Fig. 5, we demonstrate more visualizations for the non-belonging images of the Stable Diffusion v2 model, and also show their reverse-engineered images. The detailed reconstruction loss and the calibrated reconstruction loss are reported in Fig. 4 and Fig. 5. The distance metric used here is MSE. As we can observed, the belonging images have much lower calibrated reconstruction loss than the non-belonging images, further demonstrating the effectiveness of our approach.

Reverse-engineered

Reverse-engineered

Reverse-engineered

(c) The reconstruction loss and the calibrated reconstruction loss are 0.0005 and 0.0161, respectively.

Non-belonging Reverse-engineered (a) The reconstruction loss and the calibrated reconstruction loss are 0.0038 and 0.1727, respectively.

Reverse-engineered

Non-belonging

Reverse-engineered

(c) The reconstruction loss and the calibrated reconstruction loss are 0.0028 and 0.2800, respectively.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses the problem of distinguishing between images generated by a generative AI model or acquired by a camera (real images). The motivation comes from the concerns of AI community about the potential misuse and intellectual property (IP) infringement associated with image generation models. \n\nThe authors approach the classification problem (generated image vs real image) by first developing an alteration-free and model-agnostic origin attribution method via reverse-engineering (i.e., inverting the input of a particular model for a specific image) followed by computing the reconstruction loss of reverse-engineering to infer the origin. The authors provide the intuition behind the approach by stating that reverse-engineering task is easier for belonging images to a particular model than for non-belonging images generated by some other models\n\nSTRENGTHS:\nThe strengths of the paper lie in\n(1)\tIntroducing “alteration-free and model-agnostic origin attribution\" algorithm\n(2)\tAnalyzing the differences in the reconstruction loss for reverse engineering between the generated images of a given model and other images, and then checking whether the reconstruction loss of the examined sample falls within the distribution of reconstruction losses observed in the generated images. \n(3)\tEvaluating the method on eight different image generation models to quantify the accuracy\n\nWEAKNESSES:\nThe weaknesses of the paper lie in \n(1)\tMissing description of computational costs\n(2)\tReproducibility of the algorithm is the paper presents the algorithm 1 at very high level which makes it difficult to reproduce the algorithm (unless the code becomes available)\n(3)\tAssuming of having belonging images to a specific model (Section 4.4 refers to 100 images labeled as belonging images to a specific model) which might not always be the case in practice\n\nQUESTIONS:\nWould it be possible to create a table summarizing the combinatorial setups for Architecture = {the same, different} x Training Data = {the same, different, overlapping} given selected two models M1 and M2?\n\nLIMITATIONS:\nThe section “Discussion” describes the limitations of the approach in its higher computational cost than the two other approaches based on watermarking [19-22] and classifiers [23-26]. This opens up a question about tradeoffs.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper develops an origin attribution method to determine whether a specific image is generated by a particular model. The key idea is based on reverse-engineering generation models, and the decision is made by thresholding the reconstruction loss. The proposed method is evaluated on eight different image generation models.\n\nSTRENGTHS:\n1. This method could be applied to different types of generative models (model-agnostic) without requiring any extra operations in the training phase and image generation phase (alteration-free).\n\nWEAKNESSES:\n1. The novelty is lacking. The authors state that \"this paper is the first work focusing on this problem to infer if a specific sample is generated by a particular model in an alteration-free and model-agnostic manner\". However, as far as I know, there are several works working on the same problem listed below, and the proposed methodology is very similar to these works. I doubt whether the authors have conducted sufficient literature research.  \n[1] Source Generator Attribution via Inversion, CVPRW2019\n[2] On Attribution of Deepfakes, arXiv2020\n\n2. The proposed method could not be a \"perfect reverse-engineering algorithm\". Model inversion has been a difficult problem due to the nonlinearity of neural networks. Existing works make many efforts to improve the precision of inversion. The methods proposed in the work only follow the most simple optimization-based inversion method without innovations in technology.\n\n3.  The experiments are far from sufficient. In the real-world scenario, the number of unknown models is far greater than the known models. However, in Table 3, the author only uses a single model M2 as the negative model. The high accuracy doesn't reflect the performance in the open-world scenario. Besides, there is no comparison with existing methods. The author should at least compare with [1] and [2], which solve the same problem as this paper and use a similar method.\n\nQUESTIONS:\n1. How is the attribution efficiency? As the proposed method is optimization-based,  the attribution procedure may take more time than other methods by straightforward prediction. \n2. The authors should conduct more sufficient literature research and conduct more comparison experiments.\n\nLIMITATIONS:\nThe limitations are listed in the weaknesses above.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe method proposes a model-agnostic attribution method. Given a synthesized image as input, the goal is to attribute the correct model that generates the image. Different from prior work, the method is not restricted to a fixed set of models. The main idea of the work is to use reconstruction error -- the model that generates the image should also reconstruct the image better. On top of this, the work introduces a relative reconstruction measure to calibrate the difficulty of reconstruction of each image, along with a thresholding method based on hypothesis testing. The method is tested on multiple generative models or different training datasets, and the method shows strong performance in most cases.\n\nSTRENGTHS:\n1. The author clearly defines the inspector's goal and what the inspector can see. This is helpful in understanding the problem setup easily.\n\n2. The quantitative results look promising, indicating that the relative reconstruction loss is effective for this problem.\n\n3. The method section is well-motivated, where the authors introduce relative reconstruction loss for calibration, and hypothesis testing to find thresholds.\n\n4. Although the method is simple, it is interesting to have a model-agnostic attribution method.\n\nWEAKNESSES:\n1. Using reconstruction error is not new for membership inference [1]. Although membership inference focuses on whether an image is used for training the model or not, finding whether a synthesized image is generated by the model shares a similar spirit. It will be good to have more discussion on this.\n\n2. A type of generative model can have multiple ways to reconstruct an input. Currently, the paper does not provide detail about how reconstruction is done for each model. Please check the question sections for this point.\n\n3. Although the authors have mentioned in the limitation section already, the reconstruction-based algorithm requires a huge runtime cost, making it a less favorable option for model attribution.\n\n\n[1] Hilprecht et al. Monte Carlo and Reconstruction Membership Inference Attacks against Generative Models.\n\nQUESTIONS:\n1. How is reconstruction done on StyleGAN2-ADA and StyleGAN-XL? A common practice is to reconstruct the images by optimizing the extended intermediate latent space (W+) [1], instead of the random noise (z). Will these make a difference in the proposed task?\n\n2. There exist multiple reconstruction strategies for diffusion models as well [2, 3, 4]. It will be great to provide detail on this too.\n\n[1] Abdal et al. Image2StyleGAN++: How to Edit the Embedded Images?\n[2] Song et al. Denoising Diffusion Implicit Models.\n[3] Mokady et al. Null-text Inversion for Editing Real Images using Guided Diffusion Models.\n[4] Parmar et al. Zero-shot Image-to-Image Translation.\n\nLIMITATIONS:\nIn my opinion, the authors have addressed the limitations adequately.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new problem: given an image $x$ and an generate model $M$, predict whether this image $x$ belongs to $M$ or not. To this end, this paper proposes to do hypothesis testing based on the reconstruction error after conducting latent optimization to reconstruct the given image $x$. They hypothesize that if the image belongs to the model, the reconstruction error while be lower, if it does not belong to, the loss will be higher.\n\nSTRENGTHS:\n- This attribution problem is relative new, despite it has ambiguity with a more popular attribution problem, where given a generated image from a model, how can distribute the credit to each image in the training set.\n\n- their algorithm show high accuracy in their relevant but maybe limited setup.\n\nWEAKNESSES:\nwhile I understand the motivation of considering this problem, but I feel the formulation of this problem is problematic: \n\n- (1) The first is that, this framework will only work if the real images used to train these generative models do not belong to the output space of the generative models. If these training images belong to the output space, then they will be credited as the generated images of the given generative model, which is wrong. If you assume the training images do not belong to the output space, it's also weird, since this is how generative models, such as VAE, is optimized. If a generative model, such as VAE, perfectly fits the training data, your model will wrongly give IP to this generative model.\n\n- (2) relevantly, this framework will determine any reconstructed images as generated images. i.e., given any real image $x$, let's feed it through the vae autoencoder, and then get vae reconstruction as $x'$. Apparently $x'$ belongs to the output space of the vae model, but can we ethically classified it as images generated by the model, and give IP to the VAE? I do not think so.\n\n- (3) what if I train two models with the same architecture on the same dataset (just with different random seeds), then how can you determine which model generate which images? In this case, your method may wrongly attribute output from one model to the other model.\n\n- (4) This method sounds like very easy to get attacked, despite the authors argue that it's advantageous over wartermaking and classifier based one. One apparent question is that, if I do some photoshoping-ish editing or as simple as running a gaussian blurring over the generated images and real images, is this framework still able to distinguish which model generate which image?\n\n- (5) For line 301, you train the same model on different datasets, what if we train different models (architectures) on the same datasets?\n\n- (6) For the study 5.3, one question is that many other papers have shown that diffusion model can almost identically output some images used to train it. So how do you guarantee that there is no similar images in the training set for your generated images?\n\nThe calibration step is not very intuitive. Can you use any other model for calibration? will they be equally effective? why do you just choose this consistency model? it's not well justified.\n\nAssuming these generative models are white-box is too strong. The reality is that, many real images are generated by private models where you have no access to even weights, such as Mid-journey.\n\nMore realistic situation is also that, let's say we found one image is guilty, then how do we know which model is from? Then it's infeasible to iterate all models to see which model creates this given image.\n\nWriting-wise,\n\n- (1) the abstract is not easy to follow, the inverse engineering makes me think it's something very different that what you actually did, which was just latent optimization for reconstructing the output.\n- (2) because of this terminology, it also renders paragraph 67 not easy to understand what you actually do\n\nIsn't Theorem 4.2 trivial? I do not understand why authors put it as a theorem?\n\nQUESTIONS:\nsee above\n\nLIMITATIONS:\nsee above", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-07"}