SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhenting Wang*
zhenting.wang@rutgers.edu
ChenA.Chen@sony.com
Yi Zeng
yizeng@vt.edu
Lingjuan.Lv@sony.com
Image generation techniques have been gaining increasing attention recently, but concerns have been raised about the potential misuse and intellectual property (IP) infringement associated with image generation models. It is, therefore, necessary to analyze the origin of images by inferring if a specific image was generated by a particular model, i.e., origin attribution. Existing methods only focus on specific types of generative models and require additional procedures during the training phase or generation phase. This makes them unsuitable for pre-trained models that lack these specific operations and may impair generation quality. To address this problem, we first develop an alteration-free and model-agnostic origin attribution method via reverse-engineering on image generation models, i.e., inverting the input of a particular model for a specific image. Given a particular model, we first analyze the differences in the hardness of reverse-engineering tasks for generated samples of the given model and other images. Based on our analysis, we then propose a method that utilizes the reconstruction loss of reverse-engineering to infer the origin. Our proposed method effectively distinguishes between generated images of a specific generative model and other images, i.e., images generated by other models and real images.
In recent years, there has been a rapid evolution in image generation techniques. With the advances in visual generative models, images can now be easily created with high quality and diversity [1-4]. There are three important milestones in the field of image generation and manipulation, i.e., Generative Adversarial Networks (GAN) [5], Variational AutoEncoders (VAE) [6], and diffusion models [7]. Various image generation models are built based on these three models [8-13] to make the AI-generated images more realistic.
With its wide adoption, the security and privacy of image generation models becomes critical [14-19]. One severe and important issue is the potential misuse and intellectual property (IP) infringement of image generation models [16, 18]. Users may generate malicious images containing inappropriate or biased content using these models and distribute them online. Furthermore, trained models may be used without authorization, violating the model owner's intellectual property. For example, malicious users may steal the model's parameters file and use it for commercial purposes. Others may create AI-generated images and falsely claim them as their own artwork (e.g., photos and paintings) to
*Work partially done during Zhenting Wang's internship at Sony AI.
gain recognition, which also violates the model's IP. Therefore, it is essential to track the origin of AI-generated images. The origin attribution problem is to identify whether a specific image is generated by a particular model. As shown in Fig. 1, assuming we have a model  $\mathcal{M}_1$  and its generated images, the origin attribution algorithm's objective is to flag an image as belonging to model  $\mathcal{M}_1$  if it was generated by that model. On the other hand, the algorithm should consider the image as non-belonging if it was created by other models (e.g.,  $\mathcal{M}_2$  in Fig. 1) or if it is a real image.
One existing way to infer the source of specific images is image watermarking. It works by embedding ownership information in carrier images to verify the owner's identity and authenticity [20- 23]. The image watermarking-based method requires an additional modification to the generation results in a posthoc manner, which may impair the generation quality. Also, it might not necessarily reflect the use of a particular model in the process of generating an image, which can reduce its credibility as evidence in lawsuits. Furthermore, the watermark can also be stolen, and malicious users can engage in criminal activities and disguise their identities us
Fig. 1: Illustration for origin attribution problem. The origin attribution algorithm aims to judging whether the given images belong to a particular model, i.e., Model  $\mathcal{M}_1$ .
ing the stolen watermark. Another approach to identifying the source models of the generated samples [24-27] is injecting fingerprinting into the models (e.g., modifying the model architecture) and training a supervised classifier to detect the fingerprints presented in the image. While their goal is similar to ours, these methods have several limitations. Firstly, they require extra operations during the model training phase, and they cannot be applied on pre-trained models without additional operations, such as modifying the model architecture [26, 27]. Secondly, since these methods modify the training or inference process of generative models, the model's generation performance may be affected. Thirdly, these previous studies mainly focus on a particular kind of generative model, i.e., GAN [5]. In contrast, our goal is to develop an origin attribution approach for different types of generative models, including diffusion models [4, 3] (model-agnostic), without requiring any extra operations in the training phase and image generation phase (alteration-free). We summarize the differences between our method and existing methods in Table 1.
In this paper, we propose a method for origin attribution that is based on the input reverse-engineering task on generative models (i.e., inverting the input of a particular generative model for a specific image). The intuition behind our method is that the reverse-engineering task is easier for belonging images than non-belonging images. Therefore, we design our method
Table 1: Summary of the differences between our method and existing methods.
Method	Training-phase Alteration-free	Generation-phase Alteration-free	Model-agnostic
Image Watermark [20–23]	✓	✗	✓
Classifier-based [24–27]	✗	✓	✗
Ours	✓	✓	✓
based on the differences in the reconstruction loss for the reverse-engineering between generated images of a given model and other images. The origin attribution method we propose starts by using the model to generate a set of images and calculate the reconstruction loss on these generated images. To eliminate the influence of the inherent complexities of the examined images, we also calibrate the reconstruction loss by considering the hardness of the reverse-engineering on any other model that has strong generation ability, but different architectures and training data. Afterwards, the algorithm computes the calibrated reconstruction loss for the examined image and uses statistical hypothesis testing to determine if the reconstruction loss falls within the distribution of reconstruction losses observed in the generated images. This allows us to distinguish images generated by the given model from other images in a model-agnostic and alteration-free manner. Based on our design, we implemented a prototype called RONAN (Reverse-engineering-based OrigiN Attribution) in PyTorch and evaluated it on three different types of generative models (i.e., unconditional model, class-to-image model, and text-to-image model) including various GANs [1, 10, 28–30], VAEs [6], and diffusion models such as latest Consistency Model [4] and Stable Diffusion [3]. Results demonstrate our
method is effective for the "alteration-free and model-agnostic origin attribution" task. On average, RONAN achieves  $96.07\%$  of true positive rate with a false positive rate around  $5.00\%$ .
Our contributions are summed as follows: ① We introduce a new task called "alteration-free and model-agnostic origin attribution", which entails determining whether a specific image is generated by a particular model without requiring any additional operations during the training and generation phases. ② To accomplish this task, we analyze the differences in the reconstruction loss for reverse-engineering between the generated images of a given model and other images. Based on our analysis, we design a novel method that involves conducting input reverse-engineering and checking whether the reconstruction loss of the examined sample falls within the distribution of reconstruction losses observed in the generated images. ③ We evaluate our method on various different image generation models. The results show that our method effectively distinguishes images generated by the given model from other images in a model-agnostic and alteration-free manner. Our code can be found in https://github.com/ZhentingWang/RONAN.
Detection of AI-Generated Contents. Detecting AI-generated content has become extremely important with the growing concerns about the misuse of AIGC technology [31]. The detection of AI-generated content is a binary classification problem that involves distinguishing generated samples from real ones. In the CV field, existing research has found that visually imperceptible but machine-distinguishable patterns in generated images, such as noise patterns [16, 32], frequency signals [33-36] and texture representation [37] can be used as the clues of AI-generated images. Researchers also proposed methods to detect sentences generated by generative NLP models such as ChatGPT [38-41]. Although these methods achieve promising performance for distinguishing AI-generated content and real content, they cannot infer if a specific content is generated by a given generative model, which is a novel but more challenging task and is the main focus of this paper.
Tracking Origin of Generated Images. There are several ways to track the source of the generated images. Image watermarking that pastes ownership information in carrier images [20-23] can be adapted to discern whether a specific sample is from a specific source. Watermarks can take the form of specific signals within the images, such as frequency domain signals [22] or display-camera transformations [42]. However, it requires an additional modification to the generation results in a post-hoc manner, and it does not necessarily reflect whether the image was generated by a particular model when the judges use it as the evidence (different models can use the same watermark). Another way is to inject fingerprints into the model during training and train a supervised classifier on it to discern whether an image is from a fingerprinted GAN model [24-27]. For example, Yu et al. [27] modify the architecture of the convolutional filter to embed the fingerprint, and they train the generator alongside a fingerprinting classifier capable of identifying the fingerprint and its corresponding source GAN models. It requires a modified model architecture, altered training process, and an additional procedure to train a source classifier. There are several existing methods [43-45] track the origin of the generated images by exploiting the input reconstruction. However, the practicality of these approaches is limited due to their reliance on strong assumptions (see § 5.5 for more details).
We focus on serving as an inspector to infer if a specific sample is generated by a particular model in an alteration-free and model-agnostic manner. To the best of our knowledge, this paper is the first work focusing on this problem. To facilitate our discussion, we first define the belonging and non-belonging of the generative models.
Definition 3.1 (Belonging of Generative Models). Given a generative model  $\mathcal{M}:\mathcal{I}\mapsto \mathcal{X}_{\mathcal{M}}$  where  $\mathcal{I}$  is the input space and  $\mathcal{X}_{\mathcal{M}}$  is the output space. A sample  $\pmb{x}$  is a belonging of model  $\mathcal{M}$  if and only if  $\pmb {x}\in \mathcal{X}_{\mathcal{M}}$ . We call a sample  $\pmb{x}$  a non-belonging if  $x\notin \mathcal{X}_{\mathcal{M}}$ .
Inspector's Goal. Given a sample  $\pmb{x}$  and a generative model  $\mathcal{M} : \mathcal{I} \mapsto \mathcal{X}_{\mathcal{M}}$  where  $\mathcal{I}$  is the input space and  $\mathcal{X}_{\mathcal{M}}$  is the output space of  $\mathcal{M}$ , the inspector's goal is to infer if a given image  $\pmb{x}$  is a belonging of  $\mathcal{M}$ . Formally, the goal can be written as constructing an inference algorithm  $\mathcal{B} : (\mathcal{M}, \pmb{x}) \mapsto \{0,1\}$  that receives a sample  $\pmb{x}$  and a model  $\mathcal{M}$  as the input, and returns the inference result (i.e., 0 denotes belongings, and 1 denotes non-belongings). The algorithm  $\mathcal{B}$  can distinguish not only the belongings of the given model  $\mathcal{M}$  and that of the other models (e.g., trained on different training data, or have different model architectures), but also the belongings of  $\mathcal{M}$  and the natural samples that are not generated by AI. The inspector also aims to achieve the following goals:
Alteration-free: The algorithm  $\mathcal{B}$  does not require any extra operations/modifications in the training phase and image generation phase.
Model-agnostic: The algorithm  $\mathcal{B}$  can be applied to different types of image generative models with different architectures.
Inspector's Capability. The inspector has white-box access to the provided model  $\mathcal{M}$ , thus the inspector can get the intermediate output and calculate the gradient of the models. In addition, the inspector cannot control the development and training process of the provided models. Note that the white-box access to the inspected model is practical especially in the scenarios where the inspector is the owner of the inspected models.
Real-world Application. The inspection algorithm can be widely used in various scenarios where it is necessary to verify the authenticity of generated images. We provide three examples as follows:
Copyright protection of image generation models: A copyright is a kind of intellectual property (IP) that provides its owner the exclusive right to perform a creative work [46]. In this scenario, a party suspects that a specific image may have been generated by their generative model without authorization, such as if a malicious user has stolen the model and used it to generate images. The party can then request an inspector to use our proposed method to infer if the doubtful image was indeed generated by their particular model, and the resulting inference can be used as a reference in a lawsuit. It is important to clarify that inferring an image as a belonging of a model does not imply the IP of this image is totally belonging to this model. In fact, determining the ownership of the IP related to the generated images remains an unresolved challenge in the field of law. This complexity arises due to the involvement of multiple entities (such as contributors of training data, model trainers, input/prompt providers, and the models themselves) throughout the image generation process. The inferring results of our origin attribution method can serve as a valuable reference for addressing IP protection concerns, instead of a definitive conclusion.
Tracing the source of maliciously generated images: Assume a user creates malicious images containing inappropriate or biased content and distributes them on the internet. The cyber police can utilize our proposed method to infer if the image was generated by a model belonging to a specific user. The resulting inference can be used as a reference for criminal evidence in lawsuits.
Detecting AI-powered plagiarism: Our method can also be used for detecting AI-powered plagiarism. For example, imagine a scenario where an individual generates AI-created images (e.g., using Midjourney) and dishonestly presents them as their own original artwork (e.g., photographs and paintings) to gain recognition and reputation. In such cases, the model owner (e.g., Midjourney's owner) may suspect that the image is generated using their model (e.g., Midjourney). Our proposed method can then be employed to uncover such instances of AI-powered plagiarism.
Our method is built on the input reverse-engineering for image generation models. In this section, we start by formulating the input reverse-engineering task, followed by an analysis of the disparities in reconstruction loss between images generated by a particular model and those from other sources. We then proceed to present a detailed algorithm for our method.
4.1 Reverse-engineering
Recent researches [47, 17] demonstrate that the input of the modern image generative models is reconstructable by using the generated images. Here, we view the reverse-engineering as an optimization problem. Formally, it can be defined as follows:
Definition 4.1 (Input Reverse-engineering). Given a generative model  $\mathcal{M}:\mathcal{I}\mapsto \mathcal{X}_{\mathcal{M}}$ , and an image  $\pmb{x}$ , an input reverse-engineering task is optimizing the input  $\pmb{i}$  to make the corresponding output image from the model  $\mathcal{M}(i)$  as close as possible to the given image  $\pmb{x}$ .
The input reverse-engineering task is performed by a reverse-engineering algorithm  $\mathcal{A}:(\mathcal{M},\boldsymbol {x})\mapsto R$ , which can be written as:
$$
\boldsymbol {i} ^ {\star} = \underset {\boldsymbol {i}} {\arg \min } \mathcal {L} (\mathcal {M} (\boldsymbol {i}), \boldsymbol {x}), \quad \mathcal {A} (\mathcal {M}, \boldsymbol {x}) = \mathcal {L} (\mathcal {M} (\boldsymbol {i} ^ {\star}), \boldsymbol {x}) \tag {1}
$$
where  $\mathcal{L}$  is a metric to measure the distance between different images, and  $i^{\star}$  is the reverse-engineered input. The given value for the reverse-engineering algorithm  $\mathcal{A}$  is a specific image  $\pmb{x}$  and a particular model  $\mathcal{M}$ . The returned value of the algorithm  $\mathcal{A}$  is the distance between the given image and its
(a) DCGAN [1]
(b) VAE [6]
(c) StyleGAN2-ADA [10]
Fig. 2: Reconstruction loss distributions for belonging images and real images.
reverse-engineered version, i.e.,  $\mathcal{L}(\mathcal{M}(i^{\star}),\pmb {x})$  , which is called as the reconstruction loss. We use the reconstruction loss to measure the hardness of the input reverse-engineering task.
Based on the definitions and formulations, we have the following theorem:
Theorem 4.2. Given a generative model  $\mathcal{M}:\mathcal{I}\mapsto \mathcal{X}_{\mathcal{M}}$ , and a reverse-engineering algorithm  $\mathcal{A}$ , if the model is deterministic (i.e., it produces the same output given the same input) and the reverse-engineering algorithm is perfect (i.e., it can find the global minimum of the reconstruction loss for the reverse-engineering), then for any  $\pmb {x}\in \mathcal{X}_{\mathcal{M}}$  (belonging) and  $\pmb{x}^{\prime}\notin \mathcal{X}_{\mathcal{M}}$  (non-belonging) we have  $\mathcal{A}(\mathcal{M},\pmb{x}^{\prime}) > \mathcal{A}(\mathcal{M},\pmb {x})$ .
The proof for Theorem 4.2 can be found in the Appendix A. The theorem demonstrates that the reconstruction loss of the images generated by a specific model will be lower than that of images that are not generated by the model. The theorem also establishes that the distribution of reconstruction loss values for belonging and non-belonging images is perfectly separable. Thus, we can use a threshold value to separate the belonging images and non-belonging images. In the real world, many image generation models incorporate random noises into their image generation procedures to enhance the variety of images they produce. However, these models can also be deemed deterministic since we can regard all the random noises utilized in the generation procedure as parts of the inputs. On the other hand, in reality, the reverse-engineering algorithm may get stuck at a local minimum, and it is hard to guarantee the achievement of the global minimum. This is where the formula  $\mathbb{P}(\mathcal{A}(\mathcal{M},\boldsymbol{x}^{\prime}) > \mathcal{A}(\mathcal{M},\boldsymbol{x}))\geq \lambda$  becomes relevant as it serves as a relaxation for Theorem 4.2, explaining the practical scenario. In this formula,  $\lambda$  (e.g.,  $90\%$ ) acts as a separability level for distinguishing between the two distributions: belonging images and non-belonging images.
To investigate the practical scenario, we conduct experiments on the CIFAR-10 [48] dataset using DCGAN [1], VAE [6], and StyleGAN2-ADA [10] models. The results are depicted in Fig. 2, where the x-axis represents the reconstruction loss measured by the MSE (Mean Squared Error) [49] metric, and the y-axis indicates the percentage of images whose reconstruction loss value corresponds to the corresponding value on the x-axis. We use blue color to denote 100 generated images of the given model and orange to represent 100 real images randomly sampled from the training data of the model. The results indicate that the reconstruction losses of the generated images (belongings) and those not generated by this model (non-belongings) can be distinguished.
4.2 Calibration
Different images have different inherent complexities [50-52]. Some images may be harder to reverse-engineer due to their higher complexity (e.g., containing more objects, colors, and details). In that case, the reconstruction loss will also be influenced by the inherent complexities of the examined images. To increase the separability level of belonging images and others, we disentangle the influence of the inherent complexities and the belonging status by measuring the complexity of the image and use it to calibrate the reconstruction loss. By default, we use the reconstruction loss on a reference image generation model  $\mathcal{M}_r$  that is trained on a different dataset as the measurement of the complexity, i.e.,  $\mathrm{Complexity}(\pmb{x}) = \mathcal{A}(\mathcal{M}_r,\pmb{x})$  (we use the Consistency model [4] pre-trained on ImageNet dataset [53] as the reference model by default). We also discuss other implements to measure the complexity of the inspected images in Appendix F. The calibrated reconstruction loss  $\mathcal{A}'(\mathcal{M},\pmb{x})$  is defined as follows:
$$
\mathcal {A} ^ {\prime} (\mathcal {M}, \boldsymbol {x}) = \frac {\mathcal {A} (\mathcal {M} , \boldsymbol {x})}{\text {C o m p l e x i t y} (\boldsymbol {x})} \tag {2}
$$
Algorithm 1 Origin Attribution
Input: Model:  $\mathcal{M}$ , Examined Data:  $x$
Output: Inference Results: Belonging or Non-belonging
1: function INFERENCE(M, x)
2: Obtaining Belonging Distribution (Offline)
3:  $\mu, \sigma, N = \text{BelongingDistribution}(\mathcal{M})$
4:  $\triangleright$  Reverse-engineering
5:  $\mathcal{A}'(\mathcal{M},\boldsymbol{x})\gets$  Calibrated Reconstruction Loss [Eq. 2]
6: Determining Belonging
7: InferenceResults = HypothesisTesting  $(\mathcal{A}'(\mathcal{M}, \boldsymbol{x}), \mu, \sigma, N)$  [Eq. 3]
8: return InferenceResults
4.3 Belonging Inference via Hypothesis Testing
We use Grubbs' Hypothesis Testing [54] to infer if a specific sample  $\pmb{x}$  is a belonging of the particular given model  $\mathcal{M}$ . We have a null hypothesis  $H_0: \pmb{x}$  is a non-belonging of  $\mathcal{M}$ , and the alternative hypothesis  $H_1: \pmb{x}$  is a belonging of  $\mathcal{M}$ . The null hypothesis  $H_0$  is rejected (i.e., the alternative hypothesis  $H_1$  is accepted) if the following inequality (Eq. 3) holds:
$$
\frac {\mathcal {A} ^ {\prime} (\mathcal {M} , \boldsymbol {x}) - \mu}{\sigma} <   \frac {(N - 1)}{\sqrt {N}} \sqrt {\frac {\left(t _ {\alpha / N , N - 2}\right) ^ {2}}{N - 2 + \left(t _ {\alpha / N , N - 2}\right) ^ {2}}} \tag {3}
$$
Here,  $\mu$  and  $\sigma$  are the mean value and standard deviation for the calibrated reconstruction loss on belonging samples of model  $\mathcal{M}$ . Since model  $\mathcal{M}$  is given to the inspector, the inspector can calculate  $\mu$  and  $\sigma$  by using  $\mathcal{M}$  to generate multiple images with randomly sampled inputs.  $N$  is the number of generated belonging images.  $\mathcal{A}'(\mathcal{M},\boldsymbol{x})$  is the calibrated reconstruction loss of the examined image  $\boldsymbol{x}$ .  $t_{\alpha /N,N - 2}$  is the critical value of the  $t$  distribution with  $N - 2$  degrees of freedom and a significance level of  $\alpha /N$ , where  $\alpha$  is the significance level of the hypothesis testing (i.e., 0.05 by default in this paper). The critical value of the  $t$  distribution (i.e.,  $t_{\alpha /N,N - 2}$ ) can be computed using the cumulative distribution function (See Appendix B for more details).