SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our objective is to investigate whether AI models can distinguish if an image is AI-generated or not. We conduct a large and comprehensive model evaluation using Fake2M training dataset and 11 fake validation datasets to build MPBench.
We compare the following SOTA methods: (1) Wang et al. [57] proposed to finetune a classification network to give a real/fake decision for an image using Blur and JPEG augmentation with a pretrained visual backbone, such as ResNet-50 [25] and ConvNext-S [35]. (2) Ojha et al. [42] proposed to use a frozen CLIP-ViT-L [44] for backbone and train the last linear layer for the same task [42].
To ensure a fair comparison, we train each model using four different dataset settings, consisting of 1 million fake images and an equivalent number of real images: (1) Dataset Setting A: "SD-V1.5Real-dpms-25" dataset with 1M fake images and "CC3M-Train" dataset with 1M real images. (2) Dataset Setting B: "IF-V1.0-dpms++-25" dataset with 1M fake images and "CC3M-Train" dataset with 1M real images. (3) Dataset Setting C: "StyleGAN3" dataset with 87K fake images and "StyleGAN3-Train" with 87K real images. (4) Dataset Setting D: "SD-V1.5Real-dpms-25, IF-V1.0-dpms++-25, StyleGAN3, CC3M-Train, StyleGAN3-Train" dataset with 460K fake images selected from "SDV1.5Real-dpms-25" dataset, 460K fake images selected from "IF-V1.0-dpms++-25" dataset, 87K
Table 7: Quantitative comparison of five models under four training dataset settings with fourteen validation datasets. "Diff" refers to diffusion model, "AR" refers to autoregressive model and "Unk." refers to unknown model. Real (R) denotes the dataset consisting entirely of real images. Fake (F) denotes the dataset consisting entirely of fake images.
Model	Training Dataset	ImageNet-Test CelebA-HQ-Train CC3M-Val	Average Acc.	Fake	Average Acc.	Total Average Acc.
Real	SD-V2.1-dpm-25	SD-V1.5-dpm-25	IF-V1.0-dpm++10	IF-V1.0-dpm++25	IF-V1.0-dpm+50	Cogview2	Midjourney	StyleGAN3	Average Acc.
-	-	-	-	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	Diff.	AR	Unk.	GAN	-
ConvNext-S(B+J 0.1)	Dataset Setting A: SD-V1.5Real-dpms-25 (1M) CC3M-Train (1M)	95.3	99.9	99.9	98.3	48.6	99.9	85.1	92.9	57.6	55.9	41.1	72.6	55.7	41.6	37.4	62.5	70.2
ConvNext-S(B+J 0.5)	95.9	99.9	99.9	98.5	53.5	100	83.3	91.1	50.2	49.6	35.3	66.9	54.9	44.7	35.7	60.4	68.6
ResNet50(B+J 0.1)	93.0	95.3	95.6	94.6	71.7	71.8	98.6	57.2	26.6	29.0	23.6	47.9	11.8	40.3	9.1	44.3	55.1
ResNet50(B+J 0.5)	93.2	95.5	95.8	94.8	70.6	70.4	98.5	52.1	23.6	26.1	21.6	46.2	10.6	36.6	7.2	42.1	53.4
CLIP-ViT-L(LC)	49.6	87.0	75.4	70.6	73.3	86.6	97.6	93.9	77.8	71.4	84.1	90.9	86.6	88.7	86.1	85.1	82.0
ConvNext-S(B+J 0.1)	Dataset Setting B: IF-V1.0-dpms++-25 (1M) CC3M-Train (1M)	87.6	99.9	99.9	95.8	2.2	34.5	2.5	99.7	99.9	99.9	11.1	66.4	19.2	8.9	10.1	41.3	52.9
ConvNext-S(B+J 0.5)	87.8	99.9	99.9	95.8	3.9	39.1	3.9	99.6	99.9	99.8	18.5	79.2	25.8	8.1	8.0	44.1	55.2
ResNet50(B+J 0.1)	89.4	95.8	95.0	93.4	37.5	56.5	20.0	84.0	95.6	91.7	39.7	69.4	45.3	15.7	8.8	51.2	60.3
ResNet50(B+J 0.5)	90.8	95.6	94.5	93.6	41.6	58.8	21.7	82.3	95.2	91.3	47.0	79.7	56.1	18.3	6.8	54.4	62.8
CLIP-ViT-L(LC)	81.5	83.3	93.0	85.9	38.2	18.5	13.1	80.4	79.7	70.9	61.1	77.7	76.6	33.7	32.8	52.9	60.0
ConvNext-S(B+J 0.1)	Dataset Setting C: StyleGAN3 (87K) StyleGAN3-Train (87K)	58.7	81.0	62.7	67.4	44.9	42.7	39.4	40.7	42.3	42.2	35.7	39.9	53.2	41.6	69.9	44.7	49.6
ConvNext-S(B+J 0.5)	67.7	92.6	71.2	77.1	33.1	33.7	32.1	33.6	36.5	34.2	35.0	35.5	44.8	28.5	42.6	35.4	44.3
ResNet50(B+J 0.1)	31.5	14.3	39.1	28.3	67.7	62.7	68.1	65.3	71.0	67.0	58.8	60.9	60.9	81.9	90.9	68.6	60.0
ResNet50(B+J 0.5)	70.6	29.5	63.4	54.5	37.3	34.9	40.0	43.3	43.9	41.9	34.8	38.4	47.8	58.1	81.3	45.6	47.5
CLIP-ViT-L(LC)	37.6	85.9	70.2	64.5	85.7	92.9	94.8	95.3	89.5	84.0	87.5	93.0	84.6	81.4	61.9	86.4	81.7
ConvNext-S(B+J 0.1)	Dataset Setting D: SD-V1.5Real-dpms-25 (460K) IF-V1.0-dpms++-25 (460K) StyleGAN3 (87K) CC3M-Train (1M) StyleGAN3-Train (87K)	95.4	99.9	99.9	98.4	39.9	95.1	99.9	99.7	99.8	99.7	43.1	90.4	48.3	32.4	99.8	77.1	81.6
ConvNext-S(B+J 0.5)	97.7	99.9	99.9	99.1	52.8	92.9	99.9	99.5	99.7	99.3	46.4	91.7	48.6	35.0	99.9	78.7	83.0
ResNet50(B+J 0.1)	77.3	93.6	91.7	87.5	83.9	90.4	96.8	91.2	90.6	86.7	52.3	82.2	56.4	49.4	80.8	78.2	80.2
ResNet50(B+J 0.5)	84.5	94.7	92.1	90.4	84.4	89.4	96.1	88.9	89.4	85.6	52.4	83.8	55.4	45.8	69.5	76.4	79.4
CLIP-ViT-L(LC)	50.4	96.0	79.8	75.4	66.9	87.8	93.6	96.1	85.9	78.9	84.9	92.6	87.4	79.8	62.8	83.3	81.6
fake images from "StyleGAN3" dataset, 1M real images from "CC3M-Train" dataset and 87K real images from "StyleGAN3-Train" dataset.
We evaluate all the models using 11 validation datasets to build MPBench, as shown in Tab 7. To clearly show the performance of the models under different validation datasets, we directly report the classification accuracy of each model.
4.2.1 Comparative Analysis of Accuracy Across Various Models
Results. We conducted a systematic analysis of the mean accuracy achieved by different models, each trained using an identical dataset. Our findings are summarized in Tab. 7. Notably, the leading performing models for fake image detection vary depending on the training dataset. For example, under Dataset Setting A, CLIP-ViT-L(LC) excels, whereas under Dataset Setting B, ResNet50 (B+J 0.5) outperforms others. Furthermore, CLIP-ViT-L(LC) achieves the highest fake image detection accuracies of  $86.4\%$  and  $83.3\%$  under Dataset Setting C and Dataset Setting D respectively. A compelling finding from our study is the absence of a single model that consistently delivers superior performance across all dataset settings for fake image detection. However, our experiments also suggest that ConvNext models are more adept at achieving higher average accuracies in detecting real images. In all four dataset settings, ConvNext consistently achieves top accuracies— $98.5\%$ ,  $95.8\%$ ,  $77.1\%$ , and  $99.1\%$  for Dataset Setting A,B,C,D, respectively.
Analysis. Our observations highlight the need for models that perform optimally on both real and fake images. According to our study, while CLIP-ViT-L(LC) most often produces the best results for fake images, ConvNext outperforms CLIP-ViT-L(LC) in real image detection. However, real-world applications necessitate discerning between real and fake images. This underscores the need for future research to strike a balance between detection capabilities for both categories.
4.2.2 Comparative Analysis of Accuracy Across Various Training Datasets
Results. We conduct an empirical analysis of the average accuracy of various models, each trained using different dataset configurations. The results, as presented in Tab. 7, indicate that Dataset Setting D generally yields superior model performance. ConvNext and ResNet50 models specifically show marked improvements when trained on Dataset Setting D. Meanwhile, CLIP-ViT-L (LC) demonstrates comparable performance across different dataset configurations, attaining accuracies of  $82.0\%$  with Dataset Setting A and  $81.6\%$  with Dataset Setting D.
Analysis. Our experimental findings suggest that the choice of training dataset significantly influences model performance in fake image detection tasks. Notably, a diversified dataset like Dataset Setting D, which includes five distinct generative models, seems to enhance overall model accuracy. This improvement is likely due to the exposure to a broader spectrum of generative styles and variations that a diversified dataset offers. Additionally, our results highlight the aptitude of the proposed dataset, Fake2M, which features diverse data sources, for more generalized fake image detection.
4.2.3 Comparative Analysis of Accuracy Across Various Validation Datasets
Results. In our experiment, we assessed the accuracy of a trained model across various validation datasets. As depicted in Tab. 7, the performance of a model varies significantly based on the generation models, sampling methods, and sampling steps used in the validation set. For instance, the ConvNext-S (B+J 0.5) model, when trained using Dataset Setting D, displayed a broad range of validation results for fake image detection, with accuracies spanning between  $35.0\%$  and  $99.9\%$ . These findings underscore the influence of validation set characteristics on a model's performance in the task of fake image detection.
Analysis. In realistic applications, fake images can originate from a variety of generation models with a diverse range of hyperparameters. Consequently, an ideal fake image detection model should demonstrate consistent proficiency across this broad spectrum of generation settings. However, our experimental results show that existing models do not meet this requirement, indicating a need for the development of new approaches to handle the challenge of detecting fake images generated under varying settings.
4.2.4 Evaluate the best model under the same setting used in HPBench.
Based on its highest total average accuracy in MPBench, we select ConvNext-S (B+J 0.5) with Dataset Setting D as the best model. We then evaluate this model under the same setting in HPBench which consists of 50 real images and 50 fake images for testing and it achieves a  $13\%$  failure rate.
Image Generation Models. State-of-the-art text-to-image synthesis approaches such as DALL-E 2 [45], Imagen [48], Stable Diffusion [46], IF [4], and Midjourney [6] have demonstrated the possibility of that generating high-quality, photorealistic images with diffusion-based generative models trained on large datasets [49]. Those models have surpassed previous GAN-based models [12, 24, 32, 60] in both fidelity and diversity of generated images, without the instability and mode collapse issues that GANs are prone to. In addition to diffusion models, other autoregressive models such as Make-A-Scene [22], CogView [19], and Parti [59] have also achieved amazing performance.
Fake Images Generation and Detection. In recent years, there have been many works [9, 13, 15, 21, 39, 40, 42, 57, 62] exploring how to distinguish whether an image is AI-generated. These works focus on fake contents generated by GANs or small generation models [12, 24, 32, 60]. Due to the limited quality of images generated by those methods, it is easy for humans to distinguish whether a photo is AI-generated or not. However, as the quality of generated images continues to improve with the advancement of recent generative models [6, 45, 46, 48], it has become increasingly difficult for humans to identify whether an image is generated by AI or not. The need for detecting fake images has existed even before we had powerful image generators.
In this study, we present a comprehensive evaluation of both human discernment and contemporary AI algorithms in detecting fake images. Our findings reveal that humans can be significantly deceived by current cutting-edge image generation models. In contrast, AI fake image detection algorithms demonstrate a superior ability to distinguish authentic images from fakes. Despite this, our research highlights that existing AI algorithms, with a considerable misclassification rate of  $13\%$ , still face significant challenges. We anticipate that our proposed dataset, Fake2M, and our dual benchmarks, HPBench and MPBench, will invigorate further research in this area and assist researchers in crafting secure and reliable AI-generated content systems. As we advance in this technological era, it is crucial to prioritize responsible creation and application of generative AI to ensure its benefits are harnessed positively for society.
Acknowledgments and Disclosure of Funding. This research is funded by Shanghai AI Laboratory. This work is partially supported by the National Key R&D Program of China (NO.2022ZD0160100), (NO.2022ZD0160101). We would like to thank many colleagues for useful discussions, suggestions, feedback, and advice, including: Peng Ye, Min Shi, Weiyun Wang, Yan Teng, Qianyu Guo, Kexin Huang, Yongqi Wang and Tianning Zhang.
[1] 500px. https://500px.com/. Accessed: 2023-04-17. 3
[2]bbcnews:Fake trump arrest photos:How to spot an ai-generated image.https://www.bbc.com/news/ world-us-canada-65069316.Accessed:2023-04-18.2
[3]bbcnews:Sonyworldphotographyaward2023:Winnerrefusestewardafterrevealingai creation. https://www.bbc.com/news/entertainment-arts-65296763.Accessed:2023-04-17.2
[4] Deepfloyd. if. https://github.com/deep-floyd/IF. Accessed: 2023-06-7. 3, 9
[5] Google images. https://images.google.com/. Accessed: 2023-04-17. 3
[6] Midjourney. https://www.midjourney.com/. Accessed: 2023-04-17. 3,9
[7] photutorial: Number of photos (2023): Statistics, facts, & predictions. https://photutorial.com/. Accessed: 2023-04-18. 2
[8] Stable diffusion v1.5. realistic vision v2.0. https://civitai.com/models/4201/realistic-vision-v20. Accessed: 2023-06-7. 3
[9] Vishal Asnani, Xi Yin, Tal Hassner, Sijja Liu, and Xiaoming Liu. Proactive image manipulation detection. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022. 9
[10] Yogesh Balaji, Seungjun Nah, Xun Huang, Arash Vahdat, Jiaming Song, Karsten Kreis, Miika Aittala, Timo Aila, Samuli Laine, Bryan Catanzaro, Tero Karras, and Ming-Yu Liu. ediff-i: Text-to-image diffusion models with an ensemble of expert denoisers. arXiv preprint arXiv:2211.01324, 2022. 2
[11] Jordan J Bird and Ahmad Lotfi. Cifake: Image classification and explainable identification of ai-generated synthetic images. arXiv preprint arXiv:2303.14126, 2023. 5
[12] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. International Conference on Learning Representations, 2019. 2, 9
[13] Lucy Chai, David Bau, Ser-Nam Lim, and Phillip Isola. What makes fake images detectable? understanding properties that generalize. European Conference on Computer Vision, 2020. 9
[14] Yunjey Choi, Youngjung Uh, Jaejun Yoo, and Jung-Woo Ha. Stargan v2: Diverse image synthesis for multiple domains. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 4
[15] Davide Cozzolino, Giovanni Poggi, and Luisa Verdoliva. Splicebuster: A new blind image splicing detector. IEEE International Workshop on Information Forensics and Security, 2015. 9
[16] Hao Dang, Feng Liu, Joel Stehouwer, Xiaoming Liu, and Anil K. Jain. On the detection of digital face manipulation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 5
[56] Run Wang, Felix Juefei-Xu, Lei Ma, Xiaofei Xie, Yihao Huang, Jian Wang, and Yang Liu. Fakespotter: A simple yet robust baseline for spotting ai-synthesized fake faces. International Joint Conference on Artificial Intelligence, 2020. 5
[57] Sheng-Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A. Efros. Cnn-generated images are surprisingly easy to spot... for now. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020. 5, 7, 9
[58] Yabin Wang, Zhiwu Huang, and Xiaopeng Hong. Benchmarking deeppart detection. arXiv preprint arXiv:2302.14475, 2023. 5
[59] Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang, Burcu Karagol Ayan, Ben Hutchinson, Wei Han, Zarana Parekh, Xin Li, Han Zhang, Jason Baldridge, and Yonghui Wu. Scaling autoregressive models for content-rich text-to-image generation. Advances in Neural Information Processing Systems, 2022. 9
[60] Han Zhang, Jing Yu Koh, Jason Baldridge, Honglak Lee, and Yinfei Yang. Cross-modal contrastive learning for text-to-image generation. IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021. 2, 9
[61] Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint arXiv:2302.05543, 2023. 2
[62] Xu Zhang, Svebor Karaman, and Shih-Fu Chang. Detecting and simulating artifacts in GAN fake images. IEEE International Workshop on Information Forensics and Security, 2019. 9