A Sanity Check for AI-generated Image Detection
Authors: Shilin Yan
Ouxiang Li
Jiayin Cai
Yanbin Hao
Xiaolong Jiang
Yao Hu
Weidi Xie
Venue: ICLR 2025 Poster (2025)
Keywords: AI-generated image detection
Chameleon dataset
mixture-of-experts
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: With the rapid development of generative models, discerning AI-generated content has evoked increasing attention from both industry and academia. In this paper, we conduct a sanity check on whether the task of AI-generated image detection has been solved. To start with, we present Chameleon dataset, consisting of AI-generated images that are genuinely challenging for human perception. To quantify the generalization of existing methods, we evaluate 9 off-the-shelf AI-generated image detectors on Chameleon dataset. Upon analysis, almost all models misclassify AI-generated images as real ones. Later, we propose AIDE AI-generated Image DEtector with Hybrid Features, which leverages multiple experts to simultaneously extract visual artifacts and noise patterns. Specifically, to capture the high-level semantics, we utilize CLIP to compute the visual embedding. This effectively enables the model to discern AI-generated images based on semantics and contextual information. Secondly, we select the highest and lowest frequency patches in the image, and compute the low-level patchwise features, aiming to detect AI-generated images by low-level artifacts, for example, noise patterns, anti-aliasing effects. While evaluating on existing benchmarks, for example, AIGCDetectBenchmark and GenImage, AIDE achieves +3.5% and +4.6% improvements to state-of-the-art methods, and on our proposed challenging Chameleon benchmarks, it also achieves promising results, despite the problem of detecting AI-generated images remains far from being solved.

Published as a conference paper at ICLR 2025
A SANITY CHECK FOR AI-GENERATED IMAGE DETEC- TION
Shilin Yan',, Ouxiang Li ,Jayin Cai',, Yanbin , Xiaolong Jiang', Yao Hu', Weidi Xie
1 Xiaohongshu Inc. University of Science and Technology of China "Shanghai Jiao Tong University
tattoo. ysl0yy1100mmmm com weidi@ssju..dd..
Project Page: https::/shilinyannt itt.t.........E.
ABSTRACT
With the rapid development of generative models, discerning Al-generated con- tent has evoked increasing attention from both industry and academia. In this paper, we conduct a sanity check on whether the task of Al-generated im- age detection has been solved. To start with, present Chameleon dataset, consisting of Al-generated images that are genuinely challenging for human per- ception. To quantify the generalization of existing methods, evaluate 9 off- the-shelf Al-generated image detectors on Chameleon dataset. Upon analysis, almost all models misclassify Al-generated images as real ones. Later, we pro- pose AIDE (Al-generated Image DEtector with Hybrid Features), which lever- ages multiple experts to simultaneously extract visual artifacts and noise pat- terns. Specifically, to capture the high-level semantics, we utilize CLIP to com- pute the visual embedding. This effectively enables the model to discern AI- generated images based semantics and contextual information. Secondly, we select the highest and lowest frequency patches in the image, and compute the low-level patchwise features, aiming to detect Al-generated images by low-level artifacts, for example, noise patterns, anti-aliasing effects. While evaluating on existing benchmarks, for example, AIGCDetectBenchhark and Genlmage, AIDE achieves +3.5% and +4.6% improvements to state-of-the-ard methods, and on our proposed challenging Chameleon benchmarks, it also achieves promising re- sults, despite the problem of detecting Al- Al-generated images remains far from be- ing solved.
INTRODUCTION
Recently, the vision community has witnessed remarkable advancements in generative models. These methods, ranging from generative adversarial networks (GANs) (Goodfellow et al., 2014;
Zhu al., 2017; Brock et al., 2019; Karras al., 2019) to diffusion models (DMs) (Ho al., 2020; Nichol & Dhariwal, 2021; Rombach et al., 2022; Song et al., 2021; Liu et al., 2022b; Lu et al., 2022; Hertz et al., 2023; Nichol et al., 2022) have demonstrated unprecedented capabilities in synthesizing high-quality images that closely resemble real-world scenes. On the positive side, such generative models have enabled various valuable tools for artists and designers, democratizing access to ad- vanced graphic design capabilities. However, it also raises concerns about the authenticity of visual content, posing significant challenges for image forensics (Ferreira et al., 2020), misinformation combating (Xu et al., 2023a), and copyright protection (Ren al., 2024). In this paper, we consider the problem of distinguishing between images generated by AI models and those originating from real-world sources.
In the literature, although there are numerous Al-generated image detectors (Wang et al., 2020; Frank et al., 2020; Ojha al., 2023; Wang al., 2023a; Zhong et al., 2023; Ricker al., 2024) and benchmarks (Wang et al., 2020; 2023a; Zhu et al., 2024; Hong & Zhang, 2024), the prevail- ing problem formulation typically involves training models on images generated solely by GANs
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
(e.g., ProGAN (Karras et al., 2018)) and evaluating their performance datasets including im- ages from various generative models, including GANs and DMs. However, such formulation poses two fundamental issues in practice: (i) evaluation benchmarks are simple, as they often feature test sets composed of random images from generative models, rather than images that present genuine challenges for human perception; (ii) confining models to train exclusively on images from certain types of generative models (GANs or DMs) imposes an unrealistic constraint, hindering the model's ability to learn from the diverse properties exhibited by more advanced generative models.
To address the aforementioned issues, we propose two pivotal strategies. Firstly, we introduce a novel testset for Al-generated image detection, named Chameleon, manually annotated to include images that genuinely challenge human perception. This dataset has three key features: (i) decep- tively real: all Al-generated images in the dataset have passed a human perception "Turing Test", Lee. human annotators have misclassified them as real images. (ii) diverse categories: compris- ing images of human, animal, object, and scene categories, the dataset depicts real-world scenar- ios across various contexts. (iii) high resolution: with most images having resolutions exceeding 720P and going up to 4K, all images in the dataset exhibit exceptional clarity. Consequently, this test set offers more realistic evaluation of model performance. After evaluating off-the-shelf Al-generated image detectors on Chameleon, unfortunately, all detectors suffer from significant performance drops, mis-classifying the Al-generated images as real ones. Secondly, we reformulate the Al-generated image detection problem setup, which enables models to train across a broader spectrum of generative models, enhancing their adaptability and robustness in real-world scenarios.
Based on the above observation, it is clear that detecting Al-generated images remains challeng- ing, and far from being solved. Therefore, fundamental question arises: what distinguishes Al- generated images from real ones? Intuitively, such cues may appear from various aspects, including low-level textures or pixel statistics (e.g., the presence of white noise during image capturing), and high-level semantics (e.g., penguins are unlikely to be appearing on the grassland in Africa), ge- ometry principle (e.g-, perspective). physics (e.g., lighting condition). To reflect such intuition, we propose a simple Al-generated image detector, termed as A AIDE (Al-generated Image DEtector with Hybrid Features). Specifically, AIDE incorporates a DCT (Ahmed et al., 1974) scoring module to capture low-level pixel statistics by extracting both high and low-frequency patches from the im- age, which are then processed through SRM (Spatial Rich Model) filters (Fridrich & Kodovsky, 2012) to characterize the noise pattern. Additionally, pre-trained OpenCLIP (Ilharco et al., 2021) to encode the entire image. The features from various levels are fused in the channel dimension for the final prediction. To evaluate the effectiveness of our model, we conduct extensive experiments on two popular benchmarks, including AIGCDetect- Benchmark (Wang al., 2020) and GenImage (Zhu et al., 2024), for Al-generated image detection. On AlGCDetectBenchmark and Genlmage benchmarks, AIDE surpasses ttte-off-the-a (SOTA) methods by +3.5% and +4.6% in accuracy scores, respectively. Moreover, AIDE also achieves competitive performance on our Chameleon benchmark.
capture global semantics,
utilize the
Overall, our contributions are summarized as follows: (i) we present the Chameleon dataset, a meticulously curated test set designed to challenge human perception by including images that de- ceptively resemble real-world scenes. With thorough evaluation of 9 different off-the-sheif detectors, this dataset exposes the limitations of existing approaches; (ii) we present a simple mixture-of-expe model, termed AIDE, that enables to discern Al-generated images based on both low-level pixel statistics and high-level semantics; (iii) experimentally, our model achieves state-of-the-art results on public benchmarks for AIGGDDetectBenccmark (Wang et al., 2020) and GenImage (Zhu et al., 2024). While on Chame it acts as competitive baseline on realistic evaluation benchmark, to foster future research in this community.
2 RELATED WORKS
Al-generated Image Detection. The demand for detecting Al-generated images has long been present. Early studies primarily focus on spatial domain cues, such as color (McCloskey & Albright, 2018), saturation (McCloskey & Albright, 2019), co-occurrence (Nataraj et al., 2019), and reflec- tions (O' brien & Farid, 2012). However, these methods often suffer from limited generalization pabilities as generators progress. To address this limitation, CNNSpot (Wang et al., 2020) discovers that image classifier trained exclusively on ProGAN (Karras et al., 2018) generator could gener-
2

---

Published as a conference paper at ICLR 2025
alize effectively to other unseen GAN architectures, with careful pre- and post-processing and data augmentation. FreDect (Frank et al., 2020) observes significant artifacts in the frequency domain of -en-ggeertted images, attributed to the upsampling operation in GAN architectures. More cent approaches have explored novel perspectives for superior generalization ability. UnivED (Ojha al., 2023) proposes to train a universal liner classifier with pretrained CLIP-VTT (Dosovitskiy et al., 2020; Radford et al., 2021) features. DIRE (Wang et al., 2023a) introduces DIRE features, which computes the difference between images and their reconstructions from pretrained ADM (Dhariwal & Nichol, 2021), to train deep classifier. PatchCraft (Zhong et al., 2023) compares rich- texture and poor-texture patches from images, extracting the inter-pixel correlation discrepancy as universal fingerprint, which is reported to achieve the state-of-the-art (SOTA) generalization per- formance. AEROBLADE (Ricker et al., 2024) proposes a training-free detection method for latent diffusion models using autoencoder reconstruction errors. FatFormer (Liu et al., 2024a) introduces forgery-aware adapter to discern and integrate local forgery traces based on CLIP. CLIPMoLE (Liu et al., 2024b) adapts combination of shared and separate LoRAs within an MoE-based structure in deeper VIT blocks. SSP (Chen et al., 2024) employs the simplest patches for detection. How- ever, these methods only discriminate real or fake images from single perspective, often failing to generalize across images from different generators.
Al-generated Image Datasets. To facilitate Al-generated image detection, many datasets contain- ing both real and fake images have been organized for training and evaluation. Early dataset from CNNSpot (Wang al., 2020) collects fake images from GAN series generators (Goodfellow al., 2014; Zhu et al., 2017; Brock et al., 2019; Karras et al., 2019). Particularly, this dataset generates fake images exclusively using ProGAN (Karras et al., 2018) as training data and evaluates the gen- eralization ability on set of GAN- testing data. However, with recent emergence of more advanced generators, such as diffusion model (DM) (Ho et al., 2020) and its variants (Dhariwal & Nichol, 2021; Nichol & Dhariwal, 2021; Rombach et al., 2022; Song et al., 2021; Liu et al., 2022b; Lu et al., 2022; Hertz et al., 2023; Nichol et al., 2022), their realistic generations make visual dif- ferences between real and fake images progressively harder to detect. Subsequently. more datasets including DM-generated images have been proposed one after another, including DE-FAKE (Xu al., 2023b), CiFAKE (Bird & Lotfi, 2024), DiffusionDB (Wang al., 2023b), ArtiFact (Rahman et al., 2023). One representative dataset is Genlmage (Zhu et al., 2024), which comprises Ima- geNet's 1,000 classes generated using SOTA generators in both academia (e.g., Stable Diffusion (Sta, 2022)) and industry (e.g., Midjourney (mid)). More recently, Hong et al. introduce a more com- prehensive dataset, WildFake (Hong& & Zhang, 2024), which includes Al-generated images sourced from multiple generators, architectures, weights, and versions. However, existing benchmarks only evaluate Al-generated images using current foundational models with simple prompts and few mod- ifications, whereas deceptively real images from online communities usually necessitate hundreds to thousands of manual parameter adjustments.
CHAMELEON DATASET
3.1 PROBLEM FORMULATION
In this paper, our goal is to train a model that can distinguish the Al-generated images from the ones captured by the camera, i.e., y moniee (I; +) E {0, 1), where I I Hxxx denotes an input RGB image, θ refers to the learnable parameters. For training and testing, we consider the following two settings:
Train-Test Setting-I. In the literature, existing works on detecting Al-generated images (Wang al., 2020; Frank al., 2020; Ojha et al., 2023; Wang et al., 2023a; Zhong et al., 2023) have exclusively considered the scenario of training on images from single generative model, for example, ProGAN (Karras al., 2018), or Stable Diffusion (Sta, 2022), and then evaluated on images from various generative models. That is,
Grain = GGAN 9DM; Guss = (SPIGAA, GcycleGAN, ..., GSD. MMidjoumey)
Generally speaking, such problem formulation poses two critical issues: (i) evaluation benchmarks are simple, as these randomly sampled images from generative models, can be far from being photo- realistic, shown in Figure 1; (ii) confining models to train exclusively GAN- IAN-generated images imposes an unrealistic constraint, hindering the model's 's ability to learn from the diverse properties exhibited by more advanced generative models.
3

---

Published as a conference paper at ICLR 2025
Train
Tox
Trais
Test
LFGeemmeer RNl
boGAN
PriZAN
OykiAAN
HighAN
S0.14
SD:14
Maxemny
(a) AIGCDetect Denchmark
(b) Genimage Benchmark
(6) Chamnleoon Testset
Comparison of Chameleon with existing benchmarks. We visualize two contemporary Al- generated image benchmarks, namely (a) AIGCDetect Benchmark Wang et (2020) and (b) GenImage Benchmark Zhu et al. (2024), where all images are generated from publicly available generators, such as Pro- GAN (GAN-based), SD v1.4 (DM-based) and Midjournyy (commercial API). These images are generated by unconditional situations conditioned on simple prompts (e.g., photo of a plane) without delicate manual ad- justments, therehy inclined generate obvious artifacts in consistency and semantics (marked with red boxes). contrast, our Chameleon dataset in (c) aims simulate real col-world scenarios by collecting diverse images from online websites, where these online images are carefully adjusted by photographers and Al artists.
Figure 1:
Table 1: Statistics of the Chameleon testset, including over 1k high-fidelity Al-generated images from art; civ; lib, as well as similar scale of real-world photographs from uns.
Scene
Object
Animal
Human
Total
Real Images
3,574
3,578
3,998
3,713
14,863
Fake Images
2,976
2,016
313
5,865
11,170
Trran-Test Setting-II. Herein, we propose an alternative problem formulation, where the models allowed to train on images generated from a wide spectrum of generative models, and then tested on images that are genuinely challenging for human perception.
Giain (GGAN, GOM), Grest [Dehameleon
(2)
Dchameloon refers to our proposed benchmark, as detailed below. We believe this setting resembles more practical scenario for future model development in this community.
3.2 CHAMELEON DATASET
The primary objective of the Chameleon dataset is to evaluate the generalization and robustness of existing Al-generated image detectors, for a sanity check on the progress Al-generated image detection. In this section, we outline the progression of the proposed dataset, including: (i) collec- tion, (ii) curation, (iii) annotation. The statistical results our dataset are illustrated in Table and compare our dataset with existing benchmarks in Fig. 1.
3.2.1 DATASET COLLECTION
To simulate the real-world scenarios in detecting Al Al-generated images, we structure Chameleon dataset based on three main principles: (i) images must be deceptively real, and (ii) they should cover diverse range of categories, and (iii) they should also have very high image quality. Impor- tantly, each image must have (a) Creative Commons (CC BY 4.0) license, or (b) explicit permissions obtained from the owners
in our research. Herein, we present the details of image collection.
To collect images that are deceptively real, and cover sufficiently diverse categories, source user-created Al-generated images from popular Al Al-painting communities, i.e., ArtStation (art), Civitai (civ), and Liblib (lib), many of which utilize commercial APIs (e.g., Mid- journey (mid) and DALLE-3 (Ramesh al., 2022)) various LoRA modules (Hu et al., 2022) with
Fake Image Collection:
4

---

Published as a conference paper at ICLR 2025
Table Comparison Al-generated image detection testset. Our Chameleon dataset is first to encom- pass real-life scenarios for evaluation. Compared to AIGCDetectBememmar (Zhong et al., 2023), Chameleon offers greater magnitude and superior quality, rendering it more realistic in evaluation. IN represents ImageNet.
PrIGAN
StyEGAN
BiJCAN
CHEGAN
StarGAN
GaIGAN
SpAeGAN2
WFR
40M
CRAc
Mijouneyy
5D
vD
VQDM
Wallogg
DALLES
Chome
zonn
Magnittude
8.08
12.0c
4.06
2.6k
4.08
10.0N
15.9%
2004
12.0
12.06
12.08
12.06
16.08
12.04
12.04
2,04
26.0/
Resolution
256
1024
256
512
256
720P-4KK
Variety
LSUN
LUSN
IN
CekbA
COCO
LSUN
FF3Q
IN
IN
IN
IN
IN
IN
IN
Real-life
Stable Diffusion (SD) (Sta, 2022) that fine-tuned on their in-house data. Specifically, we initiate the process by utilizing GPT-4 (cha, 2022) to generate diverse query words to retrieve Al-generated images. Throughout the collection process, we enforce stringent NSFW (Not Safe For Work) re- strictions. Ultimately, our collection comprises over 150K Al-generated images.
Real Image Collection: To ensure that real and Al Al-generated images fall into the same distribution, employ identical query words to retrieve real images, mirroring the approach used for gathering Al-generated images. Eventually, we collect over 20K images from platforms like Unsplash (uns), which is an online community providing high-quality, free-to-use images contributed by photogra- phers worldwide.
3.2.2 DATASET CURATION
To ensure the collection of high-quality images, we implement a comprehensive pipeline for image cleaning: (i) we discard images with resolution lower than 448 x 448, as higher-resolution images generally provide better assessments of the robustness of existing models; (ii) due to the potential presence of violent and inappropriate content, we utilize SD's safety checker model (saf, 2022) to filter NSFW images; (iii) to avoid duplicated images, we compare their hash values to filter duplicated images. In addition to this general cleaning pipeline, we introduce CLIP (Radford et al., 2021) to further filter out images with low image-text similarity. Specifically, for Al-generated im- ages, the online website provides prompts used to generate these images, and we calculate similarity using their corresponding prompts. For real images, we used the mean of the 80 prompt templates (e.g., photo of [category) and a photo of the {category]) evaluated in CLIP's ImageNet zero-shot as the text embedding.
3.2.3 DATASET ANNOTATION
At this stage, we establish an annotation platform and recruit 20 human workers to manually label each of the Al-generated images for their category and realism. For categorization, annotators are instructed to assign each image to one of four major categories: human, animal, object, and scene. Regarding realism assessment, workers are tasked with labeling the images as either Real or AI- generated, based on the criterion of whether this image could be taken with camera. Note that, the annotators are not informed whether the images are generated by Al algorithms beforehand. Each image assessed independently by two annotators, and those have been misclassified as real by both annotators can thus be considered to pass the "perception turing test" and labeled as highly realistic. Subsequently, we retain only those images judged as highly realistic. Similarly, for real images, we follow the same procedure, retaining only those belonging to the four predefined categories, as we have done for Al-generated images.
3.2.4 DATASET COMPARISON
Our objective is to construct a sophisticated and exhaustive test set that serves as a valuable exten- sion to the current evaluation benchmarks for Al-generated image detection. Table 2, we conduct a comparative analysis between our Chameleon dataset and existing test sets. Our dataset is char- acterized by three pivotal features: (i) Magnitude. Comprised of approximately 26,000 test images, the Chameleon dataset represents the most extensive collection available, surpassing any existing test set and enhancing its robustness. (ii) Variety. Our dataset incorporates images from a vast array of real-world scenarios, surpassing the limited categorical focus of the other datasets. (iii) Resolu- tion. With resolutions spanning from 720P to 4K, With image resolutions ranging from 720P to 4K, artifacts demand more nuanced analysis, thus presenting additional challenges to the model due to
5

---

Published as a conference paper at ICLR 2025
Patchwise Feature Extraction
Petch Selection DCT Scoring
Presect
SRM
ResNet50
Huhoit
Conv
Susiet
SRM
ResNet50
Lewest
Conv
BCT Scaring
MLP Head
Charnel
Score
putckas
Concetonata
Semantic Feature Embedding
Input image
CLIP- ConvNeXt
Semontic
Project
feature
Layer
Figure 2: Overview of AIDE. It consists of a Patchwise Feature Extraction (PFE) module and Semantic Fea- ture Embedding (SFE) module in mixture of experts manner. In PFE module, the DCT Scoring module first calculates the DCT coefficients for each smashed patch and then performs weighted sum of these coefficients (weights gradually increase as the color goes from light dark).
the need for fine-grained discernment, In summary, our dataset offers a more demanding and prag- matically relevant benchmark for the advancement of Al-generated image detection methodologies.
4 METHODOL.OGY
In this section, we present AIDE (Al-generated Image DEtector with Hybrid Features), consisting of module to compute low-level statistics for texture smooth patches, a high-level semantic embedding module, and discriminator to classify the image as being generated or photographed. The overview of our AIDE model is illustrated in Fig. 2.
4.1 PATCHWISE FEATURE EXTRACTION
Here, our design leverages the disparities in low-level patch statistics between Al-generated images and real-world scenes. Models like generative adversarial networks or diffusion models often yield images with certain artifacts, such excessive smoothness or anti-aliasing effects. To capture such discrepancy, we adopt a Discrete Cosine Transform (DCT) score module identify patches with the highest and lowest frequency. By focusing these extreme patches, aim highlight the distinctive characteristics Al-generated images, thus facilitating the discriminative power of our detection system.
Patch Selection via DCT Scoring. For an RGB image, we first divide this image into multiple patches with fixed window size, I {21, .2..... Fn). H; NxN33 In case, the patch size is set to be = 32 pixels. We apply the discrete cosine transform to each of the image patches, obtaining the corresponding results in the frequency domain, Mg {z.ct x2 ...,, ddct). adlet E NxN33
To acquire the highest and lowest image patches, we use the complexity of the frequency components indicator. From this, design simple yet effective scoring mechanism, using K different band-pass filters:
1 if 2N K k <
2N K
(k +
Fk.i = otherwise
(3)
0.
where Fe,ay is the weight at the (i,j) position of the k-th filter. Next, for the m-th patch (x xNNxN33, we apply the filters FRaj Nx multiply the logarithm of the absolute DCT
coefficients dat E Nx33, and sum up all the positions to obtain the grade of the patch Gm We formulated it as
N-IN-1
m
2
ΣΣΣ
Faas log( 4ct
(4)

---

Published as a conference paper at ICLR 2025
where c is the number of patch channels. In this way, acquire the grades of all patches G {G', ,....., Gm We then sort them to identify the highest and lowest frequency patches.
Through the scoring module, with the highest frequency and the top & patches Xmin {Xming X Xmin, with the lowest frequency, where max, E N x3 X NxNx3.
obtain the top & patches Xmax {Xmaxx Xmax, .... X Xmax
Patchwise Feature Encoder.
These patches are resized to size of 256 x 256 pixels, and input into the SRM (Fridrich & Kodovsky, 2012) to extract their noise pattern. Lastly, these features are further input into two ResNet-50 (He et al., 2016) networks (/i (-) and (-) to obtain the final feature map Fmax (Xfi(max,)) ); Ji (Xmaxz maxg
--: (X Maxx) Fmin (/2 (X min: (2(minn f2( (mm,m,))))
We acquire the highest and lowest frequency embedding on the mean- mecan-pooled feature:
Faax Mean(AveragePooll Fmux)),
Fain Mean(AveragePooue (Fmin))
(5)
4.2 SEMANTIC FEATURE EMBEDDING
To capture the rich semantic features within images, such as object co-oceurrence and contextual re- lationships, we compute the visual embedding for input image with an off-the-shelf risual-language foundation model. Specifically, we adopt the ConvNeXt-based OpenCLIP model (Ilharco et al., 2021) to get the final feature map (v E R4x To capture the global contexts, we append linear projection layer followed by mean spatial pooling:
F, = avgpool(g(1)) (g))).
(6)
4.3 DISCRIMINATOR
To distinguish between AI- Al-generated images and real images, we utilize mixture-expert model for the final discrimination. At low-level, we take the average of the highest frequency featured:
Iusan avgpool(Fmax Fmin)
(7)
Then, channel-wisely concatenate the representations between it and high-level embedding F,- At last, the features are encoded into MLP to acquire the score,
= f [[avggooll(Fmenn Fsl)
(8)
where f()) denotes the MLP consisting of a linear layer, GELU (Hendrycks & Gimpel, 2016) and classifier, refers to the operation of channel-wise concatenation.
5 EXPERIMENTS
5.1 EXPERIMENTAL DETAILS
Baseline Detectors. We evaluate 9 off-the-shelf detectors including CNNSpot (Wang et al., 2020), FreDect (Frank al., 2020), Fusing (Ju et al., 2022), LNP (Liu et al., 2022a), LGrad (Tan et al., 2023), UnivFD (Ojha et al., 2023), DIRE (Wang et al., 2023a), PatchCraft (Zhong et al., 2023) and NPR (Tan al., 2024) for comparison.
Datasets. To comprehensively evaluate the generalization ability of existing approaches, we con- duct experiments across two kinds of settings: Setting-I and Sefting-II, which summarized in Sec. 3.1. For the Setting-I setting, we evaluate the detectors on two general and comprehensive benchmarks of AlGCDeetecBeenchmark (B,) (Zhong al., 2023) and GenImage (Zhu et al., 2024) (B2). For the Setting-II setting, we evaluate the detectors on our Chameleon (Bg) benchmark. More details can be found in Appendix.
Implementation Details. AIDE includes two key modules: Patchwise Feature Extraction (PFE) and Semantic Feature Embedding (SFE). For PFE channel, we first patchify each image into patches and the patch size is set to be N = 32 pixels. Then these patches are sorted using our DCT Scoring module with K = 6 different band-pas pass filters in the frequency domain. Subsequently, we select and two lowest-frqueene
patches using the calculated DCT scores. These selected patches are then resized to 256 x 256 and extracted their noise pattern using SRM (Fridrich & Kodovsky, 2012). For SFE channel,
two
the pre-trained OpenCIIP (Ilharco al., 2021) to
7

---

Published as a conference paper at ICLR 2025
Table 3: Comparison on the AlGCDetccBennhhaakk (Zhong et al., 2023). Accuracy (%) of different detec- tors (rows) in detecting real and fake images from different generators (columns). DIRE-D indicates this result comes from DIRE detector trained over fake images generated by ADM following its official setup (Wang et al., 2023a). DIRE- G indicates this baseline is trained on the same ProGAN training data others. The best result and the second-best result are marked in bold and underline, respectively.
POGAN
SYYCAN
BgGAN
CooeGAN
SIGGAN
CaNDAN
StyeGAN2
WFRR
ADM
Clide
Midjournyy
v1.4
50 v1.5
VODM
Wakong
DALLE2
Method
5D
CNNSpot
100.00
90.17
71.17
87
94.60
81.42
86.91
91.65
60.39
58.07
11.39
50.57
50.53
56.4
56.46
51.03
50.45
70.78
FreDect
99.36
78.02
81.97
78.77
94.62
80.57
66.19
50.75
63.42
54.13
45.87
38.79
39.21
77.80
40.30
34,70
64.03
Fusing
100.00
85.20
77.40
87.00
97.00
77.00
83.30
66.80
49.00
$7.20
52.20
$1.00
$1.40
55.10
$1.70
52.90
68.38
LXP
99.67
91.75
77.75
84. 10
99.92
75.39
94.64
70.85
84.73
80.52
65.55 58
85.55
85.67
74,46
82.06
88.75
81.84
LGrad
99.83
91 08
85 62
86.94
99.27
78.46
85.32
55.70
70
67.15
66.
65. 35
63.02
63.67
72.99
59.55
63.45
75. 75.34
UnivFD)
84.93
99.47
74.06 96
86.90
66.87
63.49
70.93
78.43
DIRE-G
95.47
71.75
54.46
DIREED
49.70
49.58
51.23
51.72
91.24
91.
90.90
92.45
PatchCraft
100.00
97.77
95.80
70.17
99.97
71.58
89.55
85.00
87.17
83.79
90.13
95.38
95.30
88.91
91.07
96.60
89.31
NPR
99.79
97 70
84.35
84.
96 10
99 35
82.50
98.38
65 80
69.09
78 36
77.85
78.63
78.89
78.13
76.1
64.90
82.91
AIDE
99.99
99.64
83.95
98.48
99.91
73.25
98.00
94,20
93.43
95.09
77.20
93.00
92.85
95.16
93.55
96.60
92.77
Table 4: Comparison on the Genlmage (Zhu et al,, 2024). Accuracy (%) of different detectors (rows) in detecting real and fake images from different generators (columns). These methods are trained on real images from ImageNet and fake images generated by SD v1.4 and evaluated over eight generators. The best result and the second-hest result are marked in bold and underline, respectively.
Method
Midjourney
SD v1.4
SD v1.5
ADM
GLIDE
Wukong
VQDM
BigGAN
Mean
ResNet-50 (He et al., 2016)
54,90
99.90
99.70
53.50
61.90
98.20
56.60
52.00
72.09
DeiT-S (Touvron
2021)
55.60
99.90
99.80
49,80
58.
98.90
56.90
53.50
71.56
Swin-T (Liu et al., 2021)
62.10
99.90
99.80
49.80
67.60
99.10
62.30
57.60
74.78
CNNSpot (Wang et 2020)
52.80
96.30
95.90
50.
39.80
78.60
$3.40
64.21
Spec (Zhang al., 2019)
52.00
99.40
99.20
49.70
49.80
94.80
55.60
49,80
68,79
F3Net (Qian al, 2020)
50.10
99.90
99.90
49. 90
50.00
99.90
49,90
49.90 90
68.69
GramiNet (Liu et al., 2020)
54.20
99.20
99.10
$0.30
$4.60
50.80
$1.70
69.85
DIRE (Wang et 20123a)
2023a)
60.20
99,90
94.80
50.90
55.00
99.20
50.10
50,20
70.66
UnivED (Ojha et al., 2023)
73.20
84.20
84.00
55.20
76.90
75.60
56.90
80.30
73.29
GenDet (Zhu et al., 2023)
89.60
96.10
96.10
58.00
78.40
92.80
66.50
75.00
81.56
PatchCraft (Zhong et al., 2023)
79.00
89.50
89.30
77 30
78.40
89.30
83.70
72.40
82,30
AIDE
79.38
99.74
99.76
78.54
91.82
98.65
80.26
extract semantic features. We adopt data augmentations including random JPEG compression (QF Un0form(30, 100)) and random Gaussian blur (a Uniform 3.0)) to improve the robustness of detectors. Each augmentation conducted with 10% probability. During the training phase, we use AdamW optimizer with the learning rate of 1 x 10-4 in B₁ and 5 x 10-4 in B2, respectively. The batch size is set to 32 and the model is trained on NVIDIA A100 GPUs for only 5 epochs. Our method trains very quickly, only hours are sufficient.
Metrics. In accordance with existing Al-generated detection approaches Wang et al. (2020; 2019); Zhou al. (2018), we report both classification accuracy (Ace) and average precision (AP) in our experiments. All results are averaged over both real and Al-generated images unless otherwise specified. We primarily report Acc for evaluation and comparison in the main paper, and AP results are presented in the Appendix.
5.2 COMPARISON STATE-OF- MODELS
On Benchmark AIGCDeteetBechhmrkk The quantitative results in Table 3 present the classi- fication accuracies of various methods and generators within Br. In this evaluation, all methods, except for DIRE-D, were exclusively trained on generrtee data.
AIDE demonstrates a significant advancement over the existing tfate-of-the-ar (SOTA) approach, for example, PatchCraft, achieving an average accuracy increase of 3.5%. UnivFD utilizes CLIP semantic features for detecting Al-generated images, proving effective for GAN-generated images. However, shows pronounced performance degradation with diffusion model (DM)-generated im- ages. This suggests that as generation quality improves, diffusion models produce images with fewer semantic artifacts, as depicted in Fig. (a). Our approach, which integrates semantic, low- frequeney, and frghffregeency information at the feature level, enhances detection performance, yielding a 5.2% increase for Gasebased images and a 1.7% increase for DM-based images com- pared to the SOTA method.
8

---

Published as a conference paper at ICLR 2025
Table 5: Comparison the Chameleon. Accuracy (%) of different detectors (rows) in detecting real and fake images of Chameleon. For each training dataset, the first row indicates the Ace evaluated on the Chameleon testset, and the second row gives the Acc for "fake image / real image" for detailed analysis.
Training Dataset
CNNSpat
FreDeet
Fusing
GramNet
LNP
UniveD
DIRE
PatchCraft
NPR
AIDE
ProGAN
0.0999.57
$6.94
13.7287.122
55.62
0.01/99.79
56.98
4.76/99.66
58,94
0.0999.97
57.11
3,1897.83
57.22
3,25/9.488
58.19
1.899282
53.76
2.2099.70
57,29
5.0498.46
58.37
SD v1.4
8698.53
60.11
1.37/98.57
56.86
0.0099.96 96
$7.07
17.6991.50
0.5797.01
74.97/41.99
56.62
11.669567
3.1796.3 35
2,43/100.00
58.13
20.3344 38
GenImage
60.89
57.22
$7.09
57.83
55,70
$7.81
9,80/99.25
0.89/99.55
0.02/99.96 98
8.23/98.58
7,7296.20
85.52/41.56
2.09/99.73
1.3996.52
1.68/100.00
26,80955.0
Table 6: Robustness on JPEG compression and Gaussian blur of AIDE. The accuracy (%) averaged over 16 test sets in B3 with specific perturbation.
Table 7: Ablation studies on PFE and SFE modules.
Method
Original
(F-955
QF-50
Compression
011775
(1--50
PFE-11
Module PFE-L
SFE
Meaa
CNNSpot
Fising
FreDhet
64.03
66.90 95
61.39
15.75
65-48
69:
x
/
X
75.24 x
76.09
LNP
K3.84
53.58
54.09
$3.02
$2.85
67.91
65.42
65.2
$2.69
x
x
75.26
LGrad
PatchCrraft
DIREGG
LaivPD
68.68
89.31
78.43
75.34
66.49
51.55
73.48
74.10
66.12
$1.39
71.41
74.01
65.24
69:52
50.00
69:43
64.34
50.00
58.05
67.78
70.31
64.00
75.90
71:73
89.12
63.09
68.29
24.90
64.02
62.21
68:43
73.53
61.18
61.91
56,22
72.28
/
x
/
/
x
84.20
76.70
80.69
AIDE:
92.77
75.54
74.21
70.64
69.60
$1.88
80.35
$0.85
79.86
92.77
On Benchmark Genlmage: In the experiments conducted B2, all models were trained on SD v1.4 and evaluated across eight contemporary generators. Table 4 presents the results, illustrating method's superior performance over the current state-of-the-art, PatchCraft, with 4.6% im- provement in average accuracy. The architectural similarities between SD v1.5, Wukong, and SD vl.4, noted by GenImage (Zhu et al., 2024), enable models achieve near-perfect accuracy, approaching I 100% on such datasets. Thus, evaluating generalization performance across other gen- erators, such as Midjourney, ADM, and Glide, becomes essential. Our model demonstrates either the best or second- best performance in these cases, achieving an average accuracy of 86.88%
On Benchmark Chameleon: As highlighted Sec. 1, we contend that success on existing pub- lic benchmarks may not accurately reflect real-world scenarios or the advancement in Al-generated image detection, given that test sets are typically randomly sampled from generative models with- out "Turing Test". To address potential biases related to training setups-succ suuph as generator types and image categories-ve evaluate the performance conditions. Despite their high performance on existing benchmarks, as depicted in Fig. 8, the state- of-the-art detector, PatchCraft, experiences substantial performance declines. Additionally, Table reveals significant performance decreases across all methods, with most struggling to surpass an average accuracy close to random guessing (about 50%), indicating a failure in these contexts.
existing detectors under diverse training
While our method achieves the SOTA results on available datasets, its performance Chameles remains low, especially on discovering the Al-generated images. This underscores that our dataset, Chameleon, which challenges human perception, is also extremely difficult Al models.
5.3 ROBUSTNESS TO UNSEEN PERTURBATIONS
In real-world scenarios, images inevitably encounter unseen perturbations in transmission and inter- action, complicating Al-generated image detection. Herein, we assess the performance of various methods in handling potential perturbations, such as JPEG compression (Quality Factor (QF) = 95, 90, 75, 50) and Gaussian blur (a 1.0, 2.0, 3.0, 4.0). As illustrated in Table all methods exhibit a decline in performance due to disruptions in the pixel distribution. These disruptions diminish the discriminative artifacts left by generative models, complicating the differentiation between real and Al-generated images. Consequently, the robustness of these detectors in identifying Al-generated images is significantly compromised. Despite these challenging conditions, our method consistently outperforms others, maintaining relatively higher accuracy in detecting Al- Al-generated images. This superior performance is attributed to our model's 's ability to effectively capture and leverage multi- perspective features, semantics, and noise, even when the pixel distribution is distorted.

---

Published as a conference paper at ICLR 2025
Emeat wit 12 grgeeese ALS Acgonand
Table Performance SOTA method, PatchCraft Zhong al. (2023), under B1 (left), B2 (right), and our Chameleon testset. The boundary line for Acc 50% is marked with dashed line.
Table Visualization the effectiveness of PFE and SFE modules. Without SFE, Al-generated images exhibit pronounced semantic errors that are incorrectly classified as real. Similarly, without PFE, many Al-generated images, despite lacking semantic errors, contain subtle underlying noise that also leads their misclassification as true.
5.4 ABLATION STUDIES
Our method focuses on detecting AI Al-generated images with mixture of experts, namely patchwise feature extraction (PFE-H and PFE-L for high-frequency and low-frequency patches, respectively) and semantic feature extraction (SFE). These modules collectively contribute to comprehensively identifying AI- Al-generated images from different perspectives. Herein, conduct extensive experi- ments to investigate the roles of each module on Bi.
Patchwise Feature Extraction. As shown in Table 7, removing either the high-frequency or the low-frequency patches results obvious performance degradation in terms accuracy. Without the high-frequent patches, the proposed method is unable to discern that the high- gegh-freuenccy regions of Al-generated images are smoother than those real images, resulting in performance degrada- tion. Similarly, without the low-frequency patches, the method cannot extract the underlying noise information, which is crucial for identifying Al-generated images with higher fidelity, leading to incorrect predictions.
Semantic Feature Extraction. The performance degrades significantly (76. vs 92.77% when we remove the semantic branch, shown in Table 7. Intuitively, if the branch for semantic informa- tion extraction is absent, our method struggles to effectively capture images with semantic artifacts, resulting in significant drops.
Visualization. To vividly demonstrate the effectiveness of our modules patchwise feature extraction (PFE) and semantic feature extraction (SFE), we conducted visualization, as depicted in Fig. In the first row, the absence of semantic feature extraction results in many images with evident semantic errors going undetected. Similarly, the second row shows that, without patchwise feature extraction, numerous images lacking semantic errors still contain differing underlying information that remains unrecognized. Overall, our method, AIDE, achieves the best performance.
CONCLUSION
In this paper, we have conducted a sanity check on detecting Al-generated images. Specifically, we re-examined the unreasonable assumption in existing training and testing settings and suggested new ones. terms of benchmarks, we propose a novel, challenging benchmark, termed as Chameleon, which is manually annotated to challenge human perception. We evaluate off-the-shelf models and demonstrate that all detectors suffered from significant performance declines. In terms of architec- ture, we propose simple yet effective detector, AIDE, that simultaneously incorporates low-level patch statistics and high-level semantics for Al-generated image detection. Despite our approach demonstrates state-of-the-ee
performance existing AJGGCDetecIBBenchmnrk (Zhong al., 2023)
and Genlmage (Zhu et al., 2024)) and our proposed benchmark (Chameleon) compared to previ- ous detectors, it leaves significant room for future improvement.
ACKNOWLEDGEMENTS
WX would like acknowledge the National Key R&D Program of China (No. 2022//0161400)
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Artstation. htt.:s//ww..rrtsta..oo
Civitai. https://civitai..om
Liblib. https:/ww...blii..ee
Unsplash. https://unsplash.com
Stable diffusion.
com/ http://...ootttttttt..aa.. http://hhhbbb.///aaaaaalaaaaaa/a fuusion 2022.
Chatgpt. https://chatgpiee 2022.
Stable diffusion safety
checker.
httpss//uuggiiggfcceooo///// 2022.
stable-di stablee-diffssiinn
whichfaceisreal. http:///p///////iiiicaacaaaaaaaa
2023.
https://x.he..mmmddsspooo. http://poore..mmoooolooooo////// 2023.
Nasir Ahmed, T- Natarajan, and Kamisetty Rao. Discrete cosine transform. IEEE Transactions
on Computers, 100(1):90-93, 1974.
Jordan J Bird and Ahmad Lotfi. Cifake: Image classification and explainable identification of ai- generated synthetic images. IEEE Access, 2024.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale GAN training for high fidelity natural image synthesis. In International Conference Learning Representations, 2019. URL net/ httpss:/cppereviice wwwp:/eeeeeooteeeeeeeeeeeom. Forum?id=Blxssr
Jiaxuan Chen, Jieteng Yao, and Li Niu, A single simple patch is all you need for ai-generated image detection. arXiv preprint arXiv:2402.011233, 2024.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Woo Ha. Sunghun Kim, and Jacgul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-image translation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 8789-
8797, 2018
2018.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Fei-Fei. Imagenet: A large-scale hier- archical image database. In 2009 IEEE Conference Computer Vision and Pattern Recognition, pp. 248-255. Icee, 2009.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems, 14:8780-8794 2021.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition scale. International Confer- ence on Learning Representations, 2020.
William D Ferreira, Cristiane BR Ferreira, Gelson da Cruz Júnior, and Fabrizzio Soares. A review of digital image forensics. Computers Electrical Engineering, 15:106 2020.
Joel Frank, Thorsten Fisenhofer, Lea Schönherr, Asja Fischer, Dorothea Kolossa, and Thorsten Holz. Leveraging frequency analysis for deep fake image recognition. In International Confer- ence on Machine Learning, pp. 3247-3258. PMLR, 2020.
Jessica Fridrich and Jan Kodovsky. Rich models for steganalysis of digital images. IEEE Transac- tions Information Forensics and Security, 7(3):868 882,,882-11 2012.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farles Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in Neural Informa- fion Processing Systems, 27, 2014.

---

Published as a conference paper at ICLR 2025
Shuyang Gu, Dong Chen, Jianmin Bao, Fang Wen, Bo Zhang, Dongdong Chen, Lu Yuan, and Baining Guo. Vector quantized diffusion model for text-to -itatt synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10696-10706, 2022.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp- 770-778, 2016.
Dan Hendrycks and Kevin Gimpel. Gaussian error linear units (gelus). arXiv preprint 1eXiv:6606.84// 2016.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Prompt-to-prompt image editing with cross attention control. 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neural Information Processing Systems, 33:6840-6851, 2020.
Yan Hong and Jianfu Zhang. Wildfake: A large-scale challenging dataset for ai-generated images detection, arXiv preprint MXXi:2442.1/844 2024.
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhe -hu,, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In International Con- ference Learning Representations, 2022.
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip. July 2021. URL https htt///ci.ooo/ arooioorg/000 5281/ zer.odo.. 5 5143773, If you use this software, please cite it as below.
Yan Ju, Shan Jia. Lipeng Ke. Hongfei Xue, Koki Nagano, and Siwei Lyu. Fusing global and local features for generalized ai-synthesized image detection. 2022 IEEE International Conference Image Processing (ICIP), pp. 14655-3469 IEEE, 2022.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of GANs for im- proved quality, stability, and variation. In International Conference on Learning Representations, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IÉEF/CVF Conference Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Mika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 81 8110-8119, 2020.
Tsung-Y Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Bo Liu, Fan Yang, Xiuli Bi, Bin Xiao, Weisheng Li, and Xinbo Gao. Detecting generated images by real images. In European Conference on Computer Vision, pp. 95-110. Springer, 2022a.
Huan Liu, Zichang Tan, Chuangchuang Tan, Yunchao Wei, Jingdong Wang, and Yao Zhao. Forgery- aware adaptive transformer for generalizable synthetic image detection. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 10770-10780, 2024a.
Luping Liu, Yi Ren, Zhijie Lin. and Zhou Zhao. Pseudo numerical methods for diffusion models on manifolds. In International Conference on Learning Representations, 2022b.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IFFF/CVF International Conference Computer Vision pp. 10012-10022, 2021.
12

---

Published as a conference paper at ICLR 2025
Zhengzhe Liu, Xiaojuan Qi, and Philip HS Tomr. Global texture enhancement for fake face detec- tion in the wild. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8060-8069, 2020.
Zihan Liu, Hanyi Wang, Yaoyu Kang, and Shilin Wang. Mixture of low-rank experts for transferable ai-generated image detection. arXiv preprint arXiv:2404. 204.:22404.04888,, 2024b.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE International Conference on Computer Vision, pp. 3730-3738, 2015.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for difflusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 35:5775-5787, 2022.
Scott McCloskey and Michael Albright. Detecting gan-generated imagery using color cues. arXiv preprint arXiv:1812.0022 08244777 2018.
Scott McCloskey and Michael Albright. Detecting gan gan-generated imagery using saturation cues. In 2019 IEEE International Conference on Image Processing (ICIP), pp. 4584 4588. IEEE, 2019.
Lakshmanan Nataraj, Tajuddin Manhar Mohammed, Shivkumar Chandrasekaran, Arjuna Flenner, Jawadul H Bappy, Amit Roy-Chowdhury, and BS Manjunath. Detecting gan generated fake images using CO- co-occurrence matrices. arXiv preprint arXiv: 1903. 008366 2019.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusion probabilistic models. In International Conference on Machine Learning, pp. 8162-8171. PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Mcgrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. In International Conference Machine Learning, pp- 16784-16804. PMLR, 2022.
James F O'brien and Hany Farid. Exposing photo manipulation with inconsistent reflections. ACM Trans. Graph., 31()):4-1, 2012.
Utkarsh Ojha, Yuheng Li, and Yong Jae Lee. Towards universal fake image detectors that generalize across generative models. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 24480-24489, 2023.
Taesung Park, Ming-Yu Yu Liu, Ting-Chun Wang, and Jun-Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In Proceedings of the IEEE/CVE Conference on Computer Vi- sion and Pattern Recognition, pp. 2337-2346, 2019.
Yuyang Qian, Guojun Yin, Lu Sheng, Zixuan Chen, and Jing Shao. Thinking in frequency: Face forgery detection by mining frequency- aware clues. In European Conference on Computer Vision, pp. 86-103. Springer, 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Md Awsafur Rahman, Bishmoy Paul, Najibul Haque Sarker, Zaber Ibn Abdul Hakim, and Shaikh Anowarul Fattah. Artifact: A large-scale dataset with artificial and factual images for generalizable and robust synthetic image detection. In 2023 IEEE International Conference on Image Processing (ICIP), pp. 2200-2204 IEEE, 2023.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Jie Ren, Han Xu, Pengfei He, Yingqian Cui, Shenglai Zeng, Jiankun Zhang, Hongzhi Wen, Jiayuan Ding, Hui Liu, Yi Chang, et al. Copyright protection in generative ai: A technical perspective. arXiv preprint a0X 2024.
13

---

Published as a conference paper at ICLR 2025
Jonas Ricker, Denis Lukovnikov, and Asja Fischer. Aeroblade: Training-fre free detection of latent diffusion images using autoencoder reconstruction error. In Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 9130-9144 2024.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, and Yunchao Wei. Learning gradients: Generalized artifacts representation for gan-generated images detection. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 12105-12114, 2023.
Chuangchuang Tan, Yao Zhao, Shikui Wei, Guanghua Gu, Ping Liu, and Yunchao Wei. Rethinking the up-sampling operations in enn-based generative network for generalizable deepfake detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp-
18130-28139) 2024.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference Machine Learning, pp. 10347-10357. PMLR, 2021.
Sheng-Yo Yu Wang, Oliver Wang. Andrew Owens, Richard Zhang, and Alexei Efros. Detecting photoshopped faces by scripting photoshop. In Proceedings the IEEE/CVF International Con- ference on Computer Vision, pp. 10072-10081, 2019.
Sheng-Y Yu Wang, Oliver Wang, Richard Zhang, Andrew Owens, and Alexei A Efros. Con-generated images are surprisingly easy to spot... for now. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8695-87088 8704, 2020.
Zhendong Wang, Jianmin Bao, Wengang Zhou, Weilun Wang, Hezhen Hu, Hong Chen, and Houqiang Li. Dire for diffusion diffusion-generated image detection. In Proceedings of the IEEE/CVF International Conference Computer Vision, pp.
2023a.
ZijieJ J Wang, Evan Montoya, David Munechika, Haoyang Yang, Benjamin Hoover, and Duen Horng Chau. Diffusiondh: A large-scale prompt gallery dataset for text-to-image generative models. In Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 19,-911,72 2023b.
Danni Xu, Shaojing Fan, and Mohan Kankanhalli. Combating misinformation in the era of gener- ative ai models. Proceedings of the 31st ACM International Conference on Multimedia, pp. 9291-9298, 2023a.
Qiang Xu, Hao Wang, Laijin Meng, Zhongjie Mi, Jianye Yuan, and Hong Yan. Exposing fake images generated by text-to-image diffusion models. Pattern Recognition Letters, 176:76-82, 2023b.
Fisher Yu, Ari Seff, Yinda Zhang, Shuran Song, Thomas Funkhouser, and Jianxiong Xiao. Lsun: Construction of large-scale image dataset using deep learning with humans in the loop. arXiv preprint arXiv:/506. 03365, 2015.
Xu Zhang, Svebor Karaman, and Shih-Fu Chang. Detecting and simulating artifacts in gan fake images. In 2019 JEEE international workshop on forennsis and security (WIFS), pp- 1-6. IEFE, 2019.
14

---

Published as a conference paper at ICLR 2025
Nan Zhong, Yiran Xu, Zhenxing Qian, and Xinpeng Zhang. Rich and poor texture contrast: A simple yet effective approach for ai- generated image detection. arXiv preprint arXiv:2311.12397, 2023.
Peng Zhou, Xintong Han, Vlad I Morariu, and Larry S Davis. Learning rich features for image manipulation detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1053-1061, 2018.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei A Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pp. 2223-2232, 2017.
Mingjian Zhu, Hanting Chen, Mouxiao Huang, Wei Li, Hailin Hu, Jie Hu, and Yunhe Wang. Gendet: Towards good generalizations for ai-generated image detection. arXiv preprint 2rXiv:2312.088800
2023.
Mingjian Zhu, Hanting Chen, Qiangyu Yan, Xudong Huang, Guanyu Lin, Wei Li, Zhijun Tu, Hailin Hu, Jie Hu, and Yunhe Wang. Genimage: A million-scale benchmark for detecting ai-generated image. Advances in Neural Information Processing Systems, 36, 2024.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A EXPERIMENTAL DETAILS
A.I DETECTORS
We choose a set of representative methods in Al-generated detection as baselines for comparison, including frequency-based (Frank et al., 2020; Liu et al., 2022a; Zhong et al., 2023), gradient-based (Tan et al., 2023), semantic-bused (Ojha et al., 2023), reconstruction based (Wang et al., 2023a), etc.
CNNSpot (CVPR (0200 (Wang et al., 2020) proposes that a naïve image classifier with simple data augmentations (ie. JPEG compression and Gaussian blur) can generalize surprisingly to images generated by unknown GAN-based architectures.
FreDect (ICML' (0020 (Frank et al., 2020) observes significant artifacts in the frequency domain of GAN-generated images and makes use of these artifacts for classification.
Fusing (ICIP'202)) (Ju et al., 2022) designs a two-branch model to fuse global spatial informa- tion and local informative features for training the classifier.
LNP (ECCV'2022) (Liu et al., 2022a) proposes to extract the noise pattern of images with learnable denoising network and uses noise patterns to train classifier.
LGrad (CVPR 2023) (Tan al., 2023) employs gradients computed by pretrained CNN model to present the generalized artifacts for classification.
UnivFD (CVPR 2023) (Ojha al., 2023) uses CLIP features to train binary liner classifier.
DIRE (023)) (Wang al., 2023a) observes obvious differences discrepancies between images and their reconstruction by DMs and uses this feature to train classifier.
PatchCraft (Arxiv'2024) (Zhong al., 2023) compares rich-texture and poor-texture patches from images and extracts the inter-pixel correlation discrepancy as universal fingerprint for classification.
NPR (CVPR 2024) (Tan et al., 2024) contributes to the architectures of CNN-based generators and demonstrates that the up-sampling operator can generate generalized forgery artifacts that extend beyond mere frequency-ban -based artifacts.
A.2 STATISTICS OF PUBLIC BENCHMARKS
Table 10 and Table 11 provide detailed explana- tion of Benchmark AIGCIDetectBenchhart (Zhong et al., 2023) & Genlmage (Zhu et al., 2024) intro- duced in our main paper. There are two main bench- marks here: AIGCDeteccBenchmatt and GenImage. AIGGDettetBeechharkk It is trained on ProGAN and then tested 16 different test sets, including data generated by both GAN and Stable Diffusion models. GenImage: It is trained Stable Diffusion V1.4 and tested on a large amount of data generated by Stable Diffusion, with only a small portion of GAN data included. The test sets related Stable Diffusion in AIGCDettecBennhhmakk are consistent with those used in GenImage.
Patch Size
Potch Number
(a)
(b)
Figure 3: Hyperparameter ablation of patch size and patch number introduced method.
B MORE EXPERIMENTAL RESULTS
B. AP RESULT
We additionally provide classification results regarding AP in Table 12. It is important highlight that the AP (Average Precision) metric emphasizes different aspects compared to Ace (Accuracy). While Acc focuses the overall correctness of predictions across all samples, AP provides more
16

---

Published as a conference paper at ICLR 2025
Table 10: Statistics of Benchmark AJGCDDeteBBnnchmrk. SD and WFIR refer Stable Diffusion and
whichfaceisreal, respectively. term "Number" only counts on fake images and an equal number of real images is added for each generative model from the same source.
Generator
Image Size
Number
Source
Train
ProGAN (Karras ct al., 2018)
256 256
360.0k
LSUN (Yu ct al., 2015)
ProGAN (Karras ct al., 2018)
256 256
8.0k
LSUN (Yu al., 2015)
StyleGAN (Karras et al., 2019)
256 x 256
12.0k
LSUN (Yu et al., 2015)
BigGAN (Brock et al., 2019)
256 x 256
4.0k
ImageNet (Deng al., 2009)
CycleGAN (Zhu al., 2017)
256 256
2.6k
ImageNet (Deng al., 2009)
StarGAN (Choi et al., 2018)
256 x 256
4.0k
CelebA (Liu et al., 2015)
GauGAN (Park ct al., 2019)
256 x 256
10.0k
COCO (Lin al., 2014)
StyleGAN2 (Karras al., 2020)
256 256
15.9k
LSUN (Yu ct al., 2015)
WFIR (WFI, 2023)
1024 x 1024
2.0k
FFHQ (Karras et al., 2019)
Test
ADM (Dhariwal & Nichol, 2021)
256 x 256
12.0k
ImageNet (Deng al., 2009)
Glide (Nichol ct al., 2022)
256 256
12.0k
ImageNet (Deng al., 2009)
Midjourney (mid)
1024 x 1024
12.0k
ImageNet (Deng et al., 2009)
SD v1.4 (Sta, 2022)
512 x 512
12.0k
ImageNet (Deng al., 2009)
v1.5 (Sta, 2022)
512 x 512
16.0k
ImageNet (Deng al., 2009)
VQDM (Gu et al., 2022)
256 x 256
12.0k
ImageNet (Deng et al., 2009)
Wukong (wuk, 2023)
512 512
12.0k
ImageNet (Deng al., 2009)
DALLE (Ramesh al., 2022)
256
2.0k
ImageNet (Deng al., 2009)
Table 11:
Statistics of Benchmark GenImage. SD refers to Stable Diffusion. The term "Number" only counts fake images and an equal number of real images is added for each generative model from the same sourec.
Generator
Image Size
Number
Source
Train
SD v1.4 (Sta, 2022)
512 x 512
324.0k
ImageNet Deng et al. (2009)
BigGAN (Brock et al., 2019)
256 x 256
12.0k
ImageNet (Deng et al., 2009)
ADM (Dhariwal & Nichol, 2021)
256 x 256
12.0k
ImageNet (Deng al., 2009)
Glide (Nichol et al., 2022)
x 256
12.0k
ImageNet (Deng al., 2009)
Midjourney (mid)
1024 x 1024
12.0k
ImageNet (Deng et al., 2009)
Test
SD v1.4 (Sta, 2022)
512 512
12.0k
ImageNet (Deng al., 2009)
v1.5 (Sta, 2022)
512 x 512
16.0k
ImageNet (Deng al., 2009)
VQDM (Gu et al., 2022)
256 X 256
12.0k
ImageNet (Deng et al., 2009)
Wukong (wuk, 2023)
512 512
12.0k
ImageNet (Deng al., 2009)
comprehensive evaluation of model's performance across various thresholds, particularly in han- dling imbalanced datasets. On top of that, our method still achieves SOTA performance among these baselines AP metric, which underscores the superiority of our approach. This indicates that our method not only excels in general prediction accuracy but also maintains robust performance across different decision thresholds, demonstrating its effectiveness in distinguishing between classes even in challenging scenarios.
B.2 MORE ABLATION STUDIES
B.2.1 PATCH NUMBER AND PATCH SIZE
We further ablate some key parameters defined in our method, namely the size of patches (Patch Size) and the num of selected patches (Patch Number). As shown Fig. (a), both an excessively large and overly small patch size can have certain impacts. If the patch size is too large, it may introduce additional irrelevant information, leading to interference with accurate judgment. On the other hand, if the patch size is too small, there may not be enough information available to make a proper judgment. As shown in Fig. 3 (b), for the patch number, the conclusion that there is a correlation between the number of patches and the patch size.
17

---

Published as a conference paper at ICLR 2025
Table 12:
Comparison the AIGCDeteclBenchmar (Zhong et al., 2023) benchmark. Average precision (AP %) of different detectors (rows) detecting real and fake images from different generators (columns). The best result and the second-best result are marked in bold and underline, respectively.
RoGAN
S0YeGAN
BoGAAN
ClokGA
StYGAN
CRNGAN
WFRR
5D v2.4
90
VODM
Wakang
DALLE2
Moon
Method
CNNSpol
100,00
66.24
61.00
68
99.99
46.09
Fusing
100.00
99.50
90.70
95.30
99.30
30
99.60
93.30
94.10
77.30
70.00
65.40
65.70
81.23
LNP
99.89
98.
84 33
92.83
100.00
78.85
99 59
91.45
94.20
88. 86.86
76 86
94.3
91.92
87
92 38
96 14
91.85
LGrad
100.00
98.31
92.92
95.01
100.00
95.43
97.89
57.99
72.95
S0.42
71.36
62.31
62.85
77.47
02 48
82.55
81.1
DIRE-G
UDivFD
99.08
91.74
56.68
46.91
75.25
80.56
99.34
40164
72.15
58.29
60.13
85.1
99.55
78.35
97.33
19.81
19.55
51.57
55 38
74.45
73.53
DIRE-D
99:27
99.37
83.81
PaschCruft
100,00
97.74
99.06
99.56
NPR
100.00
59.81
57.87
99.55
99.90
85.57
99.00
63.38
74.61
85.73
85.41
84
8467
81
50.55
76.72
86.7
AIDE
100,00
99.99
44
99.89
99.99
97.69
99.96
99.27
98.77
98.94
88. 13
98.26
98
99.27
98.62
99.41
98.18
Ground truth Al-generoted
Ground truth AI-generoted
AIDE AI-genercted
AIDE AI-generated
AIDE we SFE: Real
AIDE PFE: Real
Figure 4: Visualization effectiveness SFE Modules with Grad-CAM (Selvaraju al., 2017).
B.2.2 CONVNEXT AND VIT
We further explored the impact of CNN-Supervised network architectures such CLIP-ConVNeXt and ViT-Supervised CLIP-VIT. The results of CLIP-ViT on Benchmark have an average accuracy of 80.87%, which is significantly lower than the 92.77% achieved by CLIP-ConVNeXC We spec- ulate that this could be due to the CNN-supervised network architecture learning more low-level information, and for Al-generated image detection, low-level information is most crucial when the images are highly realistic.
C LIMITATIONS
Although our method achieves state-of-the-art results on publicly available datasets and demon- strates highly competitive performance on Chameleon, its performance on our own dataset still unsatisfactory. It leaves significant room for future improvement. Additionally, while the quality of our dataset is exceptionally high, its scale remains limited, and we aim to further expand its scale to better facilitate advancements in this field.
D VISUALIZATION
To more effectively verify the efficacy of the patchwise feature extraction (PFE) and semantic feature extraction (SFE) modules, we employ Crad-CAM (Selvaraju et al., 2017) to visualize the feature ar- targeted by these modules. In Figure 4 (a), it is evident that the region highlighted by the red box exhibits distinct semantic issues, which our SFE module successfully captures with clarity. Con- versely, in Figure 4 (b), there the low-level underlying noise information. Overall, our model, AIDE, demonstrates outstanding performance detector.
apparent semantic errors. and our PFE module accurately detects
18

---

Published as a conference paper at ICLR 2025
E POTENTIAL SOCIETAL IMPACTS
Given that Chameleon demonstrates the capability to surpass the "Turing Test", there exists a sig- nificant risk of exploitation by malicious entities who may utilize Al-generated imagery to engineer fictitious social media profiles and propagate misinformation. To mitigate it, we will require all users of Chameleon enter into an End-User License Agreement (EULA). Access the dataset will be contingent upon thorough review and subsequent approval of the signed agreement, thereby ensuring compliance with established ethical usage protocols.
19


Reviews:
[{"review_id": "7Via0jX4SH", "reviewer": "Reviewer_Wrdr", "summary": "This paper proposes a new dataset for AI-generated image detection named Chameleon. It also proposes a new model for AI image detection. Comprehensive evaluations are conducted.", "questions": "Mentioned in the weakness", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.A new datasets and evaluations to show the real status of AI-generated image detection are significant. This work shows that this task is far from solved.\n2.The proposed Chameleon dataset is of high-quality and from real-world distributions/sources.\n3.The proposed method is straightforward and effective. It combines both high-level semantic features and low-level pixel features.\n4.The experiments show effectiveness of the method and the hardness of the new dataset.\n5.This paper is easy to follow.", "weaknesses": "1.The experiments are mainly about the effectiveness of the proposed detection model. Some validation of the new dataset in its visual quality perspective will make the work stronger.  E.g., by human studies and comparisons between datasets.\n2.In Table 6, it is better to also list the accuracy numbers under no perturbations for convenient comparison.\n3.More recent related work can be compared or discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "E8x3j69AeH", "reviewer": "Reviewer_DQu6", "summary": "This paper challenges the assumptions in AI-generated image detection by proposing new training and testing settings. It introduces a benchmark called Chameleon to test human perception and shows significant performance drops in nine evaluated models. The paper also presents AIDE, a detector that combines low-level and high-level features, achieving state-of-the-art results but still offering room for improvement.", "questions": "See Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper is relatively well-motivated as AI-generated image detection is a crucial issue. A new dataset named Chameleon is proposed for detecting AI-generated images.  I also find the evaluations thorough. The target issues of the paper are meaningful and worth exploring. The motivation is clear. The paper is easy to follow.", "weaknesses": "1.Over the past few years, a multitude of expansive and varied collections of AI-generated imagery have been crafted. Undertaking an in-depth analysis to contrast the newly introduced Chameleon dataset against its peers, considering factors like dataset size, utilized generation techniques, would serve to underscore its unique contributions.\n\n2. It would be good if the authors collect fake images of fake news on the Internet and conduct experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KVmUkEyIYF", "reviewer": "Reviewer_L6tg", "summary": "This work aims to answer whether the AI-generated image detection problem has been solved. There are two main contributions in this submission, first, they propose a more challenging dataset, called Chameleon, to evaluate the generation ability of existing fake image detectors. Then, they propose a new detector with hybrid detection features, called AIDE (AI-generated Image DEtector with Hybrid Feature), that shows improved detection performance on this new challenging dataset, compared with the existing baseline detector.", "questions": "- Regarding the dataset construction, this work provides 4 principles in p4, one of which is to cover a wide range of categories. I quite agree with the point. However, this dataset only contains 4 categories, scene, object, animal, and human, and in some categories, there may be only hundreds of photos (e.g. Animal fake images = 313). Then, given the limited number of samples and categories, how can the authors ensure whether the images are representative enough (or cover a wide range of images to be detected)? \n\n- Regarding the dataset construction, have the authors strictly obey to the 4 principles as presented in p4, or the generated images have been carefully selected and validated in 9 baseline methods beforehand in order to create such as challenging dataset?\n\n- Experiments show 9 detectors cannot perform well on this dataset, this is good. However, I am eager to know the underlying reasons for their failure. Also, can the authors conduct a more fine-grained experiment that checks the accuracy of each of the 4 categories (scene, object, animal, and human), along with the overall category accuracy? I feel this might provide more insights. \n\n- Regarding the robustness evaluation of the proposed AIDE, if possible, please additionally evaluate JPEG compression with more realistic quality factors e.g. QF=75, QF=50 etc. Similarly, for Gaussian blur, please evaluate sigma=3 and 4.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work's main contribution is that it has proposed a more challenging dataset with diverse categories, with which 9 AI-generated image detectors cannot handle the detection problem well. In the AIGC detection problem, I think it is important to create such a challenging baseline, particularly given the fact that the existing benchmark seems saturated.\n\nBesides, a new detector with hybrid features (including frequency and semantics) is developed, and it shows improved performance on the AIGCDetectBenchmark, the GenImage benchmark, and Chameleon.", "weaknesses": "- This dataset only contains 4 categories, some of which only with hundreds of fake photos, this may not effectively cover the detection of a wide range of photos. \n\n- A recent work also adopts a mixture of expert architecture for fake image detection [1], which seems not be included as a baseline. It is supposed to discuss the major differences to AIDE, and serve as a possibly stronger baseline.\n\n- Regarding JPEG comparisons, only QF=95 and QF=90 are evaluated, more real-world compression such as QF=75, and even more severe but practical case e.g. QF=50 are supposed to be evaluated, too.\n\n[1] Liu, Z., Wang, H., Kang, Y., & Wang, S. (2024). Mixture of Low-rank Experts for Transferable AI-Generated Image Detection. arXiv preprint arXiv:2404.04883.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6OLQuRtaJn", "reviewer": "Reviewer_QHiA", "summary": "In this paper, the authors propose a new AI generated image dataset collected from the real-life scenarios, called Chameleon, in which the forgery images are genuinely challenging for human perception. Besides, this paper also provides a novel detection method, which exhibits better detection performance than previous SOTA methods.", "questions": "•\tThe questions are as mentioned above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "•\tThis paper provides a challenging AI generated image dataset, which is collected from real-life scenarios, and indicates that the problem for detecting AI-generated images is far from being solved.\n•\tThe authors provide a new detection method, achieving better performance than previous methods.\n•\tThis paper is organized well.", "weaknesses": "•\tThe rationale behind how patches with the highest and lowest frequency highlight the distinctive characteristics of AI-generated images is unclear. The manuscript lacks a theoretical explanation for the proposed method.\n\n•\tIn Table 2, comparing the entire dataset with subsets of the AIGCDetectBenchmark dataset seems unreasonable. A more appropriate comparison framework should be established.\n\n•\tFigure 4 does not provide sufficient evidence to support the claim that SFE (or PFE) can assist AIDE in focusing on specific semantic (or noise-related) clues. It may be beneficial to employ Grad-CAM to visualize the regions of interest when detecting forgery in images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KcsRKa1fOr", "reviewer": "Reviewer_9Ryx", "summary": "This paper introduces the Chameleon dataset and the AIDE detector for distinguishing between AI-generated and real images.", "questions": "Q1: What is the cost of data collection, and is it feasible to annotate more data to scale up the dataset?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "S1: Comprehensive Dataset: The Chameleon dataset offers a large-scale and diverse collection of AI-generated and real images, providing a robust foundation for training and evaluating detection models.\n\nS2: Detailed Performance Evaluation: The paper provides thorough baseline comparisons and benchmarking, highlighting the strengths and weaknesses of the AIDE detector across various scenarios.\n\nS3: Practical Applications: The focus on detecting AI-generated images is highly relevant in today's digital landscape, with the paper identifying practical areas for improvement.", "weaknesses": "W1: Method Innovation: The combination of patch-level features and global features proposed in this paper is not novel. The methods compared are mostly from the first half of 2023 and earlier, and some of the latest work is not included (such as LaRE^2, Forgery-aware Adaptive Transformer, etc.).\n\nW2: Inference Speed: The inference throughput of the proposed method is not provided, nor is it compared with previous methods. The AIDE uses multiple SRMs, ResNet50, and CLIP/ConvNext models, which are much larger than existing work like Gendet that uses just ResNet-50 as the classifier. This raises questions about whether the performance increase claimed in the article is due to increased FLOPS/PARAMS or the AIDE itself.\n\nW3: Dataset Comparison: The Chameleon dataset has not been comprehensively compared with existing datasets (such as CNNSpot, AIGCDetect, GenImage, WildFake, etc.) in terms of data size, data sources, semantic categories, generation models, image quality, and other relevant factors. Its novelty has not been sufficiently demonstrated.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)