Manifold Induced Biases for Zero-shot and Few-shot Detection of Generated Images

Authors: Jonathan Brokman, Amit Giloni, Omer Hofman, Roman Vainshtein, Hisashi Kojima, Guy Gilboa
Primary area: generative models
Keywords: zero-shot, few-shot, generated image detection, total-variation, curvature, score function, diffusion models

Abstract:
Distinguishing between real and AI-generated images, commonly referred to as  'image detection', presents a timely and significant challenge. Despite extensive research in the (semi-)supervised regime, zero-shot and few-shot solutions have only recently emerged as promising alternatives. Their main advantage is in alleviating the ongoing data maintenance, which quickly becomes outdated due to advances in generative technologies. We identify two main gaps: (1) a lack of theoretical grounding for the methods, and (2) significant room for performance improvements in zero-shot and few-shot regimes. Our approach is founded on understanding and quantifying the biases inherent in generated content, where we use these quantities as criteria for characterizing generated images. Specifically, we explore the biases of the implicit probability manifold, captured by a pre-trained diffusion model. Through score-function analysis, we approximate the curvature, gradient, and bias towards points on the probability manifold, establishing criteria for detection in the zero-shot regime. We further extend our contribution to the few-shot setting by employing a mixture-of-experts methodology. Empirical results across 20 generative models demonstrate that our method outperforms current approaches in both zero-shot and few-shot settings. This work advances the theoretical understanding and practical usage of generated content biases through the lens of manifold analysis.

Decision: accept (poster)
Reviewer scores: 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1729688987995

thread_type:
official_review

<thread_object>
creator: reviewer_qnvq

<content>
summary:
The paper presents a novel approach to detecting AI-generated images using a zero-shot and few-shot framework. By analyzing biases inherent in the manifold of pre-trained diffusion models, the authors introduce a new mathematical criterion based on score functions, curvature, and gradient analysis. This approach generalizes well to unseen generative techniques and outperforms existing methods in both zero-shot and few-shot scenarios. Extensive experiments across a diverse set of generative models further validate the effectiveness of the proposed method.

strengths:
The paper provides a sound theoretical foundation by integrating manifold analysis with diffusion models, advancing the field of generated image detection.

The empirical results show strong performance, with the proposed method outperforming current state-of-the-art approaches.

The authors conducted experiments across various datasets and generative techniques, including GANs, diffusion models, and commercial tools, providing strong evidence for the robustness of the method.

weaknesses:
According to the article, the proposed curvature and gradient based metric for detecting generated images is closely related to the score function. However, it is unclear why this method also performs well with models where score functions are not inherently applicable, such as CycleGAN. The authors are encouraged to clarify this connection and explain why the proposed metric shows strong performance even in such cases where score function analysis is not directly relevant.

There are citation issues on page 7 of the article, where footnotes and page numbers are not correctly referenced. The authors should revise the citation formatting to ensure all references are accurate and properly aligned with the content.

The proposed method appears to fit naturally within a zero-shot framework, relying solely on input samples and corresponding perturbations. It is unclear why the few-shot setting was introduced, given that zero-shot scenarios are typically more challenging and often reflect real-world situations. The authors are encouraged to clarify the need for a few-shot setting and explain why zero-shot alone would not suffice as a more compelling and realistic approach.

questions:
Please see weakness.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730374726194

thread_type:
official_review

<thread_object>
creator: reviewer_ghkq

<content>
summary:
This work addresses the challenge of distinguishing between real and AI-generated images by analyzing biases on the probability manifold of pre-trained diffusion models. The authors develop a method that offers a scalar criterion for classification in zero-shot settings, and experiments demonstrate its effectiveness against current methods.

strengths:
The theoretical analysis of current diffusion models’ score functions is comprehensive and novel, potentially inspiring further research in this area.

The proposed method generalizes well to unseen generative techniques and achieves superior performance over existing approaches in both zero-shot and few-shot settings.

weaknesses:
The implementation details in Section 4.3 should be elaborated further to enhance readability and reproducibility.


Typo errors:
- L153 and L149, inconsistent use of $\mathcal N$ and $N$.
- L146 and L170, inconsistent use of $\mathbb{R}$ and $R$
- L157, better use latex log $\log$ for clarity.
- L191, use latex \` for upper quotas.
- inconsistent use of Sec. and Section
- L298-L299, unexpected equation.

questions:
The author choose SD-1.4 to implement the proposed method, have the author tried other diffusion models, especially recently more advanced methods, such as SDXL and SD3. Does this helps improve the performance?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730580941002

thread_type:
official_review

<thread_object>
creator: reviewer_dpaq

<content>
summary:
The paper explores a novel method to detect AI-generated images, focusing on zero-shot and few-shot regimes. It identifies key challenges in the field, such as the need for data upkeep with traditional supervised learning methods and limited theoretical grounding for current approaches. The authors propose a framework based on the implicit biases within the manifold of a pre-trained diffusion model, leveraging score-function analysis to approximate manifold curvature and gradient in the zero-shot setting. They extend the method for few-shot scenarios by incorporating a mixture-of-experts strategy. The proposed method demonstrates enhanced performance across 20 generative models.

strengths:
- The idea of leveraging manifold-induced biases from pre-trained diffusion models to detect generated images is novel and interesting.
- The methodology, essential theoretical formulations, and results are well-articulated, with equations and definitions supporting the approach.
- Experimental results are promising, and Figures 1 and 2 are intuitive and easy to understand.

weaknesses:
The primary concern is the robustness; please see the Questions below.

questions:
1. The proposed method relies on pre-trained diffusion models and manifold analysis, which are implemented in high-dimensional space, potentially increasing computational costs. Could the authors provide an analysis of inference time and memory requirements?

2. The method depends on certain hyperparameters, such as perturbation strength and the number of spherical noises. How robust is the method to these parameters across different models, and what guidelines can be provided for selecting these parameters?

3. The authors tested the impact of JPEG compression on the method and reported a slight performance decrease. How does the method perform with other types of image post-processing, such as augmentation, denoising, and flipping?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730643453801

thread_type:
official_review

<thread_object>
creator: reviewer_xh3j

<content>
summary:
This paper introduces a zero/few-shot framework for detecting AI-generated images by analyzing the inherent biases of a pre-trained diffusion model. The authors hypothesize that generated images are more likely to occupy stable local maxima on this learned manifold, characterized by specific curvature and gradient properties. By approximating these properties, they create a criterion to distinguish between real and generated images without requiring large datasets or retraining. Empirical results show their method outperforms other detection approaches across multiple generative models.

strengths:
- Zero-shot and few-shot capability makes the method practical.
- The theoretical perspective is interesting.
- Empirical results seem promising.

weaknesses:
1. Some theoretical assertions, such as the assumption that generated samples are more likely to be stable local maxima on the learned manifold, are not fully justified. This assumption underpins the detection criterion, but the paper does not offer a thorough mathematical or empirical rationale to support it.
2. The paper relies heavily on approximations in score-function and curvature estimations (e.g., Eq.5, 16-18). However, there is limited discussion or analysis of the tightness of these approximations. This could lead to questions about the reliability of the approximations, especially when they form the foundation of the theoretical claims. It would be beneficial if the authors provided error bounds analysis or empirical justifications for these approximations.
3. In line 122, the authors argue that previous methods still rely on access to generative methods during training, leading to biases towards those generation techniques. However, the proposed approach also relies on a pre-trained SD1.4. How does the proposed approach avoid the bias from it? For example, a realistic sample generated from a more recent model may not sit on a stable local maximum in the learnt log probability manifold of SD1.4.
4. The presentation can be improved. The mathematical analysis can benefit from illustrative examples, while the detailed proof can be moved to appendix.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732783021516

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Discussion Summary

comment:
Dear Reviewers,

We sincerely appreciate your thoughtful feedback, which significantly enhanced our work. As we enter the final 5 days of discussions, we summarize the progress made in addressing all the concerns raised - which led to enhanced clarity, validated approximations, and demonstrated robustness, supported by key experiments presented in Table 2 and Figures 3, 4.c, 7, 10, and 11 (detailed below). We welcome any further feedback or questions you may have.

Best regards,

The Authors

**Summary of Our Contributions and Strengths:**

- **Innovative Theoretical Framework:** We present novel mathematical derivations to detect generated images as points near local maxima of the (log) probability manifold. To this end we present novel derivations, combining score-function analysis and high-dimensional considerations, for curvature and gradient approximations. The resulting formula can be applied with any pre-trained diffusion model.
- **Exceptional Generalization with a +39.1% Average AUC Improvement** over the state-of-the-art zero-shot methods, AEROBLADE and RIGID (2024). This was validated on a comprehensive benchmark of **200K images**, including real and generated samples from **20 diverse generative techniques** (Table 1, Fig. 5). The remarkable improvement stems from our method's superior cross-technique stability, as detailed in Fig. 8 of the Appendix. Furthermore, when integrated into a few-shot setting, our approach enhances performance by **+4% to +7%**, underscoring its applicability. 

**Summary of the Key Concerns Raised and our Responses and Experiments:**
- **Justification for the Local Maxima Assumption (Reviewer Xh3J):** 
1.  We build on DetectGPT by Mitchell et al., 2023 [1], which revolutionized zero-shot detection of *generated text*, relying on the same assumption (lines 69, 186 in $\textcolor{green}{\text{green}}$). Our research can be perceived as the first extension of [1] to the image domain. However, [1] uses LLM's explicit probability modeling, rendering the method unsuitable for images, since GANs and diffusion models are implicit models. To address this challenge, we base our method on score function analysis. 
2. **Fig. 3.a** further justifies this assumption with empirical evidence of reverse diffusion trajectories terminating near local maxima. We used Gaussian Mixture Model data, inspired by Song & Ermon, 2019 [2] (Details and statistics are in Appendix A, Fig. 7).
- **Error Analysis of Curvature Approximations (Reviewer Xh3J):** In **Figs. 3.b–d and 11**, we validate our curvature estimator using an analytic function with ground-truth curvature, demonstrating high reliability:
1. *Expected Behavior:* Local maxima have higher values than saddle points.
2. *Robustness:* Maxima and saddle points are separable even with low sample sizes (Fig. 3.c).
3. *Consistent Estimator:* Our curvature estimations are **empirically unbiased** (Fig. 3.c) with **exponentially decaying std** (Fig. 3.d).

- **Robustness Analysis (Reviewers DPAQ, GHKQ):** **Table 2** presents sensitivity and ablation studies, demonstrating strong robustness, maintaining SOTA performance under all tests requested by the reviewers:
1. Hyperparameter changes.
2. Image corruptions
3. Alternative, more advanced, base diffusion models. 
4. **Fig. 10** in the Appendix further details per-model hyperparameter robustness.

- **Visualization and Improved Writing (Reviewers qnVq, Xh3J, GHKQ):**
1. **Fig. 4.c** illustrates the high-dimensional *concentration of measure*, central to our derivations.
2. Writing was clarified, including explanations to calculate our criterion (Sec. 4.3), motivation for the few-shot approach (Sec. 5.2), citations in Limitations section, and typographical corrections.

Thank you once again for your time and detailed feedback. It truly helped us improve our work.

**References (Edited)**

[1] Eric Mitchell, Yoonho Lee, Alexander Khazatsky, Christopher D. Manning, and Chelsea Finn. DetectGPT: Zero-shot machine-generated text detection using probability curvature. In *International Conference on Machine Learning*, pp. 24950–24962. PMLR, 2023.

[2] Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. *Advances in Neural Information Processing Systems*, 32, 2019.

**Original comment (references inadvertently omitted):** 28.11  

**Edit to include references:** 3.12
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper presents a novel mathematical framework for quantifying bias in generated images. The key idea is that in generated images, there is bias that can distinguish them from real images. All reviewers are positive of the value of the work, its technical novelty and its strong empirical justification.