Statistical Test on Diffusion Model-based Anomaly Detection by Selective Inference

Authors: Teruyuki Katsuoka, Tomohiro Shiraishi, Daiki Miwa, Vo Nguyen Le Duy, Ichiro Takeuchi
Primary area: probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords: diffusion models, anomaly detection, statistical test, selective inference

Abstract:
Advancements in AI image generation, particularly diffusion models, have progressed rapidly. However, the absence of an established framework for quantifying the reliability of AI-generated images hinders their use in critical decision-making tasks, such as medical image diagnosis. In this study, we address the task of detecting anomalous regions in medical images using diffusion models and propose a statistical method to quantify the reliability of the detected anomalies. The core concept of our method involves a selective inference framework, wherein statistical tests are conducted under the condition that the images are produced by a diffusion model. With our approach, the statistical significance of anomaly detection results can be quantified in the form of a $p$-value, enabling decision-making with controlled error rates, as is standard in medical practice. We demonstrate the theoretical soundness and practical effectiveness of our statistical test through numerical experiments on both synthetic and brain image datasets.

Decision: reject
Reviewer scores: 5, 5, 5

Reviews and discussion:
<thread>
creation date:
1730014451666

thread_type:
official_review

<thread_object>
creator: reviewer_nqfb

<content>
summary:
This paper introduces a statistical framework for anomaly detection in medical images using diffusion models, emphasizing reliable decision-making through selective inference. The authors propose a Diffusion Model-based Anomalous Region Detection test, which quantifies the statistical reliability of detected anomalies by computing selective p-values. This approach allows for controlled error rates, supporting clinical applications where rigorous reliability is essential. They implement selective inference with a piecewise-linear reconstruction error function, leveraging polyhedral geometry, and validate their method through numerical experiments and brain imaging tests. The work aims to bridge the gap between generative AI and statistical rigor in medical diagnostics.

strengths:
This approach leverages the unique combination of polyhedral and piecewise-linear functions in constructing a selective p-value, which allows the method to yield statistically sound inferences in an area where conventional anomaly detection techniques might lack formal reliability measures. The use of selective inference in the context of diffusion-based image generation is particularly novel, as the approach traditionally applies to feature selection in simpler models.

The authors provide a thorough theoretical foundation for their test. Their experiments demonstrate both the effectiveness and robustness of the method, detailing performance under various types of noise distributions and comparing it against other statistical tests like Bonferroni correction and permutation testing. Additionally, the stepwise construction and conditioning processes for selective inference are well-documented, highlighting the method’s robustness and adaptability in different testing scenarios.

The paper is well-structured, with a logical progression from the theoretical underpinnings to the experimental validation. The core concepts, including the their test, selective p-value, and the role of selective inference, are clearly defined. Figures illustrating the architecture and reconstruction process enhance reader comprehension. The piecewise-linear and polyhedral properties are described in a way that allows technically proficient readers to understand the model’s behavior and interpretability.

This work has some potential for impact. By providing statistical reliability measures via selective inference, the authors address a key limitation in deploying AI for clinical diagnostics: the need for transparent and error-controlled anomaly detection. The methodology’s ability to maintain controlled false-positive rates aligns with clinical requirements, supporting better trust and broader adoption of generative AI in medical applications. This is a meaningful contribution.

weaknesses:
The statement "the diffusion model detects anomalies" (p.4) is somewhat misleading, as the detection is more accurately based on comparing input and reconstructed images to highlight discrepancies. This phrasing implies the model actively detects anomalies, while in practice, it passively reconstructs images, with anomalies inferred from reconstruction errors. A more accurate description could clarify the diffusion model's role and prevent misinterpretation of the model's capabilities.

Many plots, notably Figures 2, 3, and 5, lack essential elements like axis labels and legends, limiting their interpretability. For complex analyses like these, well-labeled figures are critical for readers to follow and validate the findings. 

Given the complexity of the statistical testing framework, a visual illustration of the DMAD-test would be beneficial. Many statistical papers can be challenging to interpret without sufficient diagrams, and a schematic representation of the test could improve comprehension, particularly for readers unfamiliar with selective inference. This could be placed early on to guide readers through the methodology.

While computational resources are mentioned, there is no discussion of computational time, which is critical for assessing the method’s feasibility in practical applications. Adding specific timing metrics, ideally with a breakdown by test stage, would help readers gauge whether this approach could realistically be applied to clinical workflows.

The experiments focus on brain images with tumors, limiting the generalizability of the findings. Testing on a broader dataset, such as DeepLesion (https://nihcc.app.box.com/v/DeepLesion), or including non-tumor regions within volumes could provide a more comprehensive evaluation. Additionally, validating on datasets with a mix of normal and abnormal slices would assess the model's robustness and applicability to varied clinical scenarios.

The supplementary material mentions code, but it is unclear if this is executable code or pseudocode. Given the complexity of the statistical framework, accessible and runnable code would support reproducibility and encourage other researchers to build on this work. 

While the theoretical equations appear solid, some expressions, especially those in the SI framework, are intricate. Although no clear mistakes are present, a supplementary derivation or walkthrough of the core equations would support readers who may struggle with verifying them. This could be presented as an appendix to guide readers through critical steps and foster better understanding.

questions:
- Could you clarify the reasoning behind describing the diffusion model as “detecting” anomalies?
- Could you provide details on the computational time required for the DMAD-test on a standard dataset? 
- Could you clarify whether the supplementary code provided is fully executable, and, if so, whether it includes detailed instructions for setup and testing?
- Have you considered testing on broader datasets, such as DeepLesion or other modalities?
- Given the computational complexity and reliance on high-level statistical inference, do you foresee any specific challenges in deploying this method in real-time clinical settings?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730565466581

thread_type:
official_review

<thread_object>
creator: reviewer_gg2v

<content>
summary:
This paper proposes a method for detecting and quantify the reliability of anomalous regions in images. In particular, the authors focus on images generated by diffusion model and use selective inference framework. The authors test the present method using numerical experiments and on a brain image dataset. The output of the proposed model is a p-value that can determine the significance of the detected anomaly.

strengths:
1) The authors address a highly relevant problem by proposing a method for anomaly detection. A method that identifies the significance of anomalies and enable automated anomaly detection in a hospital setting would be extremely useful and important.
2) The authors test their proposed DMAD-test through numerical experiments and on a brain imaging dataset, which provides a good way of evaluating the method's effectiveness in different contexts.

weaknesses:
3) The section on diffusion models feels somewhat out of place, as it mainly covers basic concepts without providing specifics about the model actually used. It may be more effective to reference the foundational paper on diffusion models and focus on describing details relevant to this study, such as was the same architecture of the diffusion model used for the real data and the experimental data? Could the authors provide more information on the model architecture, input image sizes, and preprocessing steps?
4) Expanding on the previous point, the quality of reconstructed images is essential for the accuracy of the resulting p-values. Reporting image quality metrics like MMD (Maximum Mean Discrepancy), SSIM, or MS-SSIM would help assess the reconstruction quality. Specifically, calculating MMD/MS-SSIM between reconstructions and real healthy images would be informative, and it could also be insightful to compare these values between reconstructions of healthy images and those containing anomalies. For reference, Table 1 of https://arxiv.org/pdf/2307.15208 provides a similar evaluation. Including these metrics would significantly strengthen the paper’s conclusions, especially considering that only 329 images were used to train the diffusion model on real data.
5) Additional details on the images used would help readers better understand the study’s setup when using the brain images. Are the images 2D or 3D brain images, and how were they pre-processed before analysis?

questions:
6) The submission used the ICRL 2024 template. Could the authors update the submission to use the ICRL 2025 template?
7) Could the authors elaborate on what is the reference image? How is the reference image different than the reconstructed?
8) Could the authors discuss their choice to focus on diffusion models rather than exploring alternative models like transformers? Why were diffusion models selected over transformers, could the authors compare the advantages and disadvantages of diffusion models versus transformers for this particular anomaly detection task? Could the authors discuss any prior work that has used transformers for similar tasks.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730697878777

thread_type:
official_review

<thread_object>
creator: reviewer_fh2u

<content>
summary:
This paper introduces a statistical significance test to reconstruction-based anomaly detection, enhancing the reliability of decision-making with synthetic images generated by AI. In the context of anomaly detection, the proposed DMAD-test method quantifies the statistical significance of anomalous regions using p-values, which helps reduce the false alarm rate. In the experimental section, DMAD-test and its variant, DMAD-test-oc, are evaluated on a synthetic dataset and a brain anomaly detection dataset.

strengths:
1. The paper motivation is well presented. The problem the paper tackles is very important, as generative models may lead to deteriorate performance due to the quality of its output.

2. The introduction of a statistical significance test to refine identified anomaly regions is quite novel, offering a new approach to anomaly localization.

weaknesses:
1. The statistical test relies solely on the mean value of the identified anomalous region, without accounting for structural differences. In medical image anomaly detection, anomalous regions may have mean values similar to those of normal regions; however, the spatial structure of tissues and anatomical patterns often show significant differences. Without considering the structure difference in the statistical test, the obtained results may still suffer from large false detection.

2. It is not clear how to generate the reference image for calculating the statistics.

3. Although the paper includes visualization examples for brain image anomaly detection, it lacks quantitative results. Without a thorough quantitative comparison to prior methods, it is difficult to assess the significance and effectiveness of the proposed approach.

questions:
Please refer to the weakness section for my questions and comments.
</content>
</thread_object>

</thread>


Metareview: