Screener: Learning Conditional Distribution of Dense Self-supervised Representations for Unsupervised Pathology Segmentation in 3D Medical Images

Authors: Mikhail Goncharov, Eugenia Soboleva, Mariia Donskova, Ivan Oseledets, Marina Munkhoeva, Maxim Panov
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Unsupervised visual anomaly detection, self-supervised learning, density estimation, medical image analysis

Abstract:
Accurate and automated anomaly segmentation is critical for assisting clinicians in detecting and diagnosing pathological conditions, particularly in large-scale medical imaging datasets where manual annotation is not only time- and resource-intensive but also prone to inconsistency. To address these challenges, we propose Screener, a fully self-supervised framework for visual anomaly segmentation, leveraging self-supervised representation learning to eliminate the need for manual labels. Additionally, we model the conditional distribution of local image patterns given their global context,  enabling the identification of anomalies as patterns with low conditional probabilities and assigning them high anomaly scores.

Screener comprises three components: a descriptor model that encodes local image patterns into self-supervised representations invariant to local-content-preserving augmentations; a condition model that captures global contextual information through invariance to image masking; and a density model that estimates the conditional density of descriptors given their global contexts to compute anomaly scores.

We validate Screener by training a fully self-supervised model on over 30,000 3D CT images and evaluating its performance on four large-scale test datasets comprising 1,820 3D CT scans across four chest and abdominal pathologies. Our framework consistently outperforms existing unsupervised anomaly segmentation methods. Code and pre-trained models will be made publicly available.

Decision: reject
Reviewer scores: 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1730292607297

thread_type:
official_review

<thread_object>
creator: reviewer_5vep

<content>
summary:
This paper introduces SCREENER, a fully self-supervised framework for anomaly detection in 3D CT images. SCREENER learns a conditional distribution of dense image patterns, leveraging a descriptor model and conditional density model to assign anomaly scores based on rarity. The method shows promising performance on several CT pathology datasets, though some aspects could benefit from further clarity, additional benchmarks, and comparisons.

strengths:
Promising Results: SCREENER achieves impressive results across multiple datasets, particularly given its unsupervised nature. The method could advance anomaly detection in settings with limited labeled data.

Use of Conditional Density Modeling: The paper’s approach to conditioning on context in the density model is innovative and contributes to improving detection accuracy by simplifying anomaly scoring.

Effective Self-Supervised Learning Approach: SCREENER’s use of self-supervised pretraining on a large CT dataset provides a valuable alternative to supervised feature extractors, addressing the scarcity of annotated medical images.

weaknesses:
Limited Comparative Evaluation: While SCREENER shows good performance on mixed CT datasets, it is challenging to assess its true impact due to the limited number of similar baseline methods for comparison. Including a maximum Dice score comparison to supervised SOTA or additional metrics from benchmarks like MOOD would allow for a clearer evaluation.

Lack of Standard Dataset Benchmarks: Although the authors focus on CT images, adding results on a standard brain dataset would help position SCREENER relative to established anomaly detection methods and increase its comparative strength even if this is not the primary goal of the work.

Domain Gap Influence: The effect of domain gaps between different CT datasets is not discussed. SCREENER is tested on a single combined dataset, but further analysis of domain variance (standard deviations, confidence intervals) would better inform its generalizability. Clarifying how domain gaps might impact performance is also relevant, as it remains unclear how well SCREENER would generalize to datasets with different characteristics.

Architecture and Sampling Details: The choice of downsampling size (h, w, s), upsampling, and overall architecture needs further discussion to make the design decisions transparent and reproducible.

Related Work and Citations: The paper could improve its citation of related work, including references to early synthetic anomaly detection methods (e.g., FPI) and CRADL, which has a similar structure involving a SIMCLR pretrained encoder and Gaussian/flow-based density models. Additionally, work on applied studies on anomaly scoring could enhance the background section.

Clarity and Detail in Writing: The paper is somewhat unclear, with missing details that could make its contributions and methodology more convincing. For example, some claims (such as those about density models or scale of CT data) are not fully supported, and the paper’s writing could more clearly convey the innovations and unique aspects of SCREENER.

Overlapping Test and Training Data: Some test sets, like LiTS and KiTS, are partially represented in the training dataset AbdomenAtlas, which could raise concerns about test set contamination. While this overlap may not significantly affect results, a clear discussion of this issue is necessary to address any potential impacts.

questions:
Can you provide results for a maximum Dice score comparison with a supervised SOTA model to better position SCREENER’s effectiveness?

How does SCREENER perform on standard anomaly detection datasets (e.g., brain datasets), and would such benchmarks enhance the comparability of results?

Could you elaborate on the potential influence of domain gaps across different datasets in SCREENER’s performance, and is there any analysis of variance across domains?

How do choices like downsampling size, h, w, s, and upsampling affect SCREENER’s performance, and could these details be further specified?

How do you address the overlap of training and test datasets, specifically with parts of LiTS and KiTS also in AbdomenAtlas?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730664524560

thread_type:
official_review

<thread_object>
creator: reviewer_px7h

<content>
summary:
The paper presents SCREENER, a self-supervised framework for anomaly segmentation in 3D medical CT images, specifically aimed at pathology detection. SCREENER learns conditional distributions of dense self-supervised representations, assigning higher anomaly scores to patterns with low conditional probability. This method includes three components: (1) a descriptor model that encodes local patterns, (2) a condition model encoding global context, and (3) a density model estimating the likelihood of descriptors, yielding anomaly scores for segmentation. Trained on 30,000 CT scans, SCREENER outperformed existing unsupervised anomaly segmentation methods across multiple pathologies, showcasing its potential in large-scale, label-scarce medical imaging tasks.

strengths:
+ **Comprehensive Introduction and Related Work:** The introduction and related work sections are thorough and well-written. They offer a solid overview of existing methods in medical anomaly segmentation and clearly highlight their limitations, supported by relevant examples from the literature.

+ **Extensive Training Dataset:** The model is trained on a dataset of over 30,000 3D CT scans, which is a significant advantage for self-supervised learning. This large dataset aids in accurately modeling the distribution of healthy CT images, allowing the model to distinguish normal from abnormal patterns more effectively.

weaknesses:
- **Unclear Illustration of the Proposed Method:** The method section lacks clarity on what parts are novel contributions versus prior work. For example, Section 2.3 largely describes existing techniques, such as SimCLR and VICReg, but it’s unclear how much these influence the new SCREENER model. The paper would benefit from clearly marking new contributions at the start of each subsection and relegating baseline methods to the experimental settings section. A figure illustrating the architecture would greatly improve comprehensibility.

- **Inappropriate Metric Selection:** The paper focuses on anomaly segmentation, yet uses anomaly detection metrics (AUROC) rather than segmentation-specific metrics, such as Dice Similarity Coefficient (DSC), Intersection over Union (IoU), Hausdorff Distance (HD), or Normalized Surface Distance (NSD). Without these, it’s difficult to assess the model's effectiveness in precise segmentation of anomalies.

questions:
1. Clarify Novelty in SCREENER: The proposed SCREENER method claims to modify the density-based UVAS framework but lacks clarity on what modifications were made and their specific advantages. Clear differentiation of new versus existing components would help readers understand the proposed method’s originality.

2. Dense Features Learned by SimCLR: The paper states that SimCLR learns “dense features” (lines 81-82), but this is misleading as SimCLR typically captures global features to distinguish one image from another. Defining what is meant by "dense features" here and how SimCLR is adapted for this purpose would improve accuracy.

3. Define Key Terms: Essential terms like "conditional distribution," "local image patterns," and "global image content" are insufficiently explained, making the method challenging to follow. Definitions and possibly examples would clarify the underlying assumptions and techniques used.

4. Figure 1 Anomaly Map Clarity: In Figure 1, the second column shows an image flagged as diseased (as indicated by attention maps), yet lacks a mask. A mask in this column or clearer labeling would clarify the model's detection process.

5. AUROC Use for Segmentation Tasks: AUROC is not ideal for segmentation tasks, which typically require spatial accuracy. Metrics like HD or NSD would provide more nuanced insights into segmentation quality, particularly for edge or boundary delineation.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671704128

thread_type:
official_review

<thread_object>
creator: reviewer_jnqd

<content>
summary:
This paper introduces a self-supervised method for anomaly segmentation in 3D medical images. The model architecture includes a descriptor, condition, and density module. However, the paper lacks clarity in its presentation, with limited explanations and no figures to illustrate the pipeline, making it challenging to grasp the overall workflow and novelty of the approach.

strengths:
1. This work introduces a large-scale dataset on pathology segmentation, integrating data from NLST, AMOS. AbdomenAtlas.
2. This work aims to solve the meaningful problem of self-supervised pathology segmentation.
3. Lacks the statistics of the claimed large dataset. The author should summarize and describe this dataset in detail.

weaknesses:
1. This work is poorly written and lacks clarity in conveying its high-level concept. The descriptions of the descriptor, condition, and density models are unclear, and there are no experiments demonstrating the effectiveness of these components.
2. A pipeline illustration figure is recommended to show the workflow of this work.

questions:
1. Add a figure to show the pipeline of this work.
2. Add the summary for the dataset statistics, such as size, density, pathology distribution.
3. Add experiments/qualitative comparison to demonstrate the effectiveness of proposed components
</content>
</thread_object>

</thread>
<thread>
creation date:
1733308898089

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear Reviewers **jnQD**, **px7H**, and **5vEP**,

We sincerely appreciate the time and effort you have invested in reviewing our submission.

We are encouraged that you have recognized the main **contributions** of our work:

- **Fully Self-Supervised Framework for visual anomaly detection (Reviewer 5vEP)**: We show that dense self-supervised representations are favorable alternative to supervised feature extractors in the density-based framework for unsupervised visual anomaly segmentation (UVAS). As a result, the proposed framework is fully self-supervised and applicable in the domains where labeled data is limited.

- **Innovative conditioning on self-supervised visual features (Reviewer 5vEP)**: We further extend the density-based UVAS framework by showing that instead of hand-crafted conditioning, e.g. on positional encodings, one can learn data-driven self-supervised condition variables. Conditioning on these representations simplifies true conditional distribution and allows us to achieve remarkable anomaly segmentation results with a very simple gaussian model of conditional density.

- **Large-scale empirical study and strong results (Reviewers jnQD, px7H, 5vEP)**: This paper presents the first large-scale study of UVAS methods in 3D medical CT images. We show that the proposed density-based framework outperforms other UVAS methods on unsupervised semantic segmentation of a wide range of pathologies in different anatomical regions, demonstrating strong generalization capabilities.

Your insightful feedback has been invaluable in refining our work, and we are grateful for the opportunity to address your concerns. Below, we summarize how we have incorporated your suggestions to enhance the clarity and overall quality of our manuscript.

**Clarity and Presentation Enhancements**

- **Added Method Illustration (Reviewers jnQD and px7H):** We have included **Figure 2** to visually depict the overall workflow of SCREENER, clearly illustrating the interactions between the descriptor model, condition model, and density model.
- **Added “Background & notation” Section (Reviewers jnQD, px7H, 5vEP):** A new “Background & notation” section (Section 2) has been added to outline existing frameworks (density-based UVAS and dense SSL) and position our innovations within the context of the literature.
- **Revised “Method” Section (Reviewers jnQD, px7H, 5vEP):** We have restructured the “Method” section (Section 3) to distinctly highlight our novel contributions versus prior work. Sections 3.1 to 3.3 now offer in-depth explanations of the descriptor model, condition model, and density model, respectively.
- **Improved Figure 1 (Reviewer px7H):** We have clarified that the second image in Figure 1 contains pneumothorax which is not annotated in the ground truth, but is correctly identified by our model.
- **Updated Dataset Information (Reviewer jnQD):** Table 1 has been updated to include detailed statistics of the datasets used, specifying the number of images with non-zero pathology masks.

**Choice of Evaluation Metrics (Reviewers px7H and 5vEP)**

We acknowledge the limitations of using pixel-level AUROC for segmentation tasks. However, due to partial and class-specific annotations in available datasets, metrics like Dice score are challenging to compute fairly. That's why, we followed the evaluation protocol from the MVTecAD benchmark, utilizing pixel-level AUROC and AUPRO metrics.

**Demonstrating the Effectiveness of Components (Reviewer jnQD)**

We have conducted extensive experiments, including ablation studies presented in Tables 3 and 4, to validate the effectiveness of each component of SCREENER. These studies demonstrate the contributions of the descriptor model, condition model, and density model to the overall performance.

**Addressing Domain Gap and Generalization (Reviewer 5vEP)**

- **Performance Across Different Datasets:** Despite inherent domain gaps between the training and test datasets, SCREENER achieves high performance on diverse chest and abdominal CT datasets.
- **No Overlap Between Training and Test Sets:** We verified that there is no overlap between our training datasets (NLST, AMOS, AbdomenAtlas) and test datasets (LIDC, MIDRC-RICORD-1a, KiTS, LiTS). This ensures the validity of our evaluation and addresses concerns about potential test set contamination.

**Concluding Remarks**

We believe that these revisions have substantially strengthened our manuscript by addressing the key concerns raised during the review process. We are confident that SCREENER represents a significant advancement in unsupervised visual anomaly segmentation and holds considerable promise for real-world medical imaging applications.

Thank you once again for your valuable insights and constructive suggestions. We are grateful for your consideration and hope that our revisions meet your expectations.
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper introduces a fully self-supervised framework, dubbed Screener, for 3D medical image anomaly segmentation. In this work, self-supervised representation is employed to completely eliminate the need for manual labeling, reducing this labor-intensive requirement for large-scale datasets. Experimental analysis shows promising results in fully self-supervised medical image anomaly segmentation.

This paper received 1x marginally above the acceptance threshold and 2x marginally below the acceptance threshold from reviewers. The questions raised by reviewers regarding this work centered around the clarity of the research question, methodology design, and key term definition. Although the authors addressed most of the concerns during the rebuttal phase, one key problem, i.e., the evaluation metrics, has not achieved consensus between the reviewer and the authors. As reviewer px7H suggested, several publicly available datasets provide per-voxel annotations for lesions across multiple organs and thus this inconsistency should be further considered and addressed. Meanwhile, I notice that the methods used for comparison are somehow limited and more recent SOTA methods are suggested to be included.

Therefore, although the clarity and readability have been largely improved during the discussion phase, further improvements are needed to enhance the comprehensiveness of this work. Rejection is recommended.