Dancing with Discrepancies: Commonality Specificity Attention GAN for Weakly Supervised Medical Lesion Segmentation

Authors: Ge Su, Tiancheng Zhao, Kaiping Zheng, Qingpeng Cai, Peng LU, Yin Wang, Jianwei Yin, Shuiguang Deng, Hangjin Jiang
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: medical image segmentation, weakly supervised segmentation

Abstract:
Increasing weakly supervised semantic segmentation methods concentrate on the target segmentation by leveraging solely image-level labels. However, few works notice that a significant gap exists in addressing medical characteristics, which demands massive attention. In this paper, we note: (i) Lesion regions typically exhibit a sharp probability distribution pattern while healthy tissues adhere to an underlying homogeneous distribution, which deviates from typical natural images; (ii) Boundaries of lesion foregrounds and structural backgrounds are blurred; (iii) Similar structures frequently appear within specific organs or tissues, which poses a challenge to concentrating models’ attention on regions of interest instead of the entire image. Thus we propose a Commonality-specificity attention GAN (CoinGAN) to overcome the above challenges, which leverages distribution discrepancies to mine the knowledge underlying images. Specifically, we propose a new form of convolution, contrastive convolution, to utilize the fine-grained perceptual discrepancies of activation sub-maps to enhance the intra-image distribution, making lesion foregrounds (specificity) and structural backgrounds (commonality) boundary-aware. Then a commonality-specificity attention mechanism and the GAN-based loss function are devised to jointly suppress similarity regions between different labels of images and accentuate discrepancy regions between different labels of images. This isolates lesion areas from the structural background. Extensive experiments are conducted on three public benchmarks. Our CoinGAN achieves state-of-the-art performance with the DSC of 71.69%, 84.73%, and 78.32% on QaTa-COV19, ISIC2018, and MoNuSeg datasets, making a significant contribution to the detection of pneumonia, skin disease, and cancer. Furthermore, the visualized results also corroborate the effectiveness of CoinGAN in segmenting medical objects.

Decision: reject
Reviewer scores: 5, 3, 5, 5

Reviews and discussion:
<thread>
creation date:
1730125794539

thread_type:
official_review

<thread_object>
creator: reviewer_eb1c

<content>
summary:
This paper introduces a novel approach to weakly-supervised medical image segmentation that integrates C-conv for intra-image discrepancy learning, effectively reducing boundary uncertainty. Additionally, it employs CSA mechanisms for inter-image discrepancy learning. The proposed method demonstrates state-of-the-art performance across three public benchmarks.

strengths:
The idea of utilizing two convolutional layers with different receptive fields to enhance boundary detection is intriguing. The improvement over baseline models is substantial.

weaknesses:
1. The discussion of motivation lacks depth. Three major challenges underpin this method:
a.	The intensity distribution of pathological images differs from that of healthy images, allowing classification networks to shortcut the learning process and overlook detailed spatial information.
b.	Lesion boundaries often appear ambiguous.
c.	Images frequently share similar anatomical structures.
Regarding the first challenge, most generative method-based approaches effectively address this issue [1-4]. For the second challenge, numerous studies have integrated boundary-aware modules into medical image segmentation [5-7], yet the authors do not discussion about existing literature. As for the third challenge, it is unclear why it is categorized as a challenge in the context of this work.
2. As a GAN-based method, the authors primarily discuss and compare their approach with CAM-based methods, neglecting comparisons with other GAN-based or diffusion-based techniques. Additionally, the domain-specific baselines referenced in the paper appear somewhat outdated.
3. The paper is not easy to follow. Especially the method part, which is difficult to understand and contains numerous ambiguities and unclear points (refer to the questions for specifics).

[1]. Hu, Xinrong, et al. "Conditional diffusion models for weakly supervised medical image segmentation." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.

[2]. Li, Jinpeng, et al. "Fast non-markovian diffusion model for weakly supervised anomaly detection in brain mr images." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.

[3].Cycles with Masked Conditional Diffusion for Unsupervised Anomaly Segmentation in MRI." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2023.

[4]. Gonzalez-Jimenez, Alvaro, et al. "SANO: Score-based Diffusion Model for Anomaly Localization in Dermatology." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.

 [5]. Prabakaran, Bharath Srinivas, Erik Ostrowski, and Muhammad Shafique. "BoundaryCAM: A Boundary-based Refinement Framework for Weakly Supervised Semantic Segmentation of Medical Images." 

[6]. Lin, Yi, et al. "Rethinking boundary detection in deep learning models for medical image segmentation." International Conference on Information Processing in Medical Imaging. Cham: Springer Nature Switzerland, 2023.

[7]. Hatamizadeh, Ali, Demetri Terzopoulos, and Andriy Myronenko. "Boundary aware networks for medical image segmentation." arXiv preprint arXiv:1908.08071 10 (2019).

questions:
1.	From my understanding, C-Conv detects the boundary and subsequently removes the local representation at that boundary. Could this lead to a loss of valuable information? Additionally, might this approach impact boundaries of certain sturctures within the foreground or background, not just the boundary between the foreground and background?
2.	In 272, what is the size of the reference samples and how they are selected and dynamically replaced?
3.	What distinguishes the proposed average buffer from traditional prototypes or memory banks?
4.	It seems that the generator only produces latent representations of the healthy distribution. How are the segmentation mask and the transformed healthy modality in Figure 6 generated?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730499440514

thread_type:
official_review

<thread_object>
creator: reviewer_sjzh

<content>
summary:
CoinGAN is designed for weakly supervised semantic segmentation (WSSS) in medical imaging. Key to CoinGAN is its use of a new convolution technique, contrastive convolution (C-Conv), and a dual attention mechanism (commonality-specificity attention.

strengths:
The paper introduces a new form of convolution that helps accentuate fine-grained perceptual discrepancies within activation sub-maps, aiding in better delineation of lesion boundaries. A dual attention mechanism is used to suppress similarities in structural backgrounds across images while highlighting unique lesion characteristics.

weaknesses:
1. The paper lacks a discussion on its generalizability across diverse medical imaging modalities and less common diseases.
2. The paper should provide a more detailed comparison with existing weak supervision methods, particularly those that do not use GAN architectures.
3. The paper lacks a detailed error analysis that could help identify the specific conditions under which the model performs poorly.
4. In my opinion, CoinGAN's performance is not acceptable since many semi-supervised models [1] can achieve much better performance with limited annotation.
[1] Li, Z., Li, Y., Li, Q., Wang, P., Guo, D., Lu, L., ... & Hong, Q. (2023). Lvit: language meets vision transformer in medical image segmentation. IEEE transactions on medical imaging.

questions:
1. Can the model adapt to other forms of medical imaging data, such as MRI or CT images?
2. Since the author claims to use image-level annotation, I can understand that the label of the COVID dataset is normal and abnormal. However, the author needs to explain the use of the MonuSeg annotation.
3. How is the model's robustness to inaccuracies and variability in image-level labels?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730671501048

thread_type:
official_review

<thread_object>
creator: reviewer_ld6q

<content>
summary:
The paper presents a method for weakly supervised medical lesion segmentation. The authors make two observations about the characteristics of lesion regions in the images and propose a GAN-based method that aims to exploit these characteristics to improve segmentation quality. The method is evaluated on three public benchmark datasets and compared with several state-of-the-art baselines. The paper also includes an ablation study.

strengths:
* The observation-based approach to design the method is interesting. From what I understood from the observations, this might be an interesting direction of research.

* The evaluation is fairly extensive, with comparisons on multiple datasets and with a number of alternative methods.

* From the results, it seems that the proposed method outperforms the other methods in the experiments.

weaknesses:
While the ideas behind the method could be interesting and the evaluation seems fairly extensive, I must admit that I found the paper very hard to follow.

From the Introduction, the assumptions and the general idea of what the method does remain unclear to me.

* Why do we need a GAN to learn from image-level labels? If we want to classify, detect, localize, segment something, why do we need a GAN? I don't think this is explained.

* The arguments about intensity and distributions are unclear. The terms are not defined (what exactly is a "sharp and high-intensity anatomical distribution" and how does this relate to the problem?). The assumptions also seem quite specific to these datasets and applications: does a high intensity always correlate with malignancies?

* The method apparently studies a "distribution shift" that is "driven" by a "GAN-based adversarial loss function", but from the Introduction it is unclear to me what this distribution shift indicates, and how it would benefit a weakly supervised segmentation model.

The description of the method is very technical and, at least for me, did not help to clarify what the method is intended to do and how it works.

Combined with the writing and word choice, which is often vague and imprecise, I found the presentation of the paper insufficient. There may be interesting ideas in the method -- apparently, it does improve performance -- but the paper did not help me to understand what they are and how they work.

questions:
Some suggestions for improvement, highlighting some of the parts that I found unclear:

* Page #1 (Introduction):
  > a diverse array of computer vision tasks, e.g., autonomous driving Jiang et al. (2024), robotics Panda et al. (2023) and medical diagnosis Huang et al. (2024).

  These are oddly specific references for such a general statement.

* Page #1 (Introduction):
  > On the contrary, some weak supervision alternatives, e.g., image-level labels He et al. (2024), points Gao et al. (2024), and bounding boxes Cheng et al. (2023), are effortless to obtain.

  I understand they are cheaper/easier to obtain, but they are not "effortless".

* Page #1 (Introduction):
  > Image-level WSSS is extremely challenging since these image-level labels solely indicate the presence or absence of the target object without specifying any location information.

  Doesn't that also depend on the type of label? It could be the size of the object, or the severity, for example. It doesn't have to a binary present/not present.

* Page #2 (Introduction):
  > Our insight is that medical segmentation hinges on pronounced discernible information, image-level supervision is vulnerable to some medical challenges pointing to an unstable convergence but the inherent discrepancy information encapsulated within the images can assist in further diving into the whole discriminative regions.

  I have no idea what this sentence is meant to say, or what the subfigures on the left are supposed to show.

* Page #2 (Introduction):
  > but such models may not grasp what makes medical segmentation overflow and bad uncontrollable shape.

  This is grammatically incorrect, and I find it hard to understand what is meant here. What does "overflow" mean? And "bad uncontrollable shape" of what? Uncontrollable by whom?

* Page #2 (Introduction):
  > As in Figure 1 (Right), sharp regions (high-intensity distribution) typically indicate a lesion that deviates from normal tissues (homogeneous distribution). The anomalous distribution shifts (high → low) may excavate valuable knowledge gaps.

  I have no idea what this means. Is this supposed to say that high-intensity pixels always indicate disease? (That might hold for this application, but isn't true in a general sense.)

  What are "anomalous distribution shifts" and what does it mean that they "excavate" knowledge gaps?

* Page #2 (Introduction):
  > GAN

  Why do we need a GAN to learn from image level labels? Wasn't the goal to classify, detect, or localize something?

* Page #2 (Introduction):
  > by suppressing inter-image strong-related areas and accentuating weak-related areas.

  Related to what?

* Page #2 (Introduction):
  > The CSA mechanism is designed to explore inter-image structural anomalies

  What are "inter-image structural anomalies"?

* Page #2 (Introduction):
  > Finally, a GAN-based adversarial loss function drives the distribution shift.

  Why does the distribution shift need to be driven? What does that mean? And wouldn't we want to reduce a distribution shift?

* Page #4 (Motivation & Overview):
  > The second answer is that the output structure lacks the constraints of background information, that is, the ignorance of common knowledge makes a free boundary.

  This is quite vague. What "background knowledge" and how would this "common knowledge" prevent a "free boundary" (and what is that anyway)?

* Page #5 (Contrastive Convolution (C-Conv) Module):
  > Thus we propose a new form of convolution, C-Conv, to address the above ambiguous elements.

  What "elements" does this refer to? What is an "ambiguous element"?

* Page #5 (Commonality-Specificity Attention (CSA) Mechanism ):
  > the CSA mechanism is proposed to delve into the inter-image distribution discrepancies

  The verb "delve" is really vague: what does CSA mechanism do with the discrepancies? Does it try to reduce them? Does it make them stronger? Does it use them for something else?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730715335050

thread_type:
official_review

<thread_object>
creator: reviewer_dqst

<content>
summary:
The paper proposes a weakly supervised semantic segmentation (WSSS) method for lesion segmentation. The main contributions of the paper are two modules: one is called contrastive convolution which focuses on the discrepancies between lesion and healthy structures to reduce the uncertainties in boundaries, the second one is a dual attention mechanism called CSA which learns inter image discrepancy with adversarial training. The experiments are performed on 3 public datasets and the method is compared to generic sota WSSS methods and to methods specific for some medical images, along with the ablation studies. Additionally, the paper demonstrate that WSSS methods that work well on natural images do not perform well on medical datasets. The results show that the proposed method achieves significant improvement.

strengths:
- The idea presented in the paper is interesting.
- The method is validated on sufficiently large datasets and compared with various SoTA methods.
- The results demonstrate that the method achieves remarkable improvement.

weaknesses:
I think the major weakness of the paper is the unclear description and lack of some important details:
- Average buffer and SElayer are crucial components of the proposed architecture; however, the details of these components are not provided in the paper. Please explain how do these components work in detail.
- x_p and x_h used in Figure 2 are not defined in the paper. To my understanding, one of them is the pathological image and the other is the healthy one. However, my understanding brings more questions regarding the datasets used in the experiments. For example, QaTA-Cov19 is a pneumonia benchmark which does not contain healthy images. Where are the healthy images used in this experiment coming from? This question is also valid for the other datasets. Please clarify.
- It is not very clear to me how does the proposed method predict segmentation masks from image-level annotations. As far as I understand, the method converts the pathological images to the healthy ones by removing the pathologies. Are the segmentation masks obtained by taking the difference between the original image and the converted one?

questions:
- How are Average buffer and SElayer components work?
- From which datasets are the healthy images used in the experiments coming from?
- How does the the algorithm predict segmentation masks from image-level annotations? Is it the region obtained after subtracting the input image and its translated version to an healthy image? If so, does this subtraction reveals any false positives? How are they removed, if any?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732296298389

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for your reviews

comment:
We would like to extend our profound gratitude to all the reviewers for their insightful comments and constructive suggestions, which have played a pivotal role in clarifying our contributions and further improving the quality of our paper.

In response to the reviews, we have conducted the following additional experiments during the rebuttal stage:
- Evaluation of CoinGAN on a different medical imaging modality and less common diseases (Reviewer SjzH (W1 & Q1)).
- Inclusion of a robustness study to demonstrate the robustness of our proposed CoinGAN to inaccuracies and variability in image-level labels (Reviewer SjzH (Q3)).
- Comparison with a state-of-the-art domain-specific diffusion-based model (Reviewer EB1C (W2)).
- Exploration of the progression of mediastinal lesions with labels reflecting severity levels rather than binary presence/absence (Reviewer LD6Q (Q3)).


Furthermore, we have provided responses to each reviewer's comments individually and in detail below, which will also be reflected in the revised manuscript. If there are any additional questions or comments, we stand ready and eager to engage in further discussions.

We are currently revising the paper and will submit the updated version shortly.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a weakly-supervised semantic segmentation framework that combines contrastive convolution and dual attention mechanisms (CSA), achieving state-of-the-art performance on medical imaging datasets. However, reviewers raised concerns about the clarity, originality, and generalizability of the method. The ideas presented are not novel, as they build on existing work, and the paper lacks adequate comparisons with recent GAN- and diffusion-based techniques. Overall, the paper was considered to make incremental contributions and suffers from insufficient clarity, requiring significant revisions before being ready for publication. As a result, the consensus was to reject the paper.