G4Seg: Generation for Online Segmentation Refinement with Diffusion Models

Authors: Tianjiao Zhang, Fei Zhang, Jiangchao Yao, Ya Zhang, Yanfeng Wang
Primary area: generative models
Keywords: diffusion models, inexact segmentation, semantic correspondence

Abstract:
This paper considers the problem of utilizing a large-scale text-to-image diffusion model to tackle the challenging Inexact Segmentation (IS) task. Unlike traditional approaches that rely heavily on discriminative-model-based paradigm or dense visual representations derived from internal attention mechanisms, our method focuses on the intrinsic generative priors in Stable Diffusion~(SD). Specifically, we exploit the pattern discrepancies between original images and mask-conditional generated images to facilitate a coarse-to-fine segmentation refinement by establishing a semantic correspondence alignment and updating the foreground probability. Comprehensive quantitative and qualitative experiments validate the effectiveness and superiority of our plug-and-play design, underscoring the potential of leveraging generation discrepancies to model dense representations and encouraging further exploration of generative approaches for solving discriminative tasks.

Decision: withdrawn
Reviewer scores: 5, 6, 6, 5, 5

Reviews and discussion:
<thread>
creation date:
1730276846675

thread_type:
official_review

<thread_object>
creator: reviewer_xtz2

<content>
summary:
In this paper, the authors  exploits the discrepancies between original images and coarse mask-conditional generated images to improve mask quality. The proposed method has performance gain on both open-vocabulary and weakly-supervised task.

strengths:
(1) The motivation of reducing discrepancies between original images and coarse mask-conditional generated images is easy for understanding.

(2) The proposed method can improve the segmentation quality of existing open-vocabulary and weakly-supervised methods.

weaknesses:
(1) The contribution of this paper is limited, and the idea of reducing discrepancies is relatively incremental.

(2) The proposed method focuses on discrepancies. For irregular objects, it is difficult to give an accurate generation. Does the proposed method have negative results on these situations.

(3) The proposed method has limited improvement by performing a forward diffusion denoising process. 

(4) The proposed method is a refinement processing, which lacks of comparison with existing mask refinement method, such as CRF.

(5) It lacks the analysis about the hyper-parameters like injection weight and mixing coefficient. For implementation details, the settings on different tasks are different.

(6) The proposed method seems increasing the computational cost during inference according to Table 5.

questions:
(1) The contribution and novelty could be improved

(2) More discussions about the discrepancies, and more experimental comparison
</content>
</thread_object>

</thread>
<thread>
creation date:
1730356699231

thread_type:
official_review

<thread_object>
creator: reviewer_a25d

<content>
summary:
This paper proposes an effective method G4Seg based on Stable Diffusion to tackle the Inexact Segmentation task. This method utilizes the difference between the generated and original image to help refine the mask. By establishing a semantic correspondence alignment, the foreground probability of the confusion area can be effectively updated. Experiments on various benchmarks validate the effectiveness and superiority of this training-free approach.

strengths:
(1) The idea that using the difference between the generated and original image to help refine the mask is interesting.

(2) This paper is generally well-written and easy to follow.

(3) The introduction of G4Seg is comprehensive and detailed.

(4) G4Seg is training-free and easy-to-use.

(5) The framework is tested across multiple benchmarks, including TSSS and WSSS, showcasing its versatility and effectiveness under different settings.

weaknesses:
(1) G4Seg is based on Stable Diffusion and suffers from heavy time costs, resulting in poor practicality especially when inference at scale.  It would be better to report the specific inference cost for each dataset, not be confined to VOC but the complex COCO.

(2) As shown in the figures, the refinement of confusion areas is limited and imperfect. There are still a certain amount of pixels that G4Seg can not correct. What is the inherent reason for this phenomenon? The struggles to select confusion areas or failures in semantic correspondence alignment? Meanwhile, G4Seg should be compared to other mask refinement techniques such as training-free dense CRF and training-based CascadePSP [1], SegRefiner[2]. It seems that dense CRF is superior and more efficient than G4Seg. For training-based methods, I understand that these works may use additional pixel-level annotations for training, but the comparison can help readers understand the gap between G4Seg and SOTA mask refinement techniques.

(3) The feature maps in Stable Diffusion are heavily downsampled (e.g., 64x64). Does G4Seg work for small objects?

(4) VOC is not an ideal dataset to validate the boundary accuracy because areas near the boundary are labeled as “void” in GT masks. Datasets with more boundary-accurate annotations, like re-labeled VOC and BIG used in CascadePSP [1], are more appropriate. 

(5) Detailed parameter sensitivity analyses for the injection weight $\alpha$ and mixing coefficient $\beta$ are lost, which is important to assess the robustness of G4Seg.

---

[1] CascadePSP: Toward Class-Agnostic and Very High-Resolution Segmentation via Global and Local Refinement.

[2] SegRefiner: Towards Model-Agnostic Segmentation Refinement with Discrete Diffusion Process.

questions:
In Table 6, how to inject boxes, points and scribbles into Stable Diffusion?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730468394098

thread_type:
official_review

<thread_object>
creator: reviewer_adsn

<content>
summary:
This paper presents an innovative diffusion-based approach to address the Inexact Segmentation (IS) task by leveraging the intrinsic generative priors inherent in Stable Diffusion. The authors propose a coarse-to-fine segmentation refinement strategy that exploits pattern discrepancies between original images and mask-conditional generated images. This is achieved through the establishment of semantic correspondence alignment and iterative updating of foreground probabilities. A key contribution of this work is the introduction of a semantic correspondence alignment methodology, which employs the pixel-wise Hausdorff distance as a discrepancy metric. The efficacy of the proposed method is demonstrated through extensive experimentation in both open-vocabulary and weakly supervised segmentation tasks. Notably, the approach consistently outperforms current state-of-the-art methods, showcasing substantial performance gains across various benchmarks.

strengths:
1.The proposed G4SEG introduces an innovative training-free framework for inexact segmentation refinement using generative models. This approach represents a significant departure from previous discriminative-based and diffusion model (DM)-based training methods, offering a novel perspective in the field.

2. The concept of explicit mask projection and semantic correspondence alignment is particularly noteworthy. By ingeniously decoupling the mask refinement process from target image reconstruction, the authors present a unique solution to the segmentation refinement problem.

3. The method demonstrates consistent performance gains when incorporated with current state-of-the-art approaches in both open-vocabulary and weakly supervised segmentation tasks. This broad applicability underscores the effectiveness and versatility of the proposed technique.

weaknesses:
1. The reliance on Stable Diffusion may potentially limit the model's efficiency compared to baseline methods, raising concerns about computational requirements and processing speed.

2. The improvements are most pronounced when applied to less accurate segmentation models, such as GroupVit, which is trained solely with text labels. When used with stronger base segmentation models (e.g., SCLIP and DiffSegmenter), the performance gains appear to be less substantial, potentially limiting the method's applicability across a broader range of models.

questions:
1. How does the proposed method handle cases where the initial segmentation is significantly inaccurate or contains multiple errors? Is there a threshold of initial accuracy below which the refinement process becomes less effective or unreliable?

2. Given that the method leverages generative priors from Stable Diffusion, how sensitive is it to domain shifts or out-of-distribution images? Have the authors explored its performance on datasets or image types that differ significantly from those used in training Stable Diffusion?

3. Could this approach be extended or adapted to those fully-supervised methods that can already achieve accurate segmentation results, such as SegFormer and Mask2Former?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730598435059

thread_type:
official_review

<thread_object>
creator: reviewer_ervm

<content>
summary:
The paper presents G4Seg, a novel approach leveraging large-scale pretrained diffusion models for refining inexact segmentation without additional training. The method capitalizes on discrepancies between an original image and its generated counterpart conditioned on a coarse segmentation mask to refine segmentation results. By applying a semantic correspondence alignment and employing pixel-wise probability updates, G4Seg proposes a training-free, plug-and-play solution applicable to weakly supervised and text-supervised semantic segmentation tasks.

strengths:
1. The proposal to use generative discrepancies from diffusion models for segmentation refinement is unique and well-explored, emphasizing the potential of generative models in traditionally discriminative tasks.

2. The method’s independence from training requirements is noteworthy, making it resource-efficient and broadly adaptable.

3. Comprehensive evaluation across standard datasets (PASCAL VOC12, PASCAL Context, MS COCO) demonstrates consistent performance improvements, achieving state-of-the-art results in certain scenarios.

weaknesses:
1.  While results are promising, deeper comparisons with similar methods (e.g., training-free segmentation approaches) could be emphasized to highlight advantages and potential trade-offs more explicitly.

2. Although tested on popular benchmarks, it is not entirely clear how G4Seg would scale to other datasets.

3. While computational cost is discussed, there is limited exploration of scenarios where G4Seg may not perform optimally (e.g., highly cluttered or ambiguous images).

questions:
Please see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730695325073

thread_type:
official_review

<thread_object>
creator: reviewer_eypu

<content>
summary:
This paper proposes an image segmentation method using a pretrained text-to-image diffusion model, specifically Stable Diffusion 2.1. Given a text prompt with a class label, which conditions the image generation in UNet layers, it computes a semantic difference between the original image and a mask-conditional generated image after a one-step denoising step. This discrepancy, which is computed by Hausdorff distance, is employed to update the foreground probability of each pixel. Initial coarse segmentation follows the settings in the selected baselines.

strengths:
This paper provides a training-free method for open vocabulary single-class segmentation.

weaknesses:
I carefully reviewed the paper. Rather than an extensive list of less critical and debatable issues, I will concentrate on the most significant concerns that influence my rating. My focus will be on the following three limitations:

1) The reliance on a specific class label in the text prompt limits this method to only a single category for a given image. It is not clear how this method extends multiple classes. Running the proposed method multiple times with multiple yet separate class labels would face the problem of post-consolidating, most likely handling contrasting segmentation results outside the diffusion steps, which seems non-optimal.  

2) The main idea is that, given an imperfect segmentation mask, the generated image will have a discrepancy from the original image. This drives the intuition given in the paper that with a more accurate segmentation mask, the probability of generating the original image is more likely to be maximized. How this is implemented is described in 3.2.2 and Appendix E. Some explanation is given 317-323; however, the explanation also highlights the reliance on a good initial segmentation mask. The ablation study does not present how sensitive the proposed method is to the initial segmentation accuracy.    

3) The segmentation mIoU improvement is marginal. Besides, the comparisons with stronger SOTA are missing OVAM (CVPR 2024) and CoDe (CVPR 2024). DeOP (ICCV 2023), which could be another interesting baseline. 

I am open to reconsidering my rating based on a satisfactory rebuttal from the authors.

questions:
How could this method extend to other UNet-based text-to-image diffusion models beyond SD2.1 (such as Wurstchen, SSD-1B, SDXL, SD1.5, etc.) and their distilled versions (such as single or reduced step variants, LCM versions)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732171614866

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We sincerely thank the reviewers for their thoughtful feedback and efforts in evaluating our work. Your valuable suggestions have greatly helped us improve the quality of our paper. We also deeply appreciate the substantial efforts invested by all the ACs and PCs in reviewing our work at the same time.



We are inspired by the encouragement from the reviewers, who evaluate our work as **resource-efficient**(ervM), **broadly adaptable**(ervM,XTz2) and **offering novel perspective**(ADSn). In response to the comments and concerns raised by the reviewers, before providing detailed responses to each reviewer individually, we also summarize the common main points here:



- **Reliance on the quality of coarse mask before refinement**.
We have conducted comprehensive experiments of our methods regarding the initial segmentation mask quality. The results indicate that our method shows limited improvement when the quality of the initial mask is very low. However, we want to emphasize that **this is a common limitation for many segmentation post-processing methods**. Moreover, by employing more advanced mask injection techniques, the improvement of our method can be significantly enhanced. More detailed discussion and experiments could be found in the response to reviewer EypU and XTz2.


- **Comparison with other segmentation refinement methods**.
We have added comparisons between our method and other mask refinement algorithms to demonstrate the effectiveness of our approach in improving inexact segmentation. Here, fundamentally different from both pixel-level supervised methods (CascadePSP/SegRefiner) and image prior-based methods (Dense CRF), **the knowledge source G4Seg relies on to enhance segmentation performance is the generative capacity of pre-trained diffusion model and the pattern discrepancy between the original and mask-conditioning reconstructed images**, making it **capable of achieving even higher performance when combined with other methods**, as shown in the following table. More detailed discussion could be found in response to reviewers XTz2 and a25d.



- **Computational cost concerns**.
Our approach is a segmentation post-processing method involving a pre-trained diffusion model. First, the computational cost of our method is **within a reasonable range compared to other segmentation post-processing methods**. Second, in scenarios requiring rapid application, the pre-trained diffusion model can be lightweight through techniques such as **pruning or distilling into a smaller model**.  More detailed discussion and experiments could be found in the response to reviewer ADSn and ervM.

| Methods             | SCLIP VOC      | SCLIP Context  | MaskCLIP VOC   | MaskCLIP Context |
| ------------------- | -------------- | -------------- | -------------- | ---------------- |
| Baseline            | 59.1           | 30.4           | 38.8           | 23.6             |
| +G4Seg              | 59.8(+0.7)     | 31.3(+0.9)     | 39.4(+0.6)     | 24.1(+0.5)       |
| +SegRefiner         | 59.3(+0.2)     | 30.7(+0.3)     | 39.1(+0.3)     | 23.9(+0.3)       |
| +CascadePSP         | 59.5(+0.4)     | 30.9(+0.5)     | 39.2(+0.4)     | 23.8(+0.2)       |
| +Densecrf           | 60.9(+1.8)     | 31.2(+0.8)     | 39.9(+1.1)     | 24.2(+0.6)       |
| +G4Seg + CascadePSP | 60.1(+1.0)     | 31.6(+1.2)     | 39.5(+0.7)     | 24.3(+0.7)       |
| +**G4Seg+Densecrf** | **62.1(+3.0)** | **32.0(+1.6)** | **40.1(+1.3)** | **24.6(+1.0)**   |
</content>
</thread_object>

</thread>


Metareview: