DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut
Authors: Paul Couairon
Mustafa Shukor
Jean-Emmanuel HAUGEARD
Matthieu Cord
Nicolas THOME
Venue: NeurIPS 2024 poster (2024)
Keywords: Segmentation
Image Segmentation
Zero-Shot Segmentation
Diffusion Features
Normalized Cut
Primary area: machine_vision
TL;DR: We propose DiffCut, a novel zero-shot image segmentation method based on a recursive normalized cut algorithm to segment diffusion features

Abstract: Foundation models have emerged as powerful tools across various domains including language, vision, and multimodal tasks. While prior works have addressed unsupervised semantic segmentation, they significantly lag behind supervised models. In this paper, we use a diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method that solely harnesses the output features from the final self-attention block. Through extensive experimentation, we demonstrate that using these diffusion features in a graph based segmentation algorithm, significantly outperforms previous state-of-the-art methods on zero-shot segmentation. Specifically, we leverage a recursive Normalized Cut algorithm that regulates the granularity of detected objects and produces well-defined segmentation maps that precisely capture intricate image details. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve as foundation vision encoders for downstream tasks.

DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut
Paul Couairon 1.2
Mustafa Shukor'
Jean-Emmanuel Haugeard
Matthieu Cord 1.3 Nicolas Thome?
Sorbonne Université, CNRS, ISIR, F-75005 Paris, France Thales, TSGF, cortAlx Labs, France
Valco.ai
Abstract
Foundation models have emerged as powerful tools across various domains includ- ing language, vision, and multimodal tasks. While prior works have addressed unsupervised semantic segmentation, they significantly lag behind supervised mod- els. In this paper, we use diffusion UNet encoder as a foundation vision encoder and introduce DiffCut, an unsupervised zero-shot segmentation method that solely harnesses the output features from the final self-attention block. Through extensive experimentation, we demonstrate that using these diffusion features in graph based segmentation algorithm, significantly outperforms previous state-of-the-art methods on zero-shot segmentation. Specifically, we leverage recursive Normal- ized Cut algorithm that regulates the granularity of detected objects and produces well-defined segmentation maps that precisely capture intricate image details. Our work highlights the remarkably accurate semantic knowledge embedded within diffusion UNet encoders that could then serve foundation vision encoders for downstream tasks. Project page: https:/ddffcutssegmentatiiinon
DifCCut
Ref. Image
DiffSeg
Figure 1: Unsupervised zero-shot image segmentation. Our DiffCCtt method exploits features from diffusion UNet encoder in a ggaph-based recursive partitioning algorithm. Compared DiffSeg [1] DiffCut provides finely detailed segmentation maps that more closely align with semantic concepts.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

Introduction
Foundation models have emerged as powerful tools across various domains, including language
[2, 3, 4J, vision [5, 6, 7], and multimodal tasks [8, 9, 10, 11, 12, 13]. Pretrained on extensive datasets, these models exhibit unparalleled generalization capabilities, marking a significant departure from training models from scratch to efficiently adapting pretrained foundation models [14, 15, 16, 17]. Utilizing pretrained models is particularly vital for dense visual tasks, alleviating the need for large annotated datasets specific to each domain. While prior works [18, 19, 20, 21, 22] have addressed unsupervised image segmentation, they significantly lag behind supervised models [23, 24, 25, 26]. Recently, SAM [27], proposed model that can produce fine fine-grained class-agnostic masks which achieves outstanding zero-shot transfer to any images. Still, it requires a huge annotated segmentation dataset well as significant training resources. Therefore, in this work, investigate alternative direction: unsupervised and zero-shot segmentation under the most constraining conditions, where no segmentation annotations or prior knowledge on the target dataset are available.
Recently, several methods have emerged to address unsupervised object detection by framing it as a graph partitioning problem, utilizing self-supervised ViT features [28, 5]. LOST [29] proposes to localize a unique object in a image by exploiting the inverse degree information to find a seed patch. TokenCut [30] splits the graph in two subsets given bipartition. FOUND [31] and MaskCut [32] extend these approaches by addressing the single object discovery limitation. While being able to localize multiple objects, the latter methods remain constrained to identify pre-determined number
of objects, making them ill-suited for a task of unsupervised image segmentation which inherently requires to adapt the number of segment uncover to the visual content.
Conversely, text-to-image diffusion models [33, 34, 35] can produce high-quality visual content from textual descriptions [36, 37, 38). indicating implicit learning of wide range of visual concepts. Recent works have tried to leverage diverse internal representations of such models for localization segmentation tasks. Several methods [39, 40, 41, 42, 43] opt to exploit image-text interactions within cross-attentic modules but are ultimately constrained by the need for meticulous input prompt design. Concurrently, [44] identifies semantic correspondences between image pixels and spatial locations of low-dimensional feature maps by modulating cross-attention modules. This method proves to be computationally intensive as it requires numerous forward inferences. On the other hand, DiffSeg [1] segment images by iteratively merging self-attention maps which only depict local correlation between patches.
In this work, we introduce DiffCut, new method for zero-shot image segmentation which solely harnesses the encoder features of pre-trained diffusion model in a recursive graph partitioning algorithm to produce fine-grained segmentation maps. Importantly, our method does not require any label from downstream segmentation datasets and its backbone has not been pre-trained dense pixel annotations such as SAM [27]. We observe in Fig. I that DiffCut produces sharp segments that nicely outline object boundaries. In comparison with the recent sfate-of-the-at unsupervised zero-shot segmentation method DiffSeg [1], the segments yielded by DiffCut, are better aligned with the semantic visual concepts, e.g. DiffCut is able uncover the urban area well as the boats in the middle row image. Our main contributions are as follows:
We leverage the features from the final self-attention block of a diffusion UNet encoder, for the task of unsupervised image segmentation. In this context, demonstrate that exploiting the inner patch-level alignment yields superior performance compared to merging self-attention maps as done in DiftSeg [1].
Compared existing graph based object localization methods e.g. TokenCut or MaskCut [30, 32], we push further and take advantage of a recursive Normalized Cut algorithm to generate dense segmentation maps. Via partitioning threshold, the method is able to regulate the granularity of detected objects and consequently adapt the number of segments to the visual content.
We perform extensive experiments to validate the effectiveness of DiffCut and show that it significantly outperforms state-of tfate-of-the-an art methods for unsupervised segmentation on standard benchmarks, reducing the gap with fully supervised models.
In addition, we exhibit the remarkable semantic coherence emerging our chosen diffusion features by measuring their patch-level alignment, which surpasses other backbones such CLIP [8] or

---

DINOv2 [5]. Our ablation studies further reveal the relevance of these diffusion features as well as the recursive partitioning approach which proves to provide robust segmentation performance. Finally, we show that DiffCut can be extended to an open open-vocabulary setting with a straightforward process leveraging a convolutional CL.IP, which even tops most dedicated methods on this task.
2 Related Work
Semantic segmentation. Semantic segmentation consists in partitioning an image into a set of segments, each corresponding to specific semantic concept. While supervised semantic segmentation has been widely explored [45, 46, 47, 27], unsupervised and zero-shot transfer segmentation for any images with previously unseen categories remains significantly more challenging and much less investigated. For example, most works in unsupervised segmentation require access to the target data for unsupervised adaption [21, 20, 19, 18]. Therefore, these methods cannot segment images that are not seen during the adaptation. Recently, DiffSeg [1] moved step forward by proposing an unsupervised and zero-shot approach that can produce quality segmentation maps without any prior knowledge on the underlying visual content.
Segmentation with Text Supervision. Recent works have shown that learning accurate segmenta- tion maps is possible with text supervision, overcoming the cost of dense annotations. These works are mostly based on image-text contrastive learning [48, 49, 50, 51], and usually exploit the features of CLIP [52, 53, 54]. MaskCLIP [52] leverages CLIP to get pseudo labels used to train a typical image segmentation model. ReCO [53] uses CLIP for dataset curation and get a reference image embedding for each class that is used to obtain the final segmentation. CLIPpy [48] proposes minimal modifications to CLIP to get dense labels. SegCLIP [54] continues to train CLIP with additional reconstruction and superpixel-basee KL loss to enhance localization. TCL [50] learns a region-text alignment to get precise segmentation masks. ViT [49] also learns masks from text supervision
and is based on a hierarchical grouping mechanism. Similarly, ViewCo [51] proposes a contrastive learning between multiple views/crops of the image and the text.
Graph-based Object Detection. Built on top of self self-supervised VIT features, various methods frame the problem of object detection as graph partitioning problem. LOST [29] aims at detecting salient object in an image using the degree of the nodes in the graph and seed expansion mechanism. Based on Normalized Cut (NCut) [55], FOUND [31] proposes identify all background patches, hence discovering all object patches as a by-product with no need for a prior knowledge of the number of objects their relative size with respect to the background. TokenCut [30] detects one single salient object in each image with a unique NCur bipartition. In an attempt to adapt TokenCut to multi-objects localization, MaskCut [32] first localizes an object and disconnects its corresponding patches to the rest of the graph before repeating the process a pre-determined number of times. As these graph partitioning methods are only able to uncover fixed number of segments, they are inadequate for task of image segmentation.
Segmentation with Diffusion Models. Diffusion models can produce high-quality visual content given a text prompt, indicating implicit learning of wide range of visual concepts and the ability of grounding these concepts in images. Therefore their internal representations appear as good candidates for visual localization tasks [56, 57, 58). ODISE [59] is one of the first training-based approaches to build a fully supervised panoptic image segmentor on top of diffusion features. Several other methods
[40, 41, 42] leverage attention modules for localization segmentation tasks. DiffuMask [42] uses the cross-modal grounding between a text input and an image in cross-attention modules to segment the referred object in a synthetic image. However, DiffuMask can only be applied to a generated image. In a zero-shot setting, [41] harnesses the image-text interaction via cross-altention score maps to complete self-attention maps and segment grounded objects. EmerDiff [44] opts not to exploit image-text interactions in cross-attention modules. Instead, it identifies semantic correspondences between image pixels and spatial locations by modulating the values of sub-region of feature maps in low-resolution cross-attention layers. These cross-attention based methods eventually prove to highly computationally intensive as multiple forward inferences are often required. On the other hand, DiffSeg [1] proposes an iterative merging process based on measuring KI. divergence among self-attention maps to merge them into valid segmentation masks. However, it appears that self-attention score maps only depict very local correlation between patches.
3

---

DiffCuut
Diffusion Models. Diffusion models [60, 61, 62] are generative models that aim to approximate a data distribution q by mapping an input noise 2T ~ NO, 1) to a clean sample 2g ~ through an iterative denoising process. In latent text-to-image (T21) diffusion models, e.g. Stable Diffusion [33], the diffusion process is performed in the latent space of a Variational AutoEncoder [63] for computational efficiency, and encode the textual inputs feature vectors from pretrained language models. Starting from noised latent vector 2 at the timestep I, denoising autoencoder eg is trained predict the noise that is added to the latent z, conditioned the text prompt The training objective writes:
[lle c (Ze, 4, (c))
where is uniformly sampled from the set of timesteps {1, ..,....2
Bilincar Upsampling
1200
Diffusion Encoder
Recursive
Masked
ang max -
2
Affinity matrix
Features Extraction
Recursive Feature Clustering
High-Resolution Concept Assignment
Figure 2: Overview of DiffCut. 1) DiffCut takes an image as input and extracts the features of the last self-attention block of diffusion UNet encoder. 2) These features are used construct an affinity matrix that serves recursive normalized cul algorithm, which outputs a segmentation map at the latent spatial resolution. 3) A high-resolution segmentation map is produced via concept assignment mechanism on the features upsampled at the original image size.
3.1 Features Extraction
An input image encoded into latent via the VQ-encoder of the latent diffusion model and a small amount of gaussian noise is added to it (not shown in Fig. 2). The obtained latent is passed to the diffusion UNet encoder, from which we only extract the output features, denoted é, from its last self-attention block. This choice design has several motivations:
Attention Limitations. In contrast to several methods that harness cross-attention modules for localization or segmentation tasks [39, 40, 41, 42], we deliberately choose not to depend on this mechanism. The accuracy of segmentation maps generated via attention modules heavily relies on the quality of the textual input which often requires an automatic captioning model combined with meticulous prompt design reach competitive performance. Besides being constrained by the maximum number of input tokens, such approach is proved to be inaccurate in the presence of cohyponyms [64] and is prone neglect subject tokens as the number of objects detect becomes large [65]. The localization and segmentation capacity with single forward inference is then constrained by the performance of the captioning model and the attention modules themselves. Exploiting only the intermediate diffusion features alleviate the computational cost of an additional captioning model and do not necessitate multiple forward inferences.
UNet Encoder Effectiveness. Previous works [66, 67, 37] have shown that diffusion features provide precise semantic information shared across objects from different domains. Building this observation, we hypothesize that the pyramidal architecture the UNet encoder capture semantically rich image representations that are well-suited for zero-shot vision tasks. To validate this assumption, we exhibit the semantic coherence emerging in the UNet encoder, evidenced by a remarkable patch level alignment in the output features of the final self-attention block. We in fact demonstrate that these features are sufficient to reach state-of-the-arr zero-shot segmentation performance.
Computational Efficiency. By solely exploiting the diffusion UNet encoder, our method offers a substantial computational gain, reducing the model size by 70% (400M vs 1.3B parameters). In contrast. DiffSeg extracts every self-attention maps of the UNet which requires full model inference.

---

3.2 Recursive Feature Clustering
Normalized Cut treats image segmentation as a graph partitioning problem [55]. Given a graph G = (V.E) where V and E are respectively a set of nodes and edges, we construct an affinity matrix W such that W is the edge between node v and vy. and a diagonal degree matrix D, with d(i) = Ej Wij. NCul minimizes the cost of partitioning the graph into two sub-graphs by solving:
(D Wa DDx
to find the eigenvector x corresponding to the second smallest eigenvalue. In the ideal case, the clustering solution only takes two discrete values. Since the solution of Eq. (2) is a continuous relaxation of the initial problem, contains continuous values and splitting point has to be determined to partition it. To find the optimal partition, we examine / evenly spaced points in æ and select the one resulting in the minimum NCut value.
Graph Affinity. Based on our observations of the patch-level alignment of diffusion features illustrated in Fig. 4, we assume that the normalized cut algorithm will produce sharp segments, each corresponding to a precise semantic concept as distinct objects would manifest as weakly connected components in patch similarity matrix. Following this intuition, we construct an affinity matrix W, by computing the cosine similarity, normalized between and 1, between each pair of patches. As the NCut criterion evaluates both the dissimilarity between different segments and the similarity within each segment, we opt to emphasize segmenss dissimilarity by raising each element to a positive integer power at:
2
W 5j
Essentially, this process maintains relatively high affinity for highly similar patches, while squashing the weights between dissimilar patches towards zero. This mechanism plays the role a soft thresholding, offering more gradual adjustment compared to setting a threshold to explicitly binarize the affinity matrix as done [30] and [32].
Recursive Partitioning. Classical spectral clustering [68] requires setting a pre-defined number of clusters to partition the graph, which a significant constraint in the context of zero-shot image segmentation where prior knowledge
the visual content is available. We therefore adopt recursive graph partitioning [55], which adapts the number of uncovered segments to the visual content via threshold, denoted r, on the maximum partitioning cost. This hyperparameter stops the recursive partitioning of a segment when its NCur value exceeds it and thereby regulates the granularity of detected objects. We demonstrate that our soft thresholding process detailed above enhances the robustness of the method which delivers competitive performance across a wide range of values. This recursive clustering process is summarized in Supplementary A.
3.3 High- Resoluttsoo Concept Assignment
Thus far, we have constructed segmentation maps (e.g. 32 x2) which are 32 times lower in resolution than the original image (e.g. 1024 x 1024). The number of segments found in each image depends on the image and the value of the hyperparameter т. Our next goal is to upscale these low-resolution maps to build accurate pixel-level segmentation maps. We propose high-resolution segmentation process that can be decomposed into the following steps:
Masked Spatial Marginal Mean. First, our objective to extract a set of representations that embeds the semantics of each segment. As shown in [37], reducing the spatial dimension of diffusion features with a Spatial Marginal Mean (SMM) effectively retains semantic information and provides accurate image descriptor. In light of this, we naturally propose to collapse the spatial dimension of each segment with a Masked SMM. This process yields a collection of semantically rich embreddigemmd denoted C.
2. Concept Assignment. A naive approach to obtain segmentation maps at the original image resolution consists in performing a nearext-neighhor upsampling. Despite its straightforwardness, this approach results in blocky output structure as all pixels within the same feature patch are assigned the same concept. Alternatively, we opt to first bilinearly upsample our low-resolution feature map å to match the original image spatial size and then proceed with the pixel/concept
5

---

assignment. Specifically, for each concept E C, we compute its cosine similarity with the upsampled features mpp This yields similarity matrix of size (H W K) where =cll. Then, the assignment process simply consists in taking the argmax across the K channels. The obtained segmentation map eventually refined with pixel-adaptive refinement module [69].
Experiments
Datasets.
Following existing works in image segmentation [21, 53, 52, 18],
the following
datasets for evaluation: a) Pascal VOC [70] (20 foreground classes), b) Pascal Context [71] (59
foreground classes), e) COCO-Object [72] (80 foreground classes), d) COCO-Stuff-21 merges the 80 things and 91 stuff categories in COCO-stuff into 27 mid-level categories, e) Cityscapes [73] (27
foreground classes) and f) ADE20K (150 foreground classes) [74]. An extra background class is considered in Pascal VOC, Pascal Context, and COCO-Object. We ignore their training sets and directly evaluate our method the original validation sets, the exception of COCO for which evaluate on the validation split curated by prior works [21, 19].
Metrics. For all datasets, we report the mean intersection over union (mloU), the most popular evaluation metric for semantic segmentation. Because our method does not provide a semantic label, we use the Hungarian matching algorithm [75] assign predicted masks to ground truth mask. For datasets including a background class, we perform a many-to-one matching to the background label (Supplementary H). As [1], we emphasize unsupervised adaptation (UA), language dependency (LD). and auxiliary image (AX). UA means that the specific method requires unsupervised training on the target dataset. Methods without the UA requirement are considered zero-shot. LDD means that the method requires text input, such as descriptive sentence for the image, to facilitate segmentation. AX means that the method requires an additional pool of reference images or synthetic images.
Implementation details. DiffCut builds on SSD- 1B [35]. a distilled version of Stable Diffusion
XL [34]. The model takes an empty string as input and we set the timestep for denoising = 50. To ensure fair comparison when evaluating our method against baselines, we set a unique value for r and a across all datasets, equal to 0.5 and 10 respectively. Following previous works, we make of PAMR [69] to refine segmentation masks. Our method runs single NVIDIA TITAN RTX (24GB) with input images of size 1024 x 1024 and can segment an image in one second.
4.1 Results on Zero-shot Segmentation
Tab. reports the mloU score for each baseline across the 6 benchmarks. Note that the numbers
shown for COCO-Stuff and Cityscapes are taken from [1]. We complete ReCo [53] and MaskCLIP
[52] scores with the results obtained in [50]. Other numbers are taken from [18]. We also note that
DiffSeg tunes the sensible merging hyperparameter on subset of images from the training set from
the respective datasets. For fair comparison, we evaluate the method fixing it to I, recommended
in the original paper, and refine the obtained masks with PAMR. This baseline is denoted DiffSeg'.
Table 1: Unsupervised segmentation results. Best method in bold, second is underlined. COCO-Stuff-27
Model
AX
UA
vOC
Context COCO-Object
Cityscapes
ADE20K
Extra- Fran-Trraning
IIC [19]
X
V
9.8
6.7
6.4
MDC [76]
V
9.8
PICIE [21]
X
13.8
12.3
PICIE+H [21]
X
14.4
EAGLE
V
27.2
U2Seg [78]
X
/
30,2
STEGO [20]
X
V
28.2
21.0
ACSeg [18]
28.1
Training-free ReCO [53]
V
X
25.1
19.9
15.7
26.3
19.3
11.2
MaskCLIP [52]
X
X
38.8
23.6
20.6
19.6
10.0
9.8
MuskCut (At 5) [321
43.4
30.1
41.7
18.7
35.7
DiffSeg [i]
X
43.6
21.2
DiffSeg'
X
X
49.8
48.8
23.2
44.2
16.8
37.7
DiffCut (Ours)
X
X
X
65.2
56.5
34.1
49.1
30.6
44.3

---

With our set of default hyperparameters, DuffCut significantly outperforms all other baselines despite not relying on language dependency, auxiliary images or unsupervised adaptation. On average. our method achieves gain of +7.3 mloU over the second best baseline. Notably, DiffCut exceeds MaskCut with an average improvement of +9.4 mloU. Additionally, it outperforms the previous state-of-the-an method in unsupervised segmentation, DiffSeg, by +5.5 mloU on COCO-Stuff and +9.4 mloU on Cityscapes. The superiority of DiffCut over these two methods demonstrates our two key contributions: the high quality of our visual features for semantic segmentation and the flexibilityy of the recursive NCut algorithm in adjusting the number segments according to the visual content of each image. The effectiveness of our method is further corroborated by our qualitative results shown in Fig. 1. comparison to DiftSeg, DiffCut provides finely detailed segmentation maps that more closely align with semantic concepts. Additional examples can be found in Supplementary N.
We note here that, as the granularity of annotations varies across target datasets, our fixed set of hyperparameters can not be in the optimal regime on each of them. Therefore, relaxing the condition on prior knowledge about the target dataset, we report in Supplementary G results of DiffCut where is loosely tuned using small subset of annotated images from the target training split.
4.2 Semantic Coherence in Vision Encoders
As good candidates for a task of unsupervised segmentation expected to be semantically coherent, we conduct a comparison between different families of foundation models on their internal alignment the patch-level. Selected models include text-to-image DMs (SSD-1B [35J), text-aligned contrastive
models (CLIP [79], SigLIP [80]) and self-supervised models (DINO [28], DINOv2 [5]_ At the exception of DINO-ViT-B/I evaluated models are of roughly similar size. approximately 300M parameters for DINOv2, CLIP-ViT-L/14 and HigLIP-iTT-L/1 and 400M for SSD-IB UNet encoder.
As in [81], we collect patch representations from var- ious vision encoders and store their corresponding target classes using the segmentation labels. Given & E(x1); E Rh and 2 = E(x2); E RP,, the patch representations of images X1 and X2 at respec- tively index and j. we compute their cosine similar- ity and use this score
1.0
0.6
DENUAVEERTI (AUCHUUSH)
binary classifier to predict if the two patches belong to the same class. Given ((x1): ); and (xx2),. the labels associated the patches, if ((x(((())) ) = 1(x2))n.) the target value for binary classi- fication is 1, else O. We present in Fig. 3 the ROC curve and AUC score for our candidate models. We observe that SSD-1B UNet encoder [35] demonstrates a greater patch-level alignment than any other candidate model with an AUC score of 0.83, even surpassing DINOv2 [5]. We further exhibit the outstanding alignment between patch representations associated to semantically similar concepts with qualitative results in Fig. We provide additional qualitative examples patch-level alignment in Supplementary M.
DINOV2VITUA 6.CC+0.99)
0.2
SSD-18 (AUC=6.83)
CITP/ (AUC--U.T)
0.0
(AUC=2.66)
0.8
FPR
Figure 3: ROC curves revealing the se- mantic coherence of vision encoders.
Rass IMAGE
TARGET IMAGE
SSD-1B
VIT-B/16
DINON2-VIT-L114
CLIP-VIT-L/14
SICLIP-
Figure 4: Qualitative results on the semantic coherence of various vision encoders. We select patch (red marker) associated the dog in REF. IMAGE. Top row shows the cosine similarity heatmap between the selected patch and all patches produced by vision encoders for REF. IMAGE. Bottom row shows the heatmap between the selected patch REF. IMAGE and all patches produced by vision encoders for TARGET. IMAGE.

---

A potential rationale for this observation lies in the superior semantic information retention of a diffusion model compared alternative backbones, attributed to its inherent capacity to set a structural image layout, internally acquired during the training phase. These results provides insight into the strong clustering results presented in previous section, as improved semantic coherence suggests that patches belonging to the same object are more effectively clustered.
4.3 Ablation study
In this section, we perform ablation studies to validate the individual choices in the design of DiffCut.
DiffCut rs DiffSeg. Dift'Sg proposes their own clustering algorithm based on a self-attention map merging process. As the original implementation uses a different diffusion backbone as ours, we validate the benefit of our method by swapping the original SDv1.4 with our stronger SSD-IB. For a fair comparison between methods, we use the default set of hyperparameters recommended in [1] and set the default merging threshold of DiffSeg to 0.5 for all datasets. Tab. 2 clearly validates recursive graph partitioning algorithm over the
the superiority of using rich semantic features self-attention merging mechanism of DiffSeg. Qualitative results shown in Fig. further display the edge of DiffCut uncovering semantic clusters. Shown results do not make use of the mask refinement module, explaining the gap with Tab. 1.
Table 2: Ablation Study. The recursive partitioning of DiffCut yields superior results to both the self-attention merging process of DiffSeg and Automated Spectral Clustering.
Model
vOC
Context
COCO-Object COCO-Stuff-27
Cityscapes
ADE20K
DiffSeg
48.2
41.2
31.7
35.4
22.3
39.9
AutoSC
61.5
53.3
29.8
46.9
25.3
38.9
DiffCCut (w/o PAMR)
62.0
54.1
32.0
46.1
28,4
42.4
Recursive Normalized Cut Automated Spectral Clustering. In DiffCut, the hyperparameter corresponds to the maximum graph partitioning cost allowed. contrast, classical spectral clustering requires to explicitly set the number of segments to be found in the graph. To validate the benefit of the recursive approach over spectral clustering, we introduce a simple yet effective bascline dubbed AutoSC. [82] proposes heuristic that estimates the number of connected components in graph with the largest relative-eigen- confident the heuristic. In context, the index of the eigen-value that maximizes this quantity can be interpreted as the number of clusters in an image. Thus, we define a set of exponents {1. 5, 10, 15) and determine the value in this set such that its element-wise exponentiation of matrix A yields the largest Laplacian relative-eigen-gapp Then, we use the index of the eigen-value maximizing the gap as the number of clusters in a k-way spectral clustering performed with the algorithm proposed in [83]. As shown in Tab. 2, DiffCut consistently outperforms AutoSC on all datasets, with a gain up to +3.5 on ADE20K, at the exception of COCO-Stuff where the latter yields slightly better results. Noting that AutoSC is already a state-of-ther thehherrt art baseline on most benchmarks, this experiment confirms the relevance of the recursive Normalized Cut to uncover arbitrary numbers of segments.
between its Laplacian eigen-values. The larger the gap, the more
4.4 Model Analysis
Hyperparameters Impact. In this section, we assess the impact of hyperparameters т and over the segmentation performance. We report in Fig. 5 the mloU for various values, with respect to partitioning threshold values 7 ranging from 0.3 to 0.97 on Cityscapes validation set. As increases, we observe a dual effect. First, since a greater value shrinks the affinity matrix components towards 0, the partitioning cost corresponding to the NCw value decreases, explaining the shift of the optimal threshold between the different curves. Second, as a increases, the range of values for which the method yields competitive performance widens, contributing to the overall robustness of the method. For example, DiffCut outperforms our
30-
25
15
10
0.3
0.4
0.5
0.6
0.7
0.0
0.9
1.0
Figure 5: Sensitivity of DiffCut. As a increases, DiffCut shows competitive results for a broad range of 7 values.
8

---

own competitive baseline AutoSC for any т between 0.35 and 0.67 when a = 10, whereas it only surpasses it between 0.92 and 0.96 when 1. Qualitatively, we observe in Fig. 6 that as increases, the method uncovers finer segments in images.
REF. IMAGE
0.3
0.5
0.7
Figure 6: Effect of T. As T corresponds to the maximum Neut value, a larger threshold loosens the constraint the partitioning algorithm and allows it to perform more recursive steps to uncover finer objects. can be interpreted as the level granularity of detected objects.
Diffusion Features. Our chosen diffusion backbone uses UNet-based architecture which consists of an encoder &, bottle- neck B and a decoder D. The hierarchical features of the encoder, with spatial resolution spectively, are injected into the decoder D via skip connections. Considering final self-attention modules at resolution and 32 32 in the encoder and decoder, we demonstrate in Tab. 3, that the encoder's 's features extracted at the lowest spatial reso- lution retain the most semantic information and are sufficient to reach optimal performance. In addition, combining different hierarchical features does not lead to any improvements and adds to the computational burden.
Table 3: Features Contribution. Hierarchical features in Eg2 pro- vide optimal performance (Pascal VOC validation set),
128 x 128, 64 x 64 and 32 32 re-
x
E
D
VOC Test
32
64
32
64
mloU
62.0
61.6
/
/
/
60.9
Open- Vocabulary Extension. To extend Dif- ICut open-vocabulary setting, we propose in Tab. 4, straightforward approach that assigns semantic label each segmentation mask. Af- ter mask proposals are generated, image is processed by a frozen convolutional CLIP visual encoder, which produces visual representations aligned with text shared embedding space via a projection layer. The embedding of cach predicted segment is obtained by mask-pooling CLIP visual features, allowing classification against category text embeddings through con- trastive matching. Specifically, let represent the embedding of a segment, and let [t, I denote the text embeddings of category names generated by the pretrained text encoder, the pre- dicted class for this segment is determined
Table 4: Open-Vocabeee Vocabulary Segmentation. A straightforward open- vocabulary extension with CNN-based CLIP yields competitive performance.
Model
LD
VOC
Context
COCO-Objec
Extra- Frin-Tranning
ViL-Seg [84]]
37.3
18.9
TCL.[50]
55.0
30.4
31.6
CLIPpy [48]
52.2
32.0
Group [49]
52.3
22.4
24.3
ViewCo [51]
52.4
23.0
23,5
SegCLIP [54]
52.6
24.7
26.5
OVSegmentor [85]
53.8
20.4
25.1
Truining-free
RcCO [53]
25.1
19.9
15.7
MaskCLIP [52]
20.6
CLIP- DIY [86]
59.9
19.7
31.0
FreeSeg-Diff [87]
31.0
DiffCut
63.0
24.6
36.0
follows: arg max,e(1.-- .N) sof pofatmar(( (onaar(ccs(ee,... [cos(e, & cos(e, (,, 12), .. cos(e, f x)). This proposed extension reaches competitive performance, even outperforming several baselines dedicated the task of open-vocabulary zero-shot semantic segmentation.
Discussion
In this work, we tackle the challenging task of unsupervised zero-shot semantic segmentation by introducing DiffCCut, method that significantly narrows the performance gap with fully supervised

---

models. DiffCut leverages the diffusion features of UNet encoder within recursive graph parti- tioning algorithm to generate sharp segmentation maps, achieving state-of-the-un results on popular benchmarks. By reusing pre-trained models in zero-shot manner, our approach can not only reduce computational resources, energy consumption, and human labor but also align with sustainable Al practices. However, because the diffusion backbone is not specifically trained for specialized domains, such as biomedical imaging, the method may struggle with out-of-distribution images. Fine-tuning the diffusion model domain-specific data could mitigate this challenge. While fully supervised, end-to-end segmentation methods currently offer better efficiency and accuracy, further advancements could close this gap, positioning tiffusion-based UNet encoders foundation models for future vision tasks.
Acknowledgement
We thank Louis Serrano, Adel Nabli, and Louis Fournier for their insightful discussions and helpful suggestions on the article. This work was granted access to the HPC resources of IDRIS under the allocation 2024-AD01 1014763 made by GENCI. We acknowledge the financial support provided by ANRT for its funding through the CIFRE grant 2022/0817 and PEPR Sharp LANR-23-PEIA-00 ANR, FRANCE 2030).
References
[1] Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-France Diffuse, attend, and segment: Unsupervised zero-shot segmentation using stable diffusion. CVPR, 2024.
[2] Tom Brown, Benjamin Mann, Nick Ryder, Mclanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems (NeurIPS), 33:1877- 1901, 2020.
[3] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302. 23071 2023.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.0872 087744 2023.
[5] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy V. Vo. Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel HAZIZA, Francisco Massa, Alaaeldin El-Nouby, Mido Assran, Nicolas Ballas, Wojeiech Galuba, Russell Howes, Po-Yao Yo- Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Herve Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
[6] Mostafa Dehghani. Josip Djolonga. Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Peter Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. International Conference on Machine Learning, pages 7480-7512. PMLR, 2023.
[7] Alaaeldin Bl-Nouby, Michal Klein, Shuangfei Zhai, Miguel Angel Bautista, Alexander Toshev, Vaishaal Shankar, Joshua M Susskind, and Armand Joulin. Scalable pre-training of large autoregressive image models. International Conference on Machine Learning. 2024.
[8] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning. pages 8748-8763. PMLR, 2021.
[9] Jean-Baptiste Alayrac Jeff Donahue, Pauline Lue, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances neural information processing systems, 35:23716-23736 2022.
10

---

[10] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptinn Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312. 1805, 2023.
[11] Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. generalist agent. arXiv preprint arXiv:2205 220::06/75.....5 2022.
[12] Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision, language, audio, and action. arXiv preprint arXiv:2312.17172, 2023.
[13] Mustafa Shukor, Corentin Dancette, Alexandre Rame, and Matthieu Cord. Unival: Unified model for image, video, audio and language tasks. Transactions on Machine Learning Research Journal, 2023.
[14) Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. Parameter-effiiient fine-tuning methods for pretrained language models: A critical review and assessment, arXiv preprint arXiv:2312.12145 2023.
[15] Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual aug- mentation of language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22056-22069, 2023.
[16] Feng Liang, Bichen Wu, Xiaoliang Dai, Kunpeng Li, Yinan Zhao, Hang Zhang, Peizhao Zhang, Peter Vajda, and Diana Marculescu. Open-vocabulary semantic segmentation with mask-adapted clip. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7061-7070, 2023.
[17] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Conditional prompt learning for rision-language models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16816-16825, 2022.
[18] Kehan Li, Zhennan Wang, Zesen Cheng. Runyi Yu, Yian Zhao, Guoli Song. Chang Liu, Li Yuan, and Jie Chen. Acseg: Adaptive conceptualization for unsupervised semantic segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 7162-7172, 2023.
[19] Xu Ji, Joao F Henriques, and Andrea Vedaldi. Invariant information clustering for unsuper- vised image classification and segmentation. In Proceedings of the IEEE/CVF international conference computer vision, pages
2019.
[20] Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T Freeman. Unsupervised semantic segmentation by distilling feature correspondences. arXiv preprint arXiv:2203.08414 2022.
[21] Jang Hyun Cho, Utkarsh Mall, Kavita Bala, and Bharath Hariharan. Picie: Unsupervised semantic segmentation using invariance and equivariance in clustering. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 16794-16804, 2021.
[22] Qianli Feng, Raghudeep Gadde, Wentong Liao, Eduard Ramon, and Aleix Martinez. Network- free, unsupervised semantic segmentation with synthetic images. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). pages 23602- 23610. June 2023.
[23] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1299 1299, 2022.
[24] Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. One- former: One transformer rule universal image segmentation. Proceedings of the IEEF/VVF Conference Computer Vision and Pattern Recognition, pages 2989-2998, 2023.
11

---

[25] Jitesh Jain, Anukriti Singh, Nikita Orlov, Zilong Huang, Jiachen Li, Steven Walton, and Humphrey Shi. Semask: Semantically masked transformers for semantic segmentation. In Proceedings of the IEEF/CVF International Conference Computer Vision, pages 752-761, 2023.
[26] Wenhai Wang, Jifeng Dai, Zhe Chen, Zhenhang Huang, Zhiqi Li, Xizhou Zhu, Xiaowei Hu, Tong Lu, Lewei Lu, Hongsheng Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14408-1444 2023.
[27] Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Wen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4022 2023.
[28] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in superviue vision transformers. In Proceedings of the IEEE/CVF international conference computer vision, pages 9650-9660, 2021.
[29] Oriane Siméoni, Gilles Puy, Huy V Vo, Simon Roburin, Spyros Gidaris, Andrei Bursuc, Patrick Pérez, Renaud Marlet, and Jean Ponce. Localizing objects with self-supervised transformers and no labels. In BMVC 2021-32nd British Machine Vision Conference, 2021.
[30] Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self- supervised transformer and normalized cut. IEEE transactions on pattern analysis and machine intelligence, 2023.
[31] Oriane Siméoni, Chloé Sekkat, Gilles Puy, Antonín Vobecky, Éloi Zablocki, and Patrick Pérez. Unsupervised object localization: Observing the background to discover objects. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 3176-3186, 2023.
[32] Xudong Wang. Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3124-3134, 2023.
[33] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[34] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
[35] Yatharth Gupta, Vishnu V Jaddipal, Harish Prabhala, Sayak Paul, and Patrick Von Platen. Progressive knowledge distillation of stable diffusion using layer level loss. arXiv preprint arXiv:2401.02677 2024.
[36] Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. In The Twelfth International Conference on Learning Representations, 2023.
[37] Danah Yatim, Rafail Fridman, Omer Bar-Tal, Yoni Kasten, and Tali Dekel. Space-time diffusion features for zero-shot text-driven motion transfer. arXiv preprint arxiv:231..17000 2023.
[38] Paul Couairon, Clément Rambour, Gean-Emmanuee Haugeard, and Nicolas Thome. Videdit: Zero-shot and spatially aware text-driven video editing. Transactions on Machine Learning Research, 2024.
[39] Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Open- vocabulary object segmentation with diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 7667-7676, 2023.
12

---

[40] Chaofan Ma, Yuhuan Yang, Chen Ju. Fei Zhang, Jinxiang Liu, Yu Wang, Ya Zhang, and Yanfeng Wang. Diffusionseg: Adapting diffusion towards unsupervised object discovery. CoRR, abs/2303.09813, 2023.
[41] Jinglong Wang, Xiawei Li, Jing Zhang, Qingyuan Xu, Qin Zhou, Qian Yu, Lu Sheng, and Dong Xu. Diffusion model is secretly a training-free open vocabulary semantic segmenter. arXiv preprint arXiv:2309.02) 02773, 2023.
[42] Weijia Wu, Yuzhong Zhao, Mike Zheng Shou, Hong Zhou, and Chunhua Shen. Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 1206-1217, 2023.
[43] Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion- based synthetic data generation for pixel-level semantic segmentation. Advances in Neural Information Processing Systems, 36, 2024.
[44] Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerg- ing pixel-level semantic knowledge in diffusion models. In The Twelfth International Conference on Learning Representations, 2024.
[45] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1290-1291 12000129,,2200 2022.
[46] Huiyu Wang, Yukun Zhu, Hartwig Adam, Alan Yuille, and Liang-Chieh Chen. Max-deeplab: End-to-end panoptic segmentation with mask transformers. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages 5463 1463-5574, 2021.
[47] Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is all you need for semantic segmentation. Advances in neural information processing systems, 34:17864- 17875, 2021.
[48] Kanchana Ranasinghe, Brandon McKinzie, Sachin Ravi, Yinfci Yang, Alexander Toshev, and Jonathon Shlens. Perceptual grouping in contrastive vision-language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5571-5584 2023.
[49] Jiarui Xu, Shalini De Mello, Sifei Liu, Wonmin Byeon, Thomas Breuel, Jan Kautz, and Xiaolong Wang. Groupvit: Semantic segmentation emerges from text supervision. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18134-18144 2022.
[50] Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for open-world semantic segmentation from only image-text pairs. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11165-11174, 2023.
[51] Pengzhen Ren, Changlin Li, Hang Xu, Yi Zhu, Guangrun Wang, Jianzhuang Liu, Xiaojun Chang, and Xiaodan Liang. Viewco: Discovering text-supervised segmentation masks via multi-view semantic consistency. In The Eleventh International Conference Learning Representations, 2023.
[52] Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pages 696-712. Springer, 2022.
[53] Gyungin Shin, Weidi Xie, and Samuel Albanie. Reco: transfer. Advances in Neural Information Processing Systems, 35:33754-33767, 2022.
Retrieve and co-segment for zero-shot
[54] Huaishao Luo, Junwei Bao, Youzheng Wu, Xiaodong He, and Tianrui Li. Segelip: Patch aggregation with learnable centers for open-voca semantic segmentation. In International Conference on Machine Learning, pages 23033-23044. PMLR, 2023.
[55] Jianbo Shi and J. Malik. Normalized cuts and image segmentation. IEEE Trunsactions on Pattern Analysis and Machine Intelligence, 22(8))888-905 2000.
13

---

[56] Junyi Zhang, Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
[57] Shoufa Chen, Peize Sun, Yibing Song, and Ping Luo. Diffusiondet: Diffusion model for object detection. Proceedings of the IEEE/CVE International Conference on Computer Vision, pages 19830-19845) 2023.
[58] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthicu Cord. Diffedit: Diffusion- based semantic image editing with mask guidance. In The Eleventh International Conference on Learning Representations, 2022.
[59] Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open-vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2023.
[60] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 6851, 2020.
[61] Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021.
[62] Jaseha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages 2256-2265. PMLR, 2015.
[63] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint 13126: 14, 2013.
[64] Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp. Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting stable diffusion using cross attention. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Pupers), pages 5644-5659, Toronto, Canada, July 2023. Association for Computational Linguisties.
[65] Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based semantic guidance for text-to-image diffusion models. ACM Transactions on Graphics (TOG), 42(4):1-10, 2023.
[66] Grace Luo, Lisa Dunlap, Dong Huk Park, Alcksander Holynski, and Trevor Darrell. Diffusion hyperfeatures: Searching through time and space for semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
[67] Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36:1363 1389, 2023.
[68] Frederick Tung, Alexander Wong, and David A. Clausi. Enabling scalable spectral clustering for image segmentation. Pattern Recognition, 43(12):4069 4076406699466,,000 4076, 2010.
[69] Nikita Araslanov and Stefan Roth. Single-stage semantic segmentation from image labels. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 4253-4262, 2020.
[70] M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-13 136, January 2015.
14

---

[71] Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam- Cho, Seong-Wir Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmentation in the wild. In IEFF Conference on Computer Vision and Pattern Recognition (CVPR), 2014.
[72] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[73] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3213-3223, 2016.
[74] Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. International Journal of Computer Vision 1723002322, 2019.
[75] Harold W. Kuhn. The hungarian method for the assignment problem. Naval Research Logisties (NRL), 52, 1955.
[76] Mathilde Caron, Piotr Bojanowski, Armand Joulin, and Matthijs Douze. Deep clustering for unsupervised learning of visual features. Proceedings of the European conference on computer vision (ECCV), pages 132-149, 2018.
[77] Chanyoung Kim, Woojung Han, Dayun Ju, and Seong Jae Hwang. Eagle: Eigen aggrega- tion learning for object-centric unsupervised semantic segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2024.
[78] Dantong NIu, Xudong Wang. Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell. Unsupervised universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 22754, June 2024.
[79] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages $748-8763. PMLR, 2021.
[80] Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 1975-11986, 2023.
[81] Jishnu Mukhoti, Tsung- Yu Lin, Omid Poursaced, Rui Wang. Ashish Shah, Philip HS Torr, and Ser-Nam Lim. Open vocabulary semantic segmentation with patch aligned contrastive learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19413-19427 2023.
[82] Jicong Fan, Yiheng Tu, Zhao Zhang, Mingbo Zhao, and Haijun Zhang. A simple approach to automated spectral clustering. Advances in Neural Information Processing Systems, 35:9907- 9921, 2022.
[83] Anil Damle, Victor Minden, and Lexing Ying. Robust and efficient multi-way spectral clustering. arXiv preprint arXiv: 08251, 2016.
[84] Quande Liu, Youpeng Wen, Jianhua Han, Chunjing Xu, Hang Xu, and Xiaodan Liang. Open- world semantic segmentation via contrasting and clustering vision-language embedding. In European Conference on Computer Vision, pages 275-292. Springer, 2022.
[85] Jilan Xu, Junlin Hou, Yuejie Zhang, Rui Feng, Wang. Yu Qiao, and Weidi Xie. Learning open- vocabulary semantic segmentation models from natural language supervision. In Proceedings of the IEFF/CVE Conference on Computer Vision and Pattern Recognition, pages 2935-2944 2023.
15

---

[86] Monika Wysoczanska, Michaël Ramamonjisoa, Tomasz Trzciński, and Oriane Siméoni. Clip- diy: Clip dense inference yields open vocabulary semantic segmentation for-free. In Proceed- ings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1403-1413, 2024.
[87] Barbara Toniella Corradini, Mustafa Shukor, Paul Couairon, Guillaume Couairon, Franco Scarselli, and Matthieu Cord. Freeseg-dine diff: Training-free free open-vocabulary segmentation with diffusion models. arXiv preprint arXiv:2403.20105, 2024.
[88] Laurynas Karazija, Iro Laina, Andrea Vedaldi, and Christian Rupprecht. Diffusion models for zero-shot open-vocabulary segmentation. arXiv preprint arXiv:2306.00 093166 2023.
[89] Ryan Burgert, Kanchana Ranasinghe, Xiang Li, and Michael S Ryoo. Peekaboo: Text image diffusion models are zero-shot segmentors. arXiv preprint arXiv:2211.13222, 2022.
[90] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances neural information processing systems, 33:6840-6851, 2020.
[91] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint 2rXfv:2204.06115, 1(2):3, 2022.
[92] Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural information processing systems. 35:36479-36694, 2022.
[93] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25578-22522 2022.
[94] Albert Q Jiang. Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprint arXiv:2310.06825,, 2023.
[95] Théophane Vallaeys, Mustafa Shukor, Matthieu Cord, and Jakob Verbeek. Improved baselines for data-efficient perceptual augmentation of llms. arXiv preprint arXiv:2403.1 13499, 2024.
[96] Jascha fohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. In International conference on machine learning, pages
PMLR, 2015.
[97] Junyi Zhang. Charles Herrmann, Junhwa Hur, Luisa Polania Cabrera, Varun Jampani, Deqing Sun, and Ming-Hsuan Yang. A tale of two features: Stable diffusion complements dino for zero-shot semantic correspondence. Advances in Neural Information Processing Systems, 36, 2024.
16

---

DiffCut: Catalyzing Zero-Shot Semantic Segmentation with Diffusion Features and Recursive Normalized Cut Supplementary Material
A Recursive Normalized Cut on Diffusion Features
We summarize in Algorithm the recursive clustering process used in DiffCut.
Algorithm Recursive Normalized Cut on Diffusion Features
Input: Z an image, EJ0, 1[ a threshold value, a E N an exponent value.
Step 1: Features Extraction.
Encode the image I
VAE the diffusion model: = Eve(TT
Add some gaussian noise to the latent image 2.
Pass the noisy latent to the diffusion UNet encoder and extract the output features from its last self-attention block: 3 (z)
Step 2: Graph Construction.
Compute the pairwise cosine cosine-similarity similarity between patches of & to set up similarity matrix Raise to the power a each element of matrix obtain the affinity matrix (see Eq. (3)).
Determine the matrix degree D.
Step 3:
NCut Problem Solving.
Solve (D) - W )z= = >Dx for eigenvector with the second smallest eigenvalue.
Use the eigenvector with the second smallest eigenvalue bipartition the graph by finding the splitting point such that the NCur value is minimized.
Step
Recursive Partitioning.
Store the current partition and retrieve the matrices W and D associated to each sub-graph.
Recursively subdivide the partitions (Step 3) until the NCut value greater than т.
Output: M a segmentation map with the spatial resolution of &.
Impact of PAMR B
To reveal the effect of the pixel-adaptive refinement module (PAMR) on our method, we compare the segmentation results on all benchmarks both with and without it enabled.
Table 5: Impact of PAMR unsupervised segmentation.
DiffCut
VOC
Context COCO-Object COCO-Stuff
Cityscapes
ADE20K
w/o PAMR
62.0
54.1
32.0
46.1
28.4
42.4
w/ PAMR
65.2
56.5
34.1
49.1
30.6
44.3
In average, PAMR allows to gain +2.5 mloU on our segmentation benchmarks. Even though this refinement module helps to better outline the contour of objects, method still reaches state-of-the- art results on unsupervised zero-shot segmentation without it.
C Additional Comparison with Mask
DiffCut, is capable of providing dense segmentation maps and dynamically adapting the number of detected segments based on the visual content of an image. In contrast, MaskCut [32] only detect
17

---

a fixed number of segments, making it less suitable for image segmentation. This limitation arises from the use of an iterative graph partitioning approach. where graph nodes associated with detected single object and cannot be refined after detection,
objects are masked. Each segment is treated which severely restricts its ability to identify a large number of objects. To highlight the superiority of our recursive partitioning over MaskCut's ss iterative process, we present below comparison between the two methods with & the number of objects to be detected varying in 20)..
Table 6: Comparison with MaskCut. DiffCut recursive partitioning algorithm yields superior results than MaskCut iterative partitioning.
Model
VOC
Context
COCO-Object COCO-Stuff-27
Cityscapes
ADE20K
DiffCut
62.0
54.1
32.0
46.1
28.4
42.4
MaskCut (/ 3)
53.7
42.3
30.9
41.8
18.0
33.7
MaskCut (h 5)
53.8
43,4
30.1
41.7
18.7
35.7
MaskCut (A: 20)
53.8
43.5
30.0
41.5
18.0
35.6
DiffCut significantly outperforms MaskCut, regardless of the chosen value of k. DiffCut's im- provement shows our two key contributions: the effectiveness our visual features for semantic segmentation and the ability of the recursive NCut algorithm to dynamically adjust the number of segments based on the visual content of each image.
D DiffCut with Alternative Diffusion Backbones
To further display the relevance of diffusion features, show that DiffCut achieves competitive performance even when using smaller diffusion backbones than SSD-IB. Specifically, we test two alternative models: SD1.4 and SSD- >SD-Vega [35] (another distilled version of SDXL). The UNet encoder in SD1.4 .4 has 260M parameters, comprising approximately 30% of the overall UNet, while the UNet encoder in SSD-vega has 240M parameters, making up around 32% of the UNet.
Table 7: Performance of DiffCut with alternative diffusion backbones.
Model
VOC
Context
COCO-Object COCO-Stuff-27
Cityscapes
ADE20K
DiffCut SD1.4
57.5
52.8
30.0
45.2
24.5
DiffCut SSD-Vega
62.2
56.4
34.9
49,5
30.1
45.7
DiffCut SSD-1B
65.2
56.5
34.1
49.1
30.6
44.3
DiffSeg
49.8
48.8
23.2
44.2
16.8
37.7
The results obtained using SD1.4 and SSD-Vega are consistent with those achieved with SSD- While the SD1.4 UNet encoder shows slight performance drop compared to SSD-1B, DiffCut still significantly outperforms DiffSeg. Notably, with the SSD- -ega UNet encoder, DiffCut delivers performance comparable to SBD-IB, despite having only half the number of parameters.
E
Mask Upsampling
Normalized Cut algorithm does not scale well with the graph size due to the generalized eigenvalue problem solve, which hinder its use on the native image resolution (e.g., 1024 x 1024). Thus, the clustering is applied in the latent space and yields segmentation maps at the latent resolution. To obtain pixel- need
Table 8: Mask Upsampling,
Strategy
VOC Test
Concept Assignment
62.0
Nearest Upsampling
61.2
level segmentation at the original image resolution, upscale the oow-resolution maps. In Tab. 8, we compare the nearest-neighbor upsampling approach versus our concept assignment upsampling and show that our proposed method obtain better results than the naive upsampling of the segmentation masks.
F
Visual Encoders KMeans Comparison
To evaluate the potential of vision encoders for zero-shot segmentation, we compare their clustering performance with a simple KMeans algorithm. For selected vision encoders, features extracted
18

---

from the last layer. The hyperparameter K is either determined by the ground-trute (K*) for each image, or fixed across the dataset. We compute the mloU with respect to the ground truth masks using the Hungarian matching algorithm [75]. Tab. 9 shows that the diffusion UNet encoder (SSD-1B) significantly outperforms other vision encoders on Pascal VOC (20 classes and no background), COCO-Stuff-s and Cityscapes. This confirms that diffusion features are well-suited for localizing and segmenting objects. Interestingly, unsupervised models such as DINOv2 are better than CLIP models, suggesting that text-aligned features does not contain accurate localization features.
Model
K"
K
Model
K"
K
Model
K"
K
SSD-IB
79.5
70.8
SSD-IB
36.4
37.8
SSD-IB
21.4
21.0
Text-aligned
Text-aligned
Text-aligned
CLIP-VTE-BI1
68.9
59.1
CLIP-VIT-B/16
31.1
31.8
CLI--VIT-B/16
14.9
14.7
CITP-VIT-1/14
67.1
60.7
CLIP-VIT-I/14
26.4
26.6
CUI--VITL/I4
14.3
14.0
SigLIP-B/16
62.9
55.3
SigLIP-B/16
25.0
25.1
Big
13.7
13.6
IP1
62.2
54.8
SigUIP-1/16
22.5
23.1
SigLIP-1/16
11.6
11.6
Unsupervised
Unsupervise
Unsupervised
DINO
73.1
62.8
DINO
33.8
34.0
DINO
18.4
17.4
DINOv2-B/14
23.8
64.8
DINOv2-R/14
31.5
32.2
DINOv2-B/14
19.5
18.9
DINOv2-L/14
73.1
64.4
DIN0
30.9
31.5
DINOv2-L/14
18.2
18.1
(a) VOC
(b)
COCO-Stuff-27
(c) Cityscapes
Table 9: K.Mann features clustering for various vision encoders.
G Hyperparameter T tuning
We show in Sec. 4.3 that the performance of the method is highly robust with respect to the value of the threshold r. However, as the granularity of annotations varies across target datasets, the value of this threshold, fixed in our experiments, can not be in the optimal regime on each benchmark. Therefore, we relax the condition on the absence of prior knowledge about the target dataset and report in Tab. 10 results of DiffCut where r is loosely tuned using small subset of annotated images (200) from the target training split. Specifically, we estimate an adequate value for 7 with grid-search in the set (0.35, 0.55, 0.75] for COCO-Object, COCO-Stuff and Cityscapes.
Table 10: Threshold tuning. Tuned is denoted with r"
DiffCut COCO-Object COCO-Stuff-27
Cityscapes
0.5
32.0
46.1
28.4
38.7
48.6
29.8
As COCO-Object and COCO-Stuff-21 offer different level of object granularity despite corresponding to the same images, a fixed value for can not perform optimally on both benchmarks. Tuning the value of this threshold allows to infer the granularity of objects expected to be uncovered in images. For example, the estimated 7* value for COCO-Stuff227 is 0.35 whereas it is 0.75 on COCO-Object whose annotations requires detect much finer objects. For Cityscapes the initial fixed value was in the good range to yield optimal performance.
H Hungarian Matching
Given a of predicted masks, our goal is to find the best matching ground-truth labels. For each predicted mask, we compute the Intersection over Union (IoU) with every ground-truth mask and select the one with the highest IoU its optimal ground-truth match. This results pairing where each predicted mask is associated with at most one ground truth mask. In datasets that include background class, this label implicitly encompasses a variety of concepts related to "things" or "stuff," which vary depending on the dataset. Since our method generates segment for every detected object an image, one-to-one matching between a single predicted cluster and the entire background does not accurately represent the model's true categorization capabilities. Therefore, in such cases, we use a many-to-one matching approach by associating the clusters that primarily overlap with the background to its ID.
19

---

I
Image Noising
Before passing the image the diffusion UNet, predefined amount of gaussian noise, controlled by parameter called the timestep, is added to it. At timestep t 0 the input image corresponds to the original image without added noise while 1000 corresponds an image transformed into pure gaussian noise. In Fig. 7, we show the segmentation performance on the validation split of Pascal VOC for timesteps values ranging from 0 to 500. We can observe that a small amount of noise, around 50, gives the best mloU score, indicating that the best semantic features are obtained with a slightly noisy input image. We note that despite a significant drop the mloU score for = 500, DiffCut still reaches state-of-the-an segmentation performance on Pascal VOC benchmark, demonstrating high robustness of the method with respect to the noising ratio.
0.62
0.60
mIoU
0.58
0.56
0.54
0
100
200
300
400
500
Timestep
Figure mloU according to the noising timestep on Pascal VOC.
Hyperparameter Sensitivity
Fig. 8 presents an additional evaluation of how the hyperparameters and influence the segmen- tation performance on Pascal VOC. Similar to the observations in Fig. 5, we notice shift in the optimal threshold across the various curves corresponding to different a values. Besides, DiffCut shows increased robustness across a wide range of t values, achieving mloU scores exceeding 60.0 when a
In contrast, for 1, the mloU only exceeds 60.0 for between 0.91 and 0.94.
0.6
0.5
mloU
0.4
0-1
0.3
15
0.3
0.4
0.5
0.6
0.7
0.8
0.9
1.0
Figure 8: Robustness of DiffCut on Pascal VOC.
K
Runtime Comparison
Tab. 11 presents runtime comparison between DiffCut, MaskCut, and DiffSeg, which are the main baselines for graph-based image clustering and diffusion- baased zero-shot segmentation, respectively.
Table 11:
Runtime Comparison.
MaskCut 5)
DiffCut
DiffSeg
SDL.4
DiffSeg
SSD-1B
images/ sec
0.84
1.11
2.75
1.25
20

---

Visualization of the effect of 7
RES. IMAGR
T- 0.3
T 0.5
7 0.7
Figure 9: Effect of As increases, DiffCut uncover finer-grained objects.
M Semantic Coherence in Vision Encoders
THAGE
TARGET INSoE
REP: IMOGEE
TARGRR IMAGE
INAGE
TARGET IMAGE
SSD-1B
DINO- VIT-
DINOV1-VIT-L14
CLIP-VIT-L/14
SIGLIP-YIT-LAA
Figure 10: Qualitative results the semantic coherence of vision encoders. We select patch (red marker) associated to the dog in REF, IMAGE. Top row shows the cosine similarity heatmap between the selected patch and all patches produced by vision encoders for REF. IMAGE. Bottom row shows the heatmap between the selected patch in REE. IMAGE and all patches produced by vision encoders for TARGET. IMAGE.
21

---

N
Additional Visualization
DINA
ach
CENTRE
TTO
Figure 11: Examples of produced segmentation maps COCO dataset.
22

---

Figure 12: Examples of our produced segmentation maps on Pascal Context dataset.
23

---

Datasets Licenses
Pascal VOC:
Pascal Context:
https//wwww.csstaafrrdddtt /ooztee//aascal-connee///
COCO: https//roccoatassttor///hooom
License: Creative Commons Attribution 4.0 License
Cityscapes:
License: This dataset is made freely available to academic and non-academic entities for non- commercial purposes such as academic research, teaching, scientific publications, or personal experi- mentation.
ADE20K: https://groups.co csail. www.:mmitiemmmmiem cdlu/isionddattstt///EEE00// License: Creative Commons BSD-3 License
24

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Our experiments reflect the scope of the paper and the claims made in the abstract and in the introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Conclusion includes some limitations of this work and potential future devel-
opment.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiitiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
25

---

Answer:
[NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof's can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We provide detailed description of the method in Sec. as well an algorithm clearly synthesizing necessary information to reproduce the method.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, descríbing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is new model (e-g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path reproducing verifying the results. 5. Open access to data and code
26

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: The datasets used in the experiments publicly available. Code will be released.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips...
onioonnlooollll icy for more details.
While encourage the release of code and data, understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /nips. wctpublic//aeccccccccccccccc /gwpidess//////eeeeee /Coeee/mmmSsbmi cttsSuubmssiion.. ssionPolicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We detail in section Sec. the datasets splits used evaluate the method as well as the value of default hyper- parameters.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented the core paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, appendix, as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: No eπrоr bars are reported as standard validation sets are used to evaluate the method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
27

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mention in Sec, 4 the type of graphic card used to run the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code Of Ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics https :tttpp:://uuur http///eenipppeeeeee cc cwtppcccc///eeeeeeceeccc w/wthiccchiideinnes....
Answer: [Yes]
Justification:
We reviewed the Code of Ethics and conformed to it.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Our method which reuses pre-trained models in zero-shot fashion can save computational ressources as well as reduce energy consumption and human labor, contributing to more sustainable Al practices.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
28

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
We do deem that this paper poses such risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Authors of used assets (data) in particular, credited in the paper. Code will be released and original owners of assets will be properly credited.
Guidelines:
The answer NA means that the paper does
existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
29

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code will be released and documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: This paper does involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
30

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
31


Reviews:
[{"review_id": "RrckYo8W4I", "reviewer": "Reviewer_jdn9", "summary": "This paper introduces an innovative unsupervised, zero-shot image segmentation method called DiffCut. This method leverages the encoder features of a pre-trained diffusion model within a recursive graph partitioning algorithm to create finely detailed segmentation maps without requiring labels from downstream segmentation datasets. DiffCut exploits features from the last self-attention block of a diffusion UNet encoder to perform image segmentation. This approach does not rely on paired image and text data, making it suitable for unsupervised and zero-shot learning tasks. The core algorithmic innovation in DiffCut is the use of a recursive Normalized Cut (NCut) that allows the model to regulate the granularity of detected objects and consequently adapt the number of segments to the visual content of each image. Compared to existing methods like DiffSeg and other graph-based object localization techniques, DiffCut significantly outperforms in terms of the quality of segmentation maps and alignment with semantic visual concepts. The effectiveness of DiffCut was validated across multiple standard benchmarks with a focus on mIoU scores, where it consistently outperformed the state-of-the-art unsupervised semantic segmentation methods.", "questions": "Please check the weakness section.", "limitations": "The authors adequately addressed the limitations.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written and logically structured, making it easy to understand.\n\n2. The authors conducted thorough experiments to demonstrate the effectiveness of DiffCut, which achieves state-of-the-art performance in many downstream tasks.\n\n3. In contrast to earlier works that employ graph-based clustering methods for unsupervised segmentation, such as TokenCut and MaskCut, this method introduces a soft thresholding technique for constructing the affinity matrix. This approach effectively maintains high affinity between highly similar patches while reducing the weights between dissimilar patches to near zero—an improvement that addresses a limitation often neglected in previous studies.", "weaknesses": "1. **[Unsupervised Instance or Panoptic Segmentation Performance]** While the paper positions itself as effective in segmenting more objects within an image, it primarily focuses on semantic segmentation, potentially overlooking the model's capability for instance discrimination. I am interested in how the model performs in unsupervised instance or panoptic segmentation tasks, as these require differentiating individual objects or integrating both semantic and instance segmentation.\n\n2. **[Technical Contribution and Comparison with MaskCut]** MaskCut is actually capable of segmenting multiple objects per image, with the ability to set a large number of masks per image. This raises questions about the novelty and technical contributions of this paper, particularly in the absence of quantitative comparisons with established methods like TokenCut [30] or MaskCut [32].\n\n3. **[Lack of Comparisons with Previous Works]** The paper does not compare its results with some previous state-of-the-art works in unsupervised panoptic and semantic segmentation, such as U2Seg [N1] and EAGLE [N2]. Notably, [N2] also explores the use of prototypes derived from eigenvectors of feature maps. \n\n4. **[Adequacy of Low-Resolution Features]** The use of low-resolution feature maps to produce concept-embeddings may result in missing small-scale objects in the images, which could lead to the exclusion of these small objects during the \"High-Resolution Concept Assignment\" phase. This could limit the model’s effectiveness in capturing medium or small-sized objects, which are crucial for detailed image segmentation. \n\n5. **[Recursive Normalized Cuts or Multi-class Spectral Clustering]** I am curious if the authors have explored to use spectral clustering for multi-entity/object segmentation. The granularity of the segmentation masks can be controlled by using a different cluster numbers. \n\n\n[N1] Niu, Dantong, Xudong Wang, Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell. \"Unsupervised universal image segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22744-22754. 2024.\n\n[N2] Kim, Chanyoung, Woojung Han, Dayun Ju, and Seong Jae Hwang. \"EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3523-3533. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and appreciate the positive feedback.\n\n#### **Weaknesses**\n#### **1. [Unsupervised Instance or Panoptic Segmentation Performance]**\nWe have not yet evaluated DiffCut’s performance in instance or panoptic segmentation. The goal here is to demonstrate that the features of a diffusion UNet surpass those of other vision backbones in precise semantic segmentation. Given that our method can generate high-quality semantic masks under the most restrictive conditions (training-free and independent of LD, AX, and UA), future work could involve relaxing these constraints and incorporating DiffCut as a high-quality pseudo-label generator within an unsupervised training pipeline—such as proposed in U2Seg [1]. This could significantly enhance performance in unsupervised semantic, instance, and panoptic segmentation.\n\n#### **2. [Technical Contribution and Comparison with MaskCut]**\nUnlike TokenCut or MaskCut, our method can provide dense segmentation maps and adapt the number of detected segments based on the visual content of an image. MaskCut and TokenCut cannot do this because they can only detect a fixed number of segments. (l.101-104). Therefore, these methods are not well-suited for an image segmentation task. Technically, MaskCut uses an iterative graph partitioning method and masks the graph nodes associated with detected objects. As a result, each segment is treated as a single object that cannot be refined once detected, which in practice severely limits its ability to identify a large number of objects. To support this claim and following the reviewer’s suggestion, we provide below a comparison between the performance of MaskCut and DiffCut. Overall, the improvement of DiffCut highlights our two main contributions: the quality of our visual features for semantic segmentation, and the capacity of the recursive Ncut algorithm to adjust the number of segments to the visual content of each image. We would be glad to include these results in the final paper.\n\n|  Model         | VOC20 | Context | COCO-Object | COCO-Stuff-27 | Cityscapes | ADE20K |\n|----------------|-------|---------|-------------|------------|------------|--------|\n| DiffCut (ours) | **62.0**  | **54.1**    | **32.0**        | **46.1**       | **28.4**       | **42.4**   |\n| MaskCut $(k=3)$  | 53.7  | 42.3    | 30.9        | 41.8       | 18.0       | 33.7   |\n| MaskCut $(k=5)$  | 53.8  | 43.4    | 30.1        | 41.7       | 18.7       | 35.4   |\n| MaskCut $(k=20)$ | 53.8  | 43.5    | 30.0        | 41.5       | 18.0       | 35.6   |\n  \n\n#### **3.  [Lack of Comparisons with Previous Works]**\nWe thank the reviewers for these recent references. In comparison with EAGLE [2], we propose a much simpler and effective training-free pipeline that reaches higher results on COCO-Stuff and Cityscapes due to the quality of the extracted features from the diffusion backbone. Their pipeline is based on DINOv1 which we proved in Fig. 3 has worse feature correspondence than our backbone. U2Seg [1] proposes a framework to unify unsupervised semantic, instance and panoptic segmentation. This method only evaluates on COCO-Stuff-27 for unsupervised semantic segmentation. We will be glad to add these baselines to our main Tab. 1.\n\n|         | COCO-Stuff-27 | Cityscapes |\n|---------|:-------------:|:----------:|\n| DiffCut |      **49.1**     |    **30.6**    |\n| DiffSeg |      43.6     |    21.2    |\n|  EAGLE  |      27.2     |    22.1    |\n|  U2SEG  |      30.2     |      -     |\n\n#### **4.  [Adequacy of Low-Resolution Features]**\nWe empirically observe in Fig. 9, 11 and 12 that the excellent quality of the features provided by the diffusion model at a reasonably low spatial resolution $(32 \\times 32)$ effectively enables the detection of medium to small-sized objects. Increasing the value of the hyperparameter $\\tau$ can even further help to detect objects at a finer granularity. This spatial resolution of feature maps does not present a significant drawback for small object localization.\n\n\n#### **5.  [Recursive Normalized Cuts or Multi-class Spectral Clustering]**\nIn Tab. 2, we explore the use of spectral clustering for multi-object segmentation by introducing a variant of DiffCut called AutoSC. This variant adapts the number of segments based on the visual content using a heuristic called “relative-eigen-gap” which estimates the number of connected components in a graph. This estimated number of connected components, $k$, is then used to determine the number of clusters in $k$-way spectral clustering. We observe that AutoSC also achieves excellent results, often comparable to those obtained with DiffCut.\n\nWe note that the main differences between AutoSC and DiffCut appear in datasets containing small objects (e.g., COCO-Object, Cityscapes, ADE20K). DiffCut’s recursive partitioning offers flexibility in detecting small objects by allowing large segments to be further divided into smaller ones, while the “relative eigen-gap” heuristic in AutoSC is effective at uncovering the most salient clusters, potentially overlooking smaller objects.\n\n[1] Niu, Dantong, Xudong Wang, Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell. \"Unsupervised universal image segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22744-22754. 2024.\n\n[2] Kim, Chanyoung, Woojung Han, Dayun Ju, and Seong Jae Hwang. \"EAGLE: Eigen Aggregation Learning for Object-Centric Unsupervised Semantic Segmentation.\" In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3523-3533. 2024.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WfrVZL9nyS", "reviewer": "Reviewer_qiwu", "summary": "This paper addresses harnessing semantic localization information in pretrained diffusion UNet models. With the proposed recursive normalized cut on the final self-attention features of the diffusion UNet encoder, this paper achieves better performance on unsupervised zero-shot segmentation comparing to previous methods utilizing other fundation vision encoders, with regulation to the granularity of details. Experiments on several benchmarks and ablation studies show the efficacy of the proposed method.", "questions": "1. In Line 223, what do you mean by \"perform a many-to-one matching\"?\n2. Are the hyperparameters the same for DiffCut on different datasets? Since you seem to use the same hyperparameters for DiffSeg?", "limitations": "This author did not discuss the limitations in the paper, which potentially could include:\n1. How is the computation efficiency of the proposed method comparing to other methods? Considering the recursive partitioning and concept assigning process.\n2. The semantic clustering knowledge in pretrained diffusion models might not be able to transfer to data domains like biomedical images.\n3. If LD, AX, or UA is given, would the proposed method still outperform the corresponding SOTAs? Or can it further improve its performance?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "This paper addresses an interesting and important problem of analyzing semantic information in pretrained generative diffusion models comparing to other fundation models. Conducting semantic clusterting on the latents of UNet encoder to reduce computation cost and then applying concept assignment while upsampling seems reasonable. The experiments are overall comprehensive and informative. The emperical results significantly outperform pervious baseline method.", "weaknesses": "My main concerns consists of several aspects:\n1. The writing seems a bit too vague, maybe with more visual illustraition of the observations? For example, can you demonstrate the \"patch-level alignment\" with visual examples? It's hard to imagine when reading this in Line 168.\n2. Are there any visual verification and description for what kind of concepts you get in Section 3.3?\n3. The experiments with the proposed DiffCut seems to mainly excute with SSD-1B. What about using other SD models? Regarding computation cost and performance benefit.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and appreciate the positive feedback.\n\n#### **Weaknesses**\n####  **1.  Writing is a bit too vague [...]. For example, can you demonstrate the \"patch-level alignment\" with visual examples?**\nFor vision tasks, we expect the vision encoder to be semantically coherent i.e to produce similar patch representations for regions belonging to the same semantic concept. We refer to it as the “patch-level alignment”. We show in Fig 4. and Fig 10.  some qualitative results that illustrate the inner patch-level alignment for different vision encoders. We visualize the patch level alignment by encoding an image and computing the cosine similarity between a reference patch and all other patches. For SSD-1B, in Fig 4. the dog is uniformly and strongly highlighted, indicating a strong patch-level alignment. Following the reviewer’s remark, we will refer to Fig.4 at l.168 for better clarity on this point.\n#### **2.  Are there any visual verification and description for what kind of concepts you get?**\nIn section 3.3, each concept is obtained via a reduction of the spatial dimension of the corresponding segment (Masked SMM l.195) . This way, a single concept has the form of a feature vector that globally describes its corresponding segment. Providing a visual description of a single concept is not straightforward and would imply a form of “decoding” of this concept. \n\n#### **3.  The experiments [...] execute with SSD-1B. What about using other SD models? [...]**\nFollowing the reviewer’s recommendation, we conduct a new evaluation to demonstrate that DiffCut provides competitive performance with other smaller diffusion backbones than SSD-1B. Specifically, we use SD1.4 and SSD-vega (another distilled version of SDXL). For SD1.4 the UNet encoder has 260M parameters (~ 30% of the UNet), for SSD-vega, the UNet encoder has 240M parameters (~32% of the UNet).\n\n|  Model   | VOC20 | Context | COCO-Object | COCO-Stuff | Cityscapes | ADE20K |\n|----------|-------|---------|-------------|------------|------------|--------|\n| SD1.4    | 57.5  | 52.8    | 30.0        | 45.2       | 24.5       | 36.7   |\n| SSD-Vega | 62.2  | 56.4    | 34.9        | 49.5       | 30.1       | 45.7   |\n| SSD-1B   | 65.2  | 56.5    | 34.1        | 49.1       | 30.6       | 44.3   |\n| DiffSeg  | 49.8  | 48.8    | 23.2        | 44.2       | 16.8       | 37.7   |\n\nThe results obtained with these two backbones are consistent with those achieved using SSD-1B. Although there is a slight performance drop with the SD1.4 encoder backbone compared to SSD-1B, the method still significantly outperforms DiffSeg. Remarkably, DiffCut with SSD-vega UNet encoder achieves performance comparable to SSD-1B, despite being only half its size. We will be glad to add these results in the final paper.\n\n#### **Questions**\n#### **1. What do you mean by \"perform a many-to-one matching\"?**\nIn datasets that include a background class, this label implicitly encompasses a variety of concepts related to \"things\" or \"stuff,\" which vary depending on the dataset. Since our method generates a segment for every detected object in an image, a one-to-one matching between a single cluster ID and the entire background's ID does not accurately represent the model's true categorization capabilities. Therefore, in such cases, we use a many-to-one matching approach by associating the clusters that primarily overlap with the background to its ID. We will be glad to include this clarification in the final paper.\n\n#### **2.  Are the hyperparameters the same for DiffCut on different datasets?**\nAs highlighted in the general response and noted in the Implementation Details section (l.230), we used a fixed set of hyperparameters, $\\tau$ and $\\alpha$, across all datasets for our evaluations. This approach ensures a fair comparison against the baselines and is particularly important for our evaluation of DiffSeg, where we also maintain a single hyperparameter value across different datasets.\n \n#### **Limitations**\n#### **1.  How is the computation efficiency of the proposed method comparing to other methods? [...]**\nThe primary bottleneck in graph-cut methods is the eigensystem solving which requires $O(n^3)$ operations with $n$ being the number of nodes in the graph. In MaskCut, the graph size remains constant, whereas in DiffCut, it has its maximum size only during the first iteration of the recursive NCut. For each subsequent partition, only the relevant nodes from the original graph are selected, reducing the eigensystem's cost. The concept assignment essentially consists in multiplying the matrix of concepts with the feature map.\n\nFollowing your remark, we provide in the general response a table comparing the runtime execution of MaskCut, DiffCut and DiffSeg. We will be glad to include these results in the final manuscript.\n\n#### **2.  The semantic clustering knowledge in pretrained diffusion models might not be able to transfer to data domains like biomedical images.**\nSince the diffusion backbone is not specifically trained to generate images in specialized domains like biomedical imaging, the method may struggle to transfer to out-of-distribution images. However, this issue could potentially be mitigated by fine-tuning the pre-trained diffusion models on the target data domain. As this limitation was not addressed in the main paper, we will discuss it in a Limitations section added to the Supplementary.\n\n#### **3.  If LD, AX, or UA is given, would the proposed method still outperform the corresponding SOTAs? [...]**\nSince our method is entirely training-free and does not rely on LD, AX, or UA, we believe that incorporating any additional information could further enhance its performance. Specifically, given the superior quality of our masks compared to MaskCut, we believe that integrating our DiffCut method to generate high-quality pseudo-masks within an unsupervised training scheme, would lead to even better performance.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GZ9RB086Of", "reviewer": "Reviewer_oADK", "summary": "This work proposes a new strategy for the task of unsupervised zero-shot segmentation using diffusion model features. The semantic maps are extracted from the U-net features of a diffusion model by applying a recursive algorithm that allows various levels of granularities of the segmentation maps. The proposed method beats competing methods on unsupervised and open-vocabular semantic segmentation on six datasets.", "questions": "ReCO is first listed training-free, in the open-vocabulary setting with extra-training, but this is not explained in the text?\n\nRegarding terminology: The section 4.4) model analysis is nicely to read. I would maybe consider replacing the term robustness with hyperparameter sensitivity (also in G). But I have no strong opinion on this.\n\nTab. 3: Did you also study the results if only 64x64 resolution were used?\n\nl.339: Could you please discuss this strategy more comprehensively in the final manuscript (or refer to a related work if existing)? It is not very clear.", "limitations": "The method seems to be profound. However, some important evaluations are missing, as explained above.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "In general, the work addresses a problem of unsupervised zero-shot segmentation that has not been addressed a lot by the community, proposing new ways to extract knowledge from the rich U-Net features of a diffusion model. While the method is not overly complex and mainly builds on previously published works, it nicely connects more conventional non-DL research from the past with the opportunities arising in foundational models, and it adds a few more technical contributions that are demonstrated to improve performance. In general, the method seems to clearly outperform previous methods on various benchmarks, including the open-vocabulary setting.\nThe paper is clearly written, and the structure allows to easily follow the work’s main components and contributions.", "weaknesses": "My main concern is that the experimental comparison seems rather unfair. While the proposed method uses powerful distilled SD-XL model the baselines only use SD1.4, which might explain most of the performance improvements. In my opinion, the ablation with respect to SD1.4 is not comprehensive enough: 1) The proposed method is not applied with SD1.4, which I think would be mandatory. 2) The DiffSeg results in Tab. 2 seem to be worse than in Tab. 1. I would appreciate a clarification of the effect of using SD-XL features. The proposed method should also be evaluated using the less powerful SD1.4 features to allow a direct comparison with the original DiffSeg method. Otherwise, the comparisons cannot be really fair, in my opinion.\n\nTab. 4: I assume, DiffSeg would again underperform DiffCut. However, having DiffSeg in this table would be a more complete comparison, not only relaying on the Hungarian algorithm-based evaluation strategy in Tab. 1.\n\nWhen considering Fig. 9, the masks do get more fine-granular for higher tau. However, the masks do not always align with the object boundaries (e.g. the surfer, the roofs, or the sign). I am wondering whether this is because of using the low-dimensional resolution and only bi-linearly upsampling?\n\nWhile the previous method DiffSeg also pursued this strategy, I am unsure if the application of the Hungarian matching algorithm to assign predicted masks to a ground truth mask is fair when comparing with methods that do not apply this strategy for the evaluation. It appears not directly comparable. I would appreciate a discussion of this fact and an argumentation why the presented evaluation strategy is reasonable and comparable to the other methods (that do not apply the Hungarian algorithm).\n\nI like the additional baseline AutoSC, which is also seems to be very effective. While the recursive and adaptive formulation of the baseline seem so to be reasonable, the gap to DiffCut is, however, not very clear. The results are almost the same. I would appreciate a more complete ablation of this. The set of chosen exponents seems to be arbitrary (l.302). Did you study whether AutoSC outperforms DiffCut if more possible alpha values were chosen? I would appreciate such an analysis and discussion of it in the rebuttal. Also, I am wondering – which model is faster, AutoSC or DiffCut?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and appreciate the positive feedback.\n\n#### **Weaknesses**\n#### **1.  [...] The proposed method is not applied with SD1.4, which I think would be mandatory. [...] to allow a direct comparison with the original DiffSeg method. [...].**\nAs mentioned in the general response, we emphasize that the ablation study carried out in Tab. 2 uses the same diffusion backbone (SSD-1B) for DiffCut and Diffseg, making the comparison between both approaches fair. Therefore, the large gain of DiffCut over DiffSeg in Tab. 2 directly validates our two main contributions: the choice of the diffusion features and the clustering algorithm. Following the reviewer’s suggestion, we present in the general response a quantitative comparison between DiffCut with SD1.4 and the original DiffSeg method. \n\n#### **2.  The DiffSeg results in Tab. 2 seem to be worse than in Tab. 1. I would appreciate a clarification of the effect of using SD-XL features.**\nThe discrepancy between the results of DiffSeg in Tab. 1 and Tab. 2 can be explained by the different backbones used as well as the absence of the refinement module in Tab. 2 as mentioned at lines 293-294. As DiffSeg relies on an attention aggregation process and iterative attention merging, its performance depends on the architecture of the backbone, especially the number and placement of attention modules which significantly differ between SSD-1B and SD1.4.\n\n#### **3.  [...] I assume, DiffSeg would again underperform DiffCut. However, having DiffSeg in this table would be a more complete comparison [...].**\nWe agree that having DiffSeg in this table would be a more complete comparison with this particular method. Nevertheless, having demonstrated in previous sections the superiority of DiffCut for generating high-quality masks, the aim here is rather to demonstrate that a straightforward extension to an open-vocabulary setting is competitive with existing approaches targeting this task.\n\n#### **4.  When considering Fig. 9, the masks [...] do not always align with the object boundaries [...]**\nThe masks not always aligning perfectly with object boundaries can indeed be attributed to using low-dimensional resolution and relying solely on bi-linear upsampling. The low-dimensional resolution does not capture the fine boundaries of objects, and bi-linear upsampling does not refine the edges to match the true boundaries. Exploring more sophisticated feature upsampling techniques could mitigate this misalignment.\n\n#### **5.  [...] I am unsure if the application of the Hungarian matching [...] is fair when comparing with methods that do not apply this strategy [...]**\nAll baselines but ACSeg, ReCO and MaskCLIP are evaluated with Hungarian matching, which is the standard evaluation protocol for methods that provide unlabeled clusters. While ACSeg also evaluates with Hungarian matching, we reported scores with labeled clusters, which are significantly higher (47.1 vs. 53.9 on VOC and 16.4 vs. 28.1 on Stuff-27). Finally, while ReCO and MaskCLIP may not seem directly comparable, we believe that assigning labels to predicted masks can offer more advantageous performance. Hungarian matching performs a one-to-one assignment between each ground-truth label and the single predicted cluster with the greatest overlap, meaning some predicted clusters may not be matched with any ground-truth label. While predicting a class may seem more challenging, it allows assigning a label to every predicted cluster, giving each a chance to be correctly classified, which can ultimately lead to better performance.\n\n#### **6.  I like the additional baseline AutoSC [...] which model is faster, AutoSC or DiffCut?**\nAutoSC is a variant of DiffCut where we replace the recursive clustering by the proposed automated spectral clustering that automatically estimates the number of clusters. Despite its simplicity, it is highly effective and demonstrates competitive performance. Differences mainly appear in datasets containing small objects (e.g., COCO-Object, Cityscapes, ADE20K). DiffCut's recursive partitioning offers flexibility in detecting small objects by allowing large segments to be further divided into smaller ones. In contrast, AutoSC is effective at uncovering the most salient clusters but may overlook smaller objects.\n\nFor the set of exponents, we chose the same set we explored for DiffCut. We also examined AutoSC's behavior with higher alpha values, but this did not lead to performance improvements as the index of the largest relative eigen-gap generally corresponds to what is found for alpha in $\\\\{1, 5, 10, 15\\\\}$. Lastly, since AutoSC does not require a recursive partitioning of the graph, it benefits from lower runtime execution.\n\n#### **Questions**\n#### **1. ReCO is first listed training-free [...]?**\nWe thank the reviewer for pointing this out. This is an oversight on our part; ReCO and MaskCLIP should be listed as training-free methods since the reported scores do not involve any training. We will update the table accordingly.\n\n#### **2. [...] I would maybe consider replacing the term robustness with hyperparameter sensitivity (also in G). [...]**\nWe appreciate the reviewer's suggestion and agree that this terminology is more appropriate for these sections. We will update the paper accordingly.\n\n#### **3. [...] Did you also study the results if only 64x64 resolution were used?**\nWe have not analyzed the results using only  $64 \\times 64$ resolution. Since the image semantics are concentrated in the layers near the UNet’s bottleneck, with the outer layers focusing more on details, we would expect the performance to be less favorable with only $64 \\times 64$ resolution.\n\n#### **4. l.339: Could you please discuss this strategy more comprehensively in the final manuscript [...].**\nWe thank the reviewer for this valuable feedback. We will discuss this strategy more comprehensively in the final manuscript with complementary details in Supplementary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "t66rsVHx8B", "reviewer": "Reviewer_2hwn", "summary": "This paper proposes a model for zero-shot semantic image segmentation based on clustering\nfeeatures from an off-the-shelf text-to-image diffusion model. Features are extracted\nfrom the U-Net used in the diffusion model. The features are then clustered using\na recursive normalized cut algorithm. The clustering is scaled up to the original\nimage resolution by semantic guidance from the extracted features. The proposed model\nconsistently outperforms previous methods on standard benchmarks for image segmentation.", "questions": "- The paper should state more clearly that it target *semantic segmention* to avoid\n  confusion with other types of image segmentation. In my view this would have a\n  major impact on the clarity of the paper.\n- How large is the computational improvement over previous methods? The differences are\n  mentioned in terms of parameter count, but a comparison in terms of runtime would be\n  interesting as well.\n- In comparison to standard semantic segmentation, the method proposed here predicts\n  unlabelled classes. Are the features consistent enough to allow for an easy (e.g.\n  linear) mapping to the labelled classes across images?", "limitations": "I don't fully agree with the statement that the method does not have potential societal\nimpacts. For example, as briefly mentioned by the authors elsewhere, resusing pre-trained\nmodels in a zero-shot fashion can save computational and labeling resources. In terms\nof energy consumption and human labour, this line of works might have a societal impact.\n\nOne limitation in my view stems from the fact that the diffusion model used as a basis\nprobably does not work well on all types of images. While unsupervised, the approach\nfollowed in the paper is dependent on the capabilities of the foundation model. This is\nnot mentioned in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The paper makes a very targeted contribution to the field. An off-the-shelf feature\n  extractor is combined with an established clustering algorithm and a novel upscaling\n  method. The method is evaluated on standard benchmarks which allows for a clear\n  comparison to prior work.\n- The model is compared to a range of state-of-the-art methods and outperforms them\n  consistently, in most cases by a substantial margin.\n- Several ablation studies are conducted to show the impact of the contributions in\n  comparison to other diffusion model based segmentation methods.", "weaknesses": "- The main contribution explitly states the improvements over the TokenCut and MaskCut\n  models, but does not compare to these models in the experiments.\n- In my view, the comparison of the respective strengths and weaknesses of different\n  base models is not detailed enough. The experiment on semantic coherence targets this\n  direction. However, beyond this analysis it would have been interesting to directly\n  compare the segmentation results after clustering that are obtained from the different\n  base models, both in terms of quantiative metrics and example cases and further\n  analyses that show the differences.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the helpful comments and appreciate the positive feedback.\n\n####  **Weaknesses**\n#### **1.  The main contribution explicitly states the improvements over the TokenCut and MaskCut models, but does not compare to these models in the experiments.**\nUnlike TokenCut or MaskCut, our method can provide dense segmentation maps and adapt the number of detected segments based on the visual content of an image. MaskCut and TokenCut cannot do this because they can only detect a fixed number of segments. (l.101-104). Therefore, these methods are not well-suited for an image segmentation task. Technically, MaskCut uses an iterative graph partitioning method and masks the graph nodes associated with detected objects. As a result, each segment is treated as a single object that cannot be refined once detected, which in practice severely limits its ability to identify a large number of objects. To support this claim and following the reviewer’s suggestion, we provide below a comparison between the performance of MaskCut and DiffCut. Overall, the improvement of DiffCut highlights our two main contributions: the quality of our visual features for semantic segmentation, and the capacity of the recursive Ncut algorithm to adjust the number of segments to the visual content of each image. We would be glad to include these results in the final paper.\n\n|  Model         | VOC20 | Context | COCO-Object | COCO-Stuff-27 | Cityscapes | ADE20K |\n|----------------|-------|---------|-------------|------------|------------|--------|\n| DiffCut (ours) | **62.0**  | **54.1**    | **32.0**        | **46.1**       | **28.4**       | **42.4**   |\n| MaskCut $(k=3)$  | 53.7  | 42.3    | 30.9        | 41.8       | 18.0       | 33.7   |\n| MaskCut $(k=5)$  | 53.8  | 43.4    | 30.1        | 41.7       | 18.7       | 35.4   |\n| MaskCut $(k=20)$ | 53.8  | 43.5    | 30.0        | 41.5       | 18.0       | 35.6   |\n\n#### **2.  In my view, the comparison of the respective strengths and weaknesses of different base models is not detailed enough. [...]**\n\nIn addition to the experiments conducted in Section 4.2 (Fig. 3 and 4), we evaluated in Supplementary D the potential of different base models for zero-shot segmentation using a simple KMeans clustering on their features for various datasets. This comparison reveals that among the different vision encoders assessed, the diffusion model shows the greatest potential for zero-shot segmentation, as the quality of clustering on its features consistently outperforms other backbones. Given this, we believe that comparing segmentation results after clustering obtained from different base models would yield similar observations.\n\n#### **Questions**\n#### **1.  The paper should state more clearly that it targets semantic segmentation to avoid confusion with other types of image segmentation. In my view this would have a major impact on the clarity of the paper.**\n    \nWe thank the reviewer for this helpful comment. We will make sure to clearly state and emphasize that this paper targets semantic segmentation to avoid confusion with other types of segmentation.\n\n#### **2.  How large is the computational improvement over previous methods? The differences are mentioned in terms of parameter count, but a comparison in terms of runtime would be interesting as well.**\n    \nThe primary bottleneck in graph-cut methods is the eigensystem solving which requires $O(n^3)$ operations. In MaskCut, the graph size remains constant, whereas in DiffCut, it has its maximum size only during the first iteration of the recursive NCut. For each subsequent partition, only the relevant nodes from the original graph are selected, reducing the eigensystem's cost. The concept assignment consists in multiplying the matrix of concepts with the feature map and retrieving the indices of the concepts with the highest similarity.\n\nFollowing your remark, we provide in the general response a table comparing the runtime execution of MaskCut, DiffCut and DiffSeg. We will be glad to include these results in the final manuscript.\n  \n#### **3.  In comparison to standard semantic segmentation, the method proposed here predicts unlabelled classes. Are the features consistent enough to allow for an easy (e.g. linear) mapping to the labelled classes across images?**\n\nWe have not explored this path yet. However, based on the results in Fig. 3 and 4, we believe that such a mapping would work well, as features from different images belonging to the same semantic class show high similarity. Consequently, we estimate that mapping these features to a class label should be consistent.\n\n####  **Limitations**\n#### **1.  I don't fully agree with the statement that the method does not have potential societal impacts. [...]**\nThank you for pointing this out. You are correct that our method which reuses pre-trained models in a zero-shot fashion can indeed save computational ressources as well as reduce energy consumption and human labor, contributing to more sustainable AI practices. We appreciate your remark and will ensure to highlight this in our paper.\n\n##### **2.  One limitation in my view stems from the fact that the diffusion model used as a basis probably does not work well on all types of images. [...]**\nSince the diffusion backbone is not specifically trained to generate images in specialized domains like biomedical imaging for example, the method may struggle to transfer to out-of-distribution images. However, this issue could potentially be mitigated by fine-tuning the pre-trained diffusion model on the target data domain. As this limitation was not addressed in the main paper, we will discuss it in a Limitations section added to the Supplementary.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their helpful comments and valuable suggestions. We would like to clarify here some key points raised by the reviewers and we then provide individual responses to each one.
  
### **Comparison with MaskCut**

Reviewers jdn9 and 2hwn mentioned that although we positioned our method as an improvement over MaskCut, we did not quantitatively compare against it. In the specific answers to R.jdn9 and R.2hwn, we explain why MaskCut was not initially designed for semantic segmentation, in contrast to state-of-the-art methods evaluated in Tab. 1. To further analyze the potential of MaskCut in semantic segmentation and meet R.jdn9 and R.2hwn’s request, we evaluate this method on the semantic segmentation task. Specifically, as MaskCut requires a predefined number of iterations $k$ to detect a fixed number of segments per image,  we evaluate MaskCut with $k$ in $\\{3, 5, 20\\}$. The quantitative comparison between DiffCut and MaskCut is shown below, and we would be glad to include it in the final paper.

|  Model         | VOC20 | Context | COCO-Object | COCO-Stuff-27 | Cityscapes | ADE20K |
|----------------|-------|---------|-------------|------------|------------|--------|
| DiffCut (ours) | **62.0**  | **54.1**    | **32.0**        | **46.1**       | **28.4**       | **42.4**   |
| MaskCut $(k=3)$  | 53.7  | 42.3    | 30.9        | 41.8       | 18.0       | 33.7   |
| MaskCut $(k=5)$  | 53.8  | 43.4    | 30.1        | 41.7       | 18.7       | 35.4   |
| MaskCut $(k=20)$ | 53.8  | 43.5    | 30.0        | 41.5       | 18.0       | 35.6   |
  
We can see that DiffCut significantly and consistently outperforms MaskCut across all datasets for any chosen $k$. These additional experiments further highlight the contributions of our DiffCut method, i.e., the diffusion features used for segmentation, and the recursive NCut method able to adapt the number of segments to the visual content of each image. In addition, we observe no significant improvements for $k=5$ or $k=20$ in comparison with $k=3$, supporting the claim that MaskCut is inadequate to detect a large number of objects due to its iterative process.


### **Clarifications in Experiments**
#### **1. Comparison to DiffSeg.**
Reviewers qiwu and oADK raised concerns about the fairness in the comparison of our DiffCut method with respect to the DiffSeg baseline. First, R.qiwu asked if a common hyperparameter setting has been used for all datasets with DiffCut, as it is the case for DiffSeg. We highlight that we used a fixed set of hyperparameters for DiffCut across all datasets for our evaluations, ensuring a fair comparison with DiffSeg, as mentioned in lines 232-233.

Additionally, R.oADK suggested that the comparison might be unfair due to the different backbones used by our method and DiffSeg. We emphasize that the ablation study carried out in Tab. 2 uses the same diffusion backbone (SSD-1B) for DiffCut and Diffseg, making the comparison between both approaches fair. Therefore, the large gain of DiffCut over DiffSeg in Table 2 directly validates our two main contributions: the choice of the diffusion features (vs self-attention maps in DiffSeg), and the clustering algorithm (recursive NCut vs iterative attention merging).

To explicitly meet R.oADK’s request, we conducted an additional comparison between DiffCut and DiffSeg, using the SD1.4 backbone as employed in the original DiffSeg method. The results shown below confirm that even with SD1.4, DiffCut significantly and consistently outperforms DiffSeg across all datasets but ADE20K where the difference of only 1pt. We will be glad to add these new results to the final paper to emphasize on the fairness of the conducted evaluation.

|  Model           | VOC20 | Context | COCO-Object | COCO-Stuff-27| Cityscapes | ADE20K |
|------------------|-------|---------|-------------|------------|------------|--------|
| DiffSeg w/ SD1.4 | 49.8  | 48.8    | 23.2        | 44.2       | 16.8       | **37.7**   |
| DiffCut w/ SD1.4 | **57.5**  | **52.8**    | **30.0**        | **45.2**       | **24.5**       | 36.7   |

#### **2. Evaluation Strategy.**
On R.oADK’s concern about the fairness of our evaluation strategy, we highlight that all methods in Tab. 1 are evaluated using Hungarian matching but ACSeg, ReCO and MaskCLIP which attribute labels to predicted masks. For these latter methods, we clarify that the Hungarian matching may leave some predicted clusters unassigned to any ground truth labels whereas assigning text labels offers a chance to each cluster to be correctly classified. For example, STEGO and ACSeg, which also evaluate with label predictions, show significantly higher performance in this setup than with Hungarian matching evaluation. Our evaluation setup is thus reasonable, making the comparison with ReCO and MaskCLIP in Tab. 1 fair.

#### **3. Runtime Comparison**
Reviewers  qiwu and 2hwn have asked for a runtime comparison between methods. Following their requests, we provide in the following table a runtime comparison between DiffCut, MaskCut and DiffSeg which are respectively the two main baselines when it comes to graph-based image clustering and diffusion-based zero-shot segmentation.
|                        | MaskCut $(k=5)$ | DiffCut | DiffSeg - SD1.4 | DiffSeg - SSD-1B |
|------------------------|---------------|---------|-----------------|-------------------|
| Images / sec | 0.84          | 1.11    | 2.75            | 1.25              |
|                        |               |         |                 |                   |

MaskCut with $k=5$ is the slowest method, segmenting 0.84 images per second. Using SSD-1B with images at $1024 \times 1024$ resolution, DiffCut’s speed is slightly lower than DiffSeg’s based on the same architecture. With the SD1.4 backbone, DiffSeg demonstrates superior runtime performance due to the smaller size of the architecture and the input image size. We will be glad to include these results in the final manuscript.

Meta-review: The paper is reviewed by 4 reviewers. All reviewers recommend acceptance. The authors should add to the additional comparisons, clarifications and discussions about limitations to the final copy or supplementary materials.
Decision: Accept (poster)