gen2seg: Generative Models Enable Generalizable Instance Segmentation
Authors: Om Khangaonkar
Hamed Pirsiavash
Venue: ICLR 2026 Poster (2026)
Keywords: generative model
instance segmentation
generalization
stable diffusion
mae
representation learning
zero-shot
Primary area: generative models

Abstract: By pretraining to synthesize coherent images from perturbed inputs, generative models inherently learn to understand object boundaries and scene compositions. How can we repurpose these generative representations for general-purpose perceptual organization?  We finetune Stable Diffusion and MAE (encoder+decoder) for category-agnostic instance segmentation using our instance coloring loss exclusively on a narrow set of object types (indoor furnishings and cars). Surprisingly, our models exhibit strong zero-shot generalization, accurately segmenting objects of types and styles unseen in finetuning. This holds even for MAE, which is pretrained on unlabeled ImageNet-1K only. When evaluated on unseen object types and styles, our best-performing models closely approach the heavily supervised SAM, and outperform it when segmenting fine structures and ambiguous boundaries. In contrast, existing promptable segmentation architectures or discriminatively pretrained models fail to generalize. This suggests that generative models learn an inherent grouping mechanism that transfers across categories and domains, even without internet-scale pretraining. Please see our website for additional qualitative figures, code, and a demo: https://reachomk.github.io/gen2seg/

Published as a conference paper at ICLR 2026
GEN2SEG: GENERATIVE MODELS ENABLE GENERALIZABLE INSTANCE SEGMENTATION
Om Khangaonkar, Hamed Pirsiavash University of California, Davis reachonk. github. geesgg
ABSTRACT
By pretraining to synthesize coherent images from perturbed inputs, generative models inherently learn understand object boundaries and scene compositions. How repurpose these generative representations for general-purpost per- ceptual organization? We finetune Stable Diffusion and MAE (encoder+decoder) for category-agnostic instance segmentation using instance coloring loss clusively on narrow set of object types (indoor furnishings and cars). Surpris- ingly, our models exhibit strong zero-shot generalization, accurately segmenting objects of types and styles unseen in finetuning. This holds even for MAE, which is pretrained unlabeled ImageNet-1K K only. When evaluated unseen object types and styles, our best- pess--eeforming models closely approach the heavily super- vised SAM. and outperform it when segmenting fine structures and ambiguous boundaries. In contrast, existing promptable segmentation architectures or dis- criminatively pretrained models fail to generalize. This suggests that generative models learn inherent grouping mechanism that transfers across categories and domains, even without internet-scale pretraining. Please see our website for addi- tional qualitative figures, code, and demo.
Figure 1: The model that generated the segmentation maps above has never masks of humans, animals, or anything remotely similar. We fine-tune generative models for instance segmentation using synthetic dataset that contains only labeled masks of indoor furnishings and cars. Despite never seeing masks for many object types and image styles present in the visual world, our models are able to generalize effectively. They also learn accurately segment fine details, occluded objects, and ambiguous boundaries.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
Humans learn to carve the visual world into discrete, persistent objects from limited experience. A toddler who has mostly handled cups, chairs, and toys at home can still recognize the zebra, giraffe, and lion their first trip to the zoo distinct objects, even without knowing what they are called. This ability extends to abstract depictions: people can understand line drawings and artworks even without prior exposure to them (Hochberg & Brooks 1962). These observations suggest human vision acquires general, transferable mechanisms for grouping pixels into objects.
Modern vision systems often show "zero-shot" transfer new datasets without domain-specific fine-tuning, but typically rely on broad labeled datasets that attempt to cover many object categories and styles. We ask different question through the lens of instance segmentation: can model learn from only very narrow slice of the visual world and still generalize to unseen object types and image styles? We focus on stricter "zero-shot" setting, where we explore how the model performs on object types it has never seen mask for.
We hypothesize that generative models are particularly well posed to succeed at this task. Because they learn to synthesize scenes from minimal cues (e.g., text prompt or corrupted image), they must learn to implicitly represent the parts that make up the image. This allows generative models to produce scene compositions unlike anything seen in pretraining data, such as "a Van-Gogh style painting of a panda driving a car."
We introduce simple finetuning method that taps into these generative priors. We first experiment with a Masked Autoencoder (MAE; encodertdecoder) pretrained on ImageNet- IK only end-to-end using only mask supervision from two narrow synthetic domains (indoor furnishings and cars) to generate object instance groupings. Despite this limited supervision, our model is able to generalize new object types unseen in finetuning, such people and animals, novel image styles, such as impressionist art and x-rays. When we experiment with limiting the diversity or complexity of our finetuning dataset, this generalization persists, suggesting it is due the generative prior.
To explore the effects of internet-scale pretraining, we apply our finetuning method to Stable Diffu- sion 2. Without seeing any labeled masks from the evaluation categories, our finetuned Stable Dif- fusion achieves performance comparable to SAM across five datasets of distinct domains. Beyond generalization, our models consistently produce crisper boundaries than SAM (e.g., on BSDS500), behavior that persists even when finetuned on datasets with polygonal edges (e.g.. COCO). They also excel at segmenting fine structures (e.g., on iShape) and exhibit object-part compositionality, despite never receiving part-level supervision. We hypothesize these behaviors arise directly from generative pretraining, which must model fine edges, delicate structures, and part-whole relation- ships to synthesize detailed scenes.
Together, our findings argue that generative pretraining encodes an inherent grouping mechanism that extends beyond both the object types and the image styles seen during finetuning. We hope utilizing the generative representations learned from image synthesis can pave the way for more generalizable and human-like perception, enabling advances in fields where detailed scene under- standing is critical, such as robotics, medical imaging, and autonomous systems.
2 RELATED WORK
2.1 GENERATIVE MODELS FOR PERCEPTION
Generative models, originally developed for image synthesis, have increasingly been adapted for perception tasks in computer vision. longstanding viewpoint in the field (dating back to Hinton's early work) (Hinton 2007) posits that learning to generate data can aid in recognizing it. Early work on GANs (Goodfellow et al. 2014) evaluated whether representations learned by generating images (Radford et al. 2015) videos (Vondrick et al. 2016) transferred well to image classification or action recognition, respectively, but performance was always far below discriminatively pretrained models. Some works utilized inpainting (Pathak al. 2016) and colorization (Larsson al. 2017; Zhang et al., 2016) 2017) as pretext tasks for representation learning. but these were subsequently
surpassed by discriminative pretraining (Noroozi et al. 2017 Gidaris al. 2018 He et al. 2020).
2

---

Published as a conference paper at ICLR 2026
Ours
Input
DINO-B
MAE-B
MAE-H
SD (10 Classes)
SD
SAM
DRAM COCO
EgoHOS
iShape
PIDRay
Figure 2: To showcase the potential of generative models for instance segmentation, we highlight an example from each evaluation dataset where most all of our models outperform SAM, despite never having seen masks of these object types. SAM often fails on fine structures (wires) or ambigu- ous boundaries (horses & carriage), leaving black regions where no object was detected. DINO-B also performs poorly, suggesting that generative pretraining (e.g.. MAE, Stable Diffusion) learns strong priors perceptual grouping.
Recent advancements have demonstrated the efficacy of diffusion models (Sohl-Dickstein et al., 2015 Ho et al., 2020) in various visual tasks. key advantage of recent diffusion models is the sheer scale of their pretraining: learning from over 2 billion images (Schuhmann et al. 2022 has the potential to outscale existing discriminatively trained models. Their large-scale generative pre- training has since been transferred to many perceptual tasks, such as 3D reconstruction (Liu et al. 2023b Poole et al., 2022 Wang et al. 2023a), semantic (Baranchuk er al., 202T Li et al. 2023b Kawano & Aoki 2024 Tian et al. 2024) and amodal segmentation (Ozguroglu al. 2024 Chen ct al. (2024), monocular depth (K et al. 2024 Zhao et al. (203), surface normals (Fu et al. 2024), optical flow (Ravishankar al. 2024), correspondence (Tagg al. 2023). and classification (Li jet al. 2023a). Other works have shown that depth, normals, albedo, and segmentation can emerge (albeit with low quality) from generative models (Bhattad al. 2023 Namekata al. 20244] Kar- mann & Urfalioglu 2024) without finetuning, suggesting similar representations may emerge from the data alone (Dravid al. 2023 Huh et al. (2024). While there some prior works that finetune diffusion models for instance segmentation (Fan et al. 2024 Zhao et al. 2025), these works focus on building competitive instance segmenters using large scale data, while we explore generalization through the lens of instance segmentation.
parallel line of work explores representation learning through Masked Autoencoders (MAE) (He et al., (022), which achieve state-of-the--rt performance across many visual tasks by pretraining to reconstruct masked image tokens before fine-tuning on discriminative objectives. A common practice, however, is to discard the decoder despite it containing rich pixel-level generative features prior to fine-tuning. Recent work (Bar et al.) 2022) has demonstrated the encoder and decoder's joint capacity generalize through visual prompting. generating masks without explicit supervi- sion. In contrast, our approach end-to-end fine- tinettunes the encoder and decoder and shows that it can generalize objects whose masks not observed during fine-tuning.

---

Published as a conference paper at ICLR 2026
Image
MAE-H
SD
Figure 3: Our models assign similar colors compositionally related parts of scene. Vader's mask and body (top), or the bowties and shirts (bottom) are separated by subtly different hues, while distinet colors partition unrelated parts such as his leg and the poles (top). the dogs and text (bottom). This emerges without any part-level supervision, suggesting generative models learn hierarchical scene representations. See Figures 14 20 and Table 7 for more examples and results.
2.2 INSTANCE SEGMENTATION
Recent advancements have introduced category-agnostic instance segmentation, enabling models to segment objects without prior knowledge of their classes. The most well known of these is SAM (Kirillov et al.) 2023), which learns zero-shot promptable segmentation by finetuning an MAE backbone and mask decoder on the massive SA-1B dataset. SAM and its recent successor SAM2 (Ravi al. 2024) represent breakthrough in obtaining general, sategory-agnostee masks without per-dataset training. However, unlike SAM, which was trained on a massive labeled dataset, our approach aims leverage generative knowledge to achieve broad instance segmentation. Further- more, our strongest model is trained for only 29 hours on four RTX6000 Ada (48GB) GPUs on less than 87,000 images and 3.7 million masks of only select categories, while SAM was trained for 68 hours on 256 A100 (80GB) GPUs on 11 million images and 1.1 billion masks of many aspects of the visual world.
Recent works (Wang et al. 2024 2023b 2022) train instance segmenters from pseudo- pseudo-masks auto-
matically derived from contrastive self supervisd features via normalized cuts (Shi & Malik | (000). or similarity and thresholding. While both our work and theirs use less labelled data compared to prior works (Kirillov al., 2023), our goals differ. They leverage vast, diverse unlabeled images to synthesize pseudo-labels for downstream finetuning, to reduce reliance on manually annotated la- bels. In contrast, we use manually annotated labels, but deliberately restrict annotation diversity, to build models that generalize to many novel object types when finetuned on very few. Additionally, the noisy pseudo-labels require iterative retraining, which takes 2-15x longer than our single-pass zero-shot generalization and our
fine-tuning. Many of the findings from our work, such robustness to noisy labels, provide a path to eventually combine these two directions future work.
METHOD
We aim to adapt pretrained generative models, such as Stable Diffusion or MAE, to perform category agnostic instance segmentation. Instance segmentation, in particular, is pixel level task, at which generative models are naturally primed to excel. Specifically, we hypothesize they inherently learn to understand object boundaries and groupings because they must synthesize the objects' and boundaries themselves. In contrast, most SOTA models, such as SAM (Kirillov et al.) 2023), extract features using encoder that discards low-level details. As result, they must learn mask
structure
4

---

Published as a conference paper at ICLR 2026
predictors feature pyramid networks from scratch to gradually upsample these features to higher resolutions (Lin et al. 2017 Li et al. 2022).
3.1 INSTANCE SEGMENTATION AS IMAGE TO IMAGE TRANSLATION
Recent instance segmentation models predict sets of binary masks, each representing object in- stance (Carion et al., 2020 Cheng et al. (022). However, it is not obvious how to enable generative R W to easily decode to this style. Drawing in- spiration from work on image-to-image translation (Isola et al. 2017), we encode our ground truth as an RGB image with a unique color for each instance and black color for the background. We find that both the Stable Diffusion VAE and MAE models can decode these masks with effectively no loss in quality. Thus, we finetune our model by forwarding our RGB image (or its latent) without adding noise or masking, decoding the output, and optimizing in pixel space with respect to the RGB ground truth.
models, designed to map from R X
Unfortunately, one cannot simply assign each ground truth mask to a color and train the model to regress since there are many possible accurate color assignments. Thus, we propose our instance coloring loss based two key properties of an RGB segmentation mask. First, the color of all pixels within a mask should have low variance. Second, the color a mask should not be predicted anywhere outside the mask. By emphasizing these two properties, we can learn a model of instance segmentation without needing to enforce specific colors for object masks. Simply, our finetuned model should ensure each instance is assigned unique color that is consistent across its pixels. Our finetuning strategy draws on the same intuition as priors works that aim to cluster features for supervised instance segmentation (De Brabandere al., 2017 Kong & Fowlkes) 2018). We designed it to enable a training strategy that is simple, intuitive, and agnostic to model archifecture with fast inference.
More formally, assuming an image with n instances, let Ω be the set of all pixels in the image, and i (O,..., n2} the instance index where 0 always the background. We define the set of pixels for instance i as:
S, i e Ω pixel j belongs to instance i).
For each instance, we define the mean embedding (or representative color) as
1
Mi.c
Pj.c and Ma.c 0, {1 {1... n), Ve E {0, 1,2
[S
JES;
where Pj.c is channel e of the predicted color pixel j. We force the background mask's mean to be black (Mo.e 0) to follow standard convention and distinguish it from objects.
Our loss is composed of three components:
1. Intra-Instance Variance Loss: To ensure that the predictions within an instance are consistent, we use smooth & loss that encourages each pixel's prediction be close to the instance mean. This is defined as
Cvar
Le(p Hac.)
1=0
jES. c=0
where L, denotes the smooth & loss. We find that using smooth & loss over the standard & loss converges better does not sharply penalize outliers.
2. Instanen Separation Loss: We define the following loss to encourage the color of pixels outside the instance be pushed away from the instance's 's mean color, ensuring that distinct regions do not converge to similar colors:
Seep
Σ JET. 1 2 ( mi,e)²
(4)
V
S
T=
where T, = 2 \ S; denote the set of pixels outside instance i. The loss is designed to saturate as the distance increases so that pixels far away from μ4 in color value do not dominate the loss. We include V/S. in the denominator to emphasize smaller objects.
5

---

Published as a conference paper at ICLR 2026
Input
MAE-B Features & Mask
SimpleClick
SD Features & Mask
SAM
Figure 4: For qualitative comparison, we showease several results for promptable segmentation using our features. Our finetuned MAE-B and SimpleClick are trained on the same data, using the same backbone, yet our MAE-B strongly outperforms SimpleClick due to generative prior. Our finetuned Stable Diffusion has never seen mask of the object type it is segmenting, but performs similar to SAM, which has been heavily supervised on over a billion masks of all types. Prompt points are shown in green on the input.
Mean-Level Separation Loss: further separate instances, we design the following loss similar to the above one, but for mask centroids:
Lmean
(5)
nn + 1) ogisisn 1 Eo 2j,c)²
where the first fraction simply normalizes by the total number of comparisons.
Finally, we finetune a pretrained diffusion model by minimizing our instance coloring loss Lic:
Lı Lvar + >sep + Ameaau
where Asep and Amean are hyperparameters controlling the importance of each loss term.
3.2 POINT- POUNT-PROMPTABL INSTANCE SEGMENTATION
So far, our finetuned model assigns a color F(r, u) = Pj to each pixel where (x, y) is the location of pixel j. Since pixels belonging to the same object instance encouraged to have similar colors, we develop a simple method to "point-prompt" the feature map F for binary masks (similar to SAM (Kirillov al. 2023)). We intentionally opt not to train separate mask decoder to showcase that our model's 's output features truly represent object instance shapes. Additionally, existing mask decoders specialized architectures designed upsample from low resolutions to the original image size, while our features are at the same resolution as the original image. Our simple prompting method can be viewed as analogous similar evaluation methods in representation learning used to show that features truly represent the desired task, such as nearest-neighbor classification, semantic segmentation, or tracking (Caron al. 2021). One can potentially improve our results by training promptable high-resolution mask decoder on top of our features, which we leave for future work.
For each prompt point p IR? which contains the and location of the prompt pixel, we calculate the query vector (pp E IR33 Gaussian weighted average of the predicted color at the neighborhood
of p by 4p = Ex, w(x.y) P(z.y) where w(I, y) is Gaussian function with mean p and standard Σx.y ((x,))
deviation (W, H) and W and H are the width and height of F.
We then compute query-feature similarity map:
Sp(z, v) = min(1, F y) 9pl12
(7)

---

Published as a conference paper at ICLR 2026
Method
CoCO
COCO COCO
DRAM
EgoHOS
iShape
PIDRay
SAM
57.0
59.5
56.9
50.2
56.4
16.8
44.2
SimpleClick
1.4
0.6
0.2
2.4
1.6
1.6
1.5
DINO-B
35.0
11.0
1.7
29.4
14.8
27.4
14.9
MAE-B
44.6
17.8
2.9
34.3
28.9
31.1
21.6
MAE-H
50.0
23.2
3.5
40.3
31.9
34.9
24.1
SD
57.6
38.8
8.5
48.2
40.0
51.4
30.9
Table We evaluate zero-shot mloU a single prompt point on wide spread of datasets. We match or recover high percentage of performance (minimum 70%) all datasets except COCO This suggests that, for larger objects, our models have learned strong object-level representations that transfer across categories and styles. Our baselines, SimpleClick and DINO-B are far below MAE-B, suggesting this generalization is unique to generative models. Additionally, we strongly outperform SAM on the iShape dataset, which evaluates segmentation of detailed and complex struc- tures. SAM is trained on SA-1B. All other models are trained on our limited Hypersim+VK2.
normalize it between [0, 1], and smooth it with a joint bilateral filter (Petschnigg al. 2004 (using F as guidance), thus averaging the similarities close in both pixel location (z, y) and feature value F(x,y). Finally, assuming set of & point prompts, take the per-pixel maximum across similarity maps, and threshold the merged similarity map to produce the binary mask.
4 EXPERIMENTS
4.1 DATASETS
Training: Inspired by previous work (Ke et al. 2024). we train model exclusively on synthetic data. We combine two datasets: Hypersim (Roberts et al. 2021) and Virtual Kitti 2 (Cabon et al., 2020). Hypersim provides a rich variety
indoor scenes (i.e. bathrooms, bedrooms, libraries, and kitchens), while Virtual Kitti 2 focuses on outdoor driving scenes, with annotations limited to cars. Both datasets are realistic and do not contain other styles. After removing images with too few masks, our dataset comprises of 86,000 images (66,000 from Hypersim and 20,000 from Virtual Kitti 2), with a sampling strategy that selects Hypersim images 90% of the time. Neither dataset includes annotations of people, animals, and several other categories. Additionally, while the number of images comparable to existing instance segmentation datasets, the diversity substantially lower. Our subset of Hypersim contains multiple views sampled from just 457 scenes, while Virtual Kitti contains just 5 videos (each 115 seconds long), repeated from different viewing angles and weather conditions. A list of labeled object types in Hypersim is available in Appendix D
Evaluation: Our finetuned models have seen masks some limited object categories in a single synthetic and realistic style. We aim to evaluate models' zero-shot generalization to unseen categories and styles. We select 5 datasets from (Kirillov et al., 2023), each of which contains very different domain from the others: COCO (The COCO 2017 validation (Lin al. 2014), but we exclude object types seen in finetuning, The list of categories are presented in Appendix D). DRAM (Cohen al., (022) (art), EgoHOS (Zhang et al.) 2022b) (egocentric). iShape (Yang et al. 2021) (complex and fine structures), and PIDRay (Zhang et al. 2022a) (luggage x-rays). We describe the details and motivation for each dataset Appendix A.2
4.2 MODELS AND BASELINES
Understanding the conditions under which generalization is possible is central to understanding our findings. In practice, generalization depends two types of factors: the choice of model architec- ture and the properties of the training data. First, we implement our method on several generative models and compare them with some baselines. Unless specified, the models finetuned using the loss and datasets described above. We finetune Stable Diffusion variants at 480 640 (Hypersim) and 368 x 1024 (Virtual Kitti 2) and Inaeranet-retrrained models at 224x 224.
7

---

Published as a conference paper at ICLR 2026
Data
coco
COCO
COCO
DRAM
EgoHOS
iShape
PIDRay
exc
Original
50.0/57.6
23.2/38.8
3.5/8.5
40.3/48.2
31.9/40.0
34.9/51.4
24.1/30.9
COCO 53.6/64.0
18,8/44.0
2.9/9.8
48.1/51.2
26.8/35.2
33.4/41.2
25.7/31.9
ClevrTex
40.0/47.1
19.4/21.6
1.9/7.5
23.5/28.0
21.4/21.9
27.6/32.1
22.2/23.7
10 classes
54.8/56.1
21.7/35.2
2.7/5.0
40.1/45.1
29.4/38.5
33.0/53.6
17.6/22.8
5 classes 42.1/47.6
16.2/29.7
1.4/6.4
34.2/38.2
23.7/34.4
28.5/48.5
15.2/19.4
Table 2: We explore how the diversity of the finetuning domain impacts generalization. Interestingly, model still performs well with real-world data (COCO) synthetic thape-segmentttion datasets (ClevrTex). Furthermore, performance with just 10 classes from Hypersim results in nearly identical performance the full dataset (33+ classes), suggesting generalization emerges without diverse finetuming data. However, some degree of diversity is still needed for optimal performance, as seen by the performance drops ClevrTex or with only 5 classes. We report results with both the MAE- H and SD backbones as MAE-H/SD.
We apply our method to four different settings that include diffusion and MAE models: Stable Diffusion (SD) (Rombach et al. 2021): Following (Garcia al. 2024), we set N in an N- step latent diffusion model and finetune end-to-end without noise. Using the frozen VAE to decode the U-Net output, we optimize pixel-space loss, yielding one-step, deterministic image segmenter. MAE with Decoder (MAE-B/H) (He et al., 2022): We finetune an MAE with its decoder end-to- end to showcase that a strong generative prior learned solely from ImageNet-IK images without internet-scale pretraining text supervision can effectively generalize. We use MAE ViT-B for direct comparison to DINO-B/SimpleClick (see below) and MAE VÝT-H to demonstrate scalability.
We compare with the following baselines: SimpleClick (Liu et al. 2023a): A SOTA promptable segmenter using an MAE-B ViTDet (Li et al. 2022) backbone. We finetune SimpleClick using its released training code on our synthetic dataset to show that existing architectures cannot generalize well beyond supervised categories. DINO + VAE (DINO-B) (Caron et al., [02T]: To test whether generalizable object groupings are exclusive to generative pretraining, we attach DINO frozen VAE decoder (from Stable Diffusion) via simple up-conv and fine-tune end-to-end. DINO pro- vides the discriminative features, while the VAE can decode to object shapes unseen in finetuning. Segment Anything (SAM) (Kirillov et al.) 2023): We use SAM ViT-H off-the-shee as a high-water mark for generalization that is supervised on the huge SA-IB dataset with annotated masks.
We examine the role of training data by varying both the data domain and number of object cat- egories. We train on (i) the full MS-COCO train split or (ii) ClevrTex (Karazija et al.) (02T), a synthetic, shape-centric dataset. COCO has real-world visual diversity but includes noisier labels while ClevrTex contains pixel-perfect annotations but a narrow set of simple object types. As our Hypersim and Virtual Kitti 2 mix contains both complex shapes and high quality labels, comparing these regimes helps disentangle the impact of object complexity from annotation quality. To study category diversity, masks in finetuning. To ensure the model does not see the masks of unknown categories, we disable the loss for pixels within the bounding box of all unknown objects. Additional details regarding models and datasets appear in Appendix A.7
also experiment with restricting Hypersim only 10 and 5 classes of labeled
4.3 ZERO-SHOP PROMPTABLE INSTANCE, SEGMENTATION
We evaluate our model on the task of zero-shot point- promptabee instance segmentation. Following Kirillov al. 2023), ) we first evaluate our models' ability to produce reasonable masks using only a single prompt point at the ground truth object center and comparing the predicted mask's IoU with the respective ground truth. Then, following the so-called "golden" standard of prompting (Ravi
et al. 2024 Kirillov et al., 2023 Liu et al.) 22033a] Sofiiuk et al. 2022 Lin et al. 2022 Sofiikk et al.] 2020). we iteratively find the largest contiguous area of the ground truth with no mask predicted yet, and seleet the next prompt point in that area closest the area's 's center. As promptable segmentation is a highly ambiguous task, we do not necessarily expect to get high IoU, but we do hope for it to approach our high-water mark, SAM.
8

---

Published as a conference paper at ICLR 2026
COCO train image
Annotated Masks
SD (COCO)
Model
Edge AP
Model
Edge AP
DINO-B
33.2
MAE-H (ClevrTex)
86.4
MAE-B
53.7
MAE-H (10 classes)
86.9
Sobel
65.0
SD (ClevrTex)
88.5
MAE-H (COCO)
75.9
SD (COCO)
89.7
SAM
79.0
SD (5 classes)
91.7
MAE-H
80.5
SD (10 classes)
92.5
MAE-H (5 classes)
83.1
SD
93.4
Figure 5: We evaluate the zero-shot edge AP for recall less than 20% of the edges synthesized from models and baseline segmentations BSDS500. Nearly all generative models outperform SAM, even when trained on polygonal mask edges from COCO, Observe how in the figure above, though the annotation has very noisy edges, SD (COCO) does not learn this and instead pre- dicts smoother and more perceptually aligned boundaries. This suggests that our models' ability to produce detailed groupings is due to the generative prior, and not any specific dataset.
Results. First, we examine model performance using a single prompt at the object center (Table D. On all evaluation datasets except COCOM/S marginally exceeds SAM, despite never seeing labeled masks for these categories. This indicates that generative models learn transferable object-level features, particularly for larger objects and intricate details (evidenced by superior results on the iShape dataset). Interestingly. SAM struggles in some cases because it seems to group by texture for out-of-distribution cases, such as art.
exe (Medium and Small), our model approaches or
Our method also surpasses both baselines, SimpleClick and DINO-B. SimpleClick, as expected, struggles with zero-shot mask generation. SimpleClick's ' failure to generalize represents weakness in existing segmentation architectures: nearly all models use mask predictors finetuned from scratch Kirillov et al. 2023 Wang al. 2024). When faced with object types unlike anything seen in finetuning, these models will fail to generalize as the mask predictor has only seen the object types in finetuning and lacks any other visual priors. Our method deliberately uses only generatively pre- trained parameters, so that the entire model retains broad visual priors for test-time generalization.
Our finetuned DINO-B model successfully activates objects, but struggles to separate their in- stances. We hypothesize this is because self-distillation (and discriminative pretraining in general) over-emphasize semantics via invariant representations, meaning they enforce that the output rep- resentation does not change across augmentations. In contrast, to succeed at instance segmentation, one must learn equivariant representations, meaning they account for changes in the scale, shape, or structure of the image (and the objects within) (He 2017). We hypothesize that generative models are well posed to learn equivariant representations because they must learn synthesize plausible image for every corrupted input they receive.
However, our models have limitations segmenting small objects, likely due to biases from pre- training: Stable Diffusion's 's text conditioning emphasizes large, prominent objects, while MAE's ImageNet-1K 1K pretraining prefers central "main" objects. Additionally, models like SAM and other state-of-the--rt segmenters (He et al., 2017: Cheng et al. 2022 Cai & Vasconcelos 2019) fine- tune at 1024x 1024 resolution, whereas we fine-tune at lower resolutions: x80x 640 (Hypersim) and 368 x 1024 (Virtual Kitti 2) for Stable Diffusion variants, and x 224 for ImageNet-basedd models. We expect that stronger generative models, such as FLUX.1, would enhance small-object segmentation. We leave this future work due to their large parameter counts.

---

Published as a conference paper at ICLR 2026
We also find that our models generalize well, even without synthetic complex datasets, as shown in Table Surprisingly. our models learn to generalize even with only 5 object types (books, chairs, lamps, tables, and pillows) seen in finetuning, cubes or spheres (ClevrTexx. Furthermore, our models still excel at segmenting fine structures (as shown iShape) when finetuned COCO, which mostly contains polygonal mask annotations. Additionally, we see only minor improvement when generative models are finetuned with COCO, suggesting that our models' zero-shot generalization is very close to the "upper bound" of seeing the objects in finetuning. This suggests generative pretraining is a powerful paradigm to learn gen- eralizable instance grouping priors.
when only finetuned
simple shapes such as
4.4 ZERO-SHOT EDGE DETECTION
We evaluate our segmentation features on the task of edge detection with the BSDS500 dataset (Martin al., 2001). It is important to note that we are not simply trying to find all edges in the image, but only object boundaries. Following (Kirillov et al., 2023), we simply apply a Sobel filter on the predicted features followed by nonmax suppression to thin the edges. We use the edge detection described above on the original image as a "weak" baseline and on the output of SAM's AutoMaskGenerator as a "strong" baseline. We report AP for recall less than 20%. We explain this choice (and display the full precision-recall curves) in Appendix [
Results. Our model accurately delineates the edges of primary objects in each image despite never seeing mask of the object type. As shown in Table our SID model produces much finer edges compared to SAM. Additionally, our MAE-H model marginally outperforms SAM despite being supervised at less than 5% of the resolution (224 x 224 vs 1024 x 1024). We also observe that when we change the training data from synthetic Hypersim+VK2 to human-labeled COCO, we observe decrease of less than 5 points in edge AP on both SD and MAE-H models. Because COCO's edges are coarse and polygonal, the relatively small performance drop supports our hypothesis that our model's accurate and clean edges stem from generative pre-training and are not due to the bias from the synthetic data. More interestingly, our SD (COCO) model is still better than SAM by more than 10 points for recall less than 20%. Both COCO and SA-1B have similar polygonal edges, which suggests by learning to synthesize scenes, generative models inherently learn a detailed representation of object boundaries. Our precise edges also persist (and are sometimes even stronger than the base model) when dataset complexity reduces, suggesting the generative model "defaults" to predicting clean edges when segmenting an object.
5 CONCLUSION
Our findings suggest that learning to generate visual reality inherently teaches a detailed understand- ing of its constituent parts. Our models are able to segment objects and styles nothing like anything any labels seen in finetuning, yet still achieve competitive performance with heavily supervised models. As we continue to scale generative models and diversify their pretraining data, their ability to perceive the visual world will only grow. Learning to leverage these powerful representations for a wide variety of visual tasks has the potential to enable a new frontier in generalizable perception.
REFERENCES
Amir Bar, Yossi Gandelsman, Trevor Darrell, Amir Globerson, and Alexei A. Efros, Visual prompt- ing via image inpainting. arXiv preprint arXiv:2209.0066 2022.
Dmitry Baranchuk, Ivan Rubachev, Andrey Voynov, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. arXiv preprint 12..03 2021.
Anand Bhattad, Daniel McKee, Derek Hoiem, and D. A. Forsyth. Stylegan knows normal, depth, albedo, and more, 2023.
org/ absaabb/s355 00987
Yohann Cabon, Naila Murray, and Martin Humenberger. arXiv:2001. 10773, 2020.
Virtual kitti 2. arXiv preprint
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance 906.
segmentation, 2019. URL
10

---

Published as a conference paper at ICLR 2026
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pp. 213-229, Springer, 2020.
Mathilde Caron, Hugo Touvron. Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 9650-9660, 2021.
Kaihua Chen, Deva Ramanan, and Tarasha Khurana. Using diffusion priors for video amodal seg- mentation. arXiv preprint arXiv:2412.0462. 2024.
Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. De- tect what you can: Detecting and representing objects using holistic models and body parts. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1971-1978, 2014.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299 2022.
Nadav Cohen, Yael Newman, and Ariel Shamir: Semantic Segmentation in Art Paintings. Computer Graphics Forum, 2022. ISSN 1467-8659. doi: 10.111 14473.
Bert De Brabandere, Davy Neven, and Luc Van Gool. Semantic instance segmentation with discriminative loss function. arXiv preprint arXiv: (0t::/770.0220 1708.0251, 2017.
Amil Dravid, Yossi Gandelsman, Alexei A. Efros, and Assaf Shocher. Rosetta neurons: Mining the common units in a model zoo. Proceedings of the IEEF/CVF International Conference
thecvf. com/ content/ on Computer Vision, 2023. URL MewuoossMiuiinnooo the Cwwmon__unnaiaaaoee Units_
BBt/2222//papers/Daaaa in. Model__odoooooo 200 ICCV 1022_paper.pp_r
BaceettaNNuuroe
Yue Fan, Yongqin Xian, Xiaohua Zhai, Alexander Kolesnikov, Muhammad Ferjad Naeem, Bernt Schiele, and Federico Tombari. Toward a diffusion-based generalist for dense vision tasks. arXiv preprint 20t1::2007.00003 2024.
Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from single image. In European Conference on Computer Vision, pp. 241-258. Springer, 2024.
Gonzalo Martin Garcia, Karim Abou Zeid, Christian Schmidt, Daan de Geus, Alexander Hermans, and Bastian Leibe. Fine-tuning condicooni diffusion models is easier than you think. arXiv preprint 1tt.:2249911335353 2024.
Spyros Gidaris, Praveer Singh, and Nikos Komodakis. Unsupervised representation learning by predicting image rotations. arXiv preprint arXiv; 1803.0:803.07728, 2018.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. In Advances in Neural Information Processing Systems, pp. 2672-2680, 2014.
Kaiming He. Mask r-cnn tutorial. ICCV 2017. Presentation slides retrieved from http:2017_tttorrallkaaaaaiiaaaam
2017.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738. 2020.

---

Published as a conference paper at ICLR 2026
Kaiming He, Xiangyu Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVE Conference on Com- puter Vision and Pattern Recognition, pp. 16000-16009, 2022.
Geoffrey E Hinton. To recognize shapes, first learn to generate images. Progress in brain research, 165:535-547, 2007.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Advances in Neural Information Processing Systems, volume 33, pp. 1850- 2020.
Julian Hochberg and Virginia Brooks. Pictorial recognition as an unlearned ability: A study of one child's performance. the american Journal of Psychology, 75(4):624 628, 1962.
Minyoung Huh, Brian Cheung, Tongzhou Wang, and Phillip Isola. The platonic representation hy- pothesis. arXiv preprint arXiv:2405.07987, 2024. URL htt.atts//aavvv..orr/aaass////// 07987
Phillip Isola, Jun-Yan Zhu, Tinghui Zhou, and Alexei A Efros. Image-to-image translation with conditional adversarial networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1125-1134, 2017.
Laurynas Karazija, Iro Laina, and Christian Rupprecht. Clevrtex: A texture-rich benchmark for unsupervised multi-object segmentation. arXiv preprint arXiv:2111.1026; 2021.
Markus Karmann and Onay Urfalioglu. Repurposing stable diffusion attention for training- unsupervised interactive segmentation. arXiv preprint arXiv:2411.10411 2024. 2024.
Yasufumi Kawano and Yoshimitsu Aoki. Maskdiffusion: Exploiting pre-trained diffusion models
for semantic segmentation. IEEE Access, 2024.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Kon- rad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 9492-9502, 2024.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander Berg, Wan-Yen Lo, et al. Segment anything. Proceed- ings of the JEEE/CVF international conference computer vision, pp. 4015-4026, 2023.
Shu Kong and Charless C. Fowlkes. Recurrent pixel embedding for instance grouping. In Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
Theodoros Kouzelis, Ioannis Kakogeorgiou. Spyros Gidaris, and Nikos Komodakis. Eq-vae: Equivariance regularized latent space for improved generative image modeling. arXiv preprint 2.Xiv:2502.00509, 2025.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as proxy task for visual understanding, In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6874-6883, 2017.
Alexander C. Li, Mihir Prabhudesai, Shivam Duggal, Ellis Brown, and Deepak Pathak. Your dif- fusion model is secretly a zero-shot classifier. arXiv preprint 1rXiv22303.16203, 2023a. URL https: /ttpsi////xvv...//
arxiv. org/ abs abs /2003. 16203
Yanghao Li, Hanzi Mao, Ross Girshick, and Kaiming He. Exploring plain vision transformer back- bones for object detection, 2022. URL htttps://axxivv.orgg/ http:/aaxx.v...../aaaa/a
Ziyi Li, Qinye Zhou, Xiaoyun Zhang, Ya Zhang, Yanfeng Wang, and Weidi Xie. Open-vocabulary object segmentation with diffusion models. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pp. 7667-7676, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitickk. Microsoft coco: Common objects in context. In Computer Mision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
12

---

Published as a conference paper at ICLR 2026
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017.
Zheng Lin, Zheng-Peen Penng Duan, Zhao Zhang, Ch Guo, and Ming-Ming Cheng. Focuseut: Div- ing into focus view in interactive segmentation. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 2637-2646, 2022.
Qin Liu, Zhenlin Xu, Gedas Bertasius, and Marc Niethammer. Simpleclick: Interactive image segmentation with simple vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2020022220000 22300, 2023a.
Ruoshi Liu, Rundi Wu, Basile Van Hoorick, Pavel Tokmakov, Sergey Zakharov, and Carl Vondrick. Zero-1-to-3; sero-shot one image to 3d object, 2023b.
D. Martin, C. Fowlkes, D. Tal, and J. Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proc. 8th Int'l Conf. Computer Vision, volume 2. pp. 416-423, July 2001.
Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerg- ing pixel-level semantic knowledge in diffusion models. In The Twelfth International Confer- ence Learning Representations, 2024. URL https: httts::/openrevieet wwwpeeewwwwwwweeeeeeeeeeeeeeeeee htt.://wweetttottmee/ee/ee foremffo
YqyTXmF8Y2
Mehdi Noroozi, Hamed Pirsiavash, and Paolo Favaro. Representation learning by learning to count. In Proceedings of the IEEE international conference on computer vision, pp. 1898-5906, 2017.
Ege Ozguroglu, Ruoshi Liu, Dídac Surís, Dian Chen, Achal Dave, Pavel Tokmakov, and Carl Vondrick. pix2gestalt: Amodal segmentation by synthesizing wholes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3536-25444 2016.
Georg Petschnigg. Richard Szeliski, Maneesh Agrawala, Michael Cohen, Hugues Hoppe, and Ken- taro Toyama. Digital photography with flash and no-flash image pairs. ACM transactions graphics (TOG), 23(3) 2004.
Ben Poole, Ajay Jain, Jonathan T. Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXivv 2022.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint arXiv: 151::0611..4444 2015.
Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, al. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.007 14, 2024.
Rahul Ravishankar, Zeeshan Patel, Jathushan Rajasegaran, and Jitendra Malik. Scaling properties of diffusion models for perceptual tasks, arXiv preprint 1iXi::241100803: 2024.
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In Proceedings of the IEEE/CVF international conference computer vision, pp. 10912-10922, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in neural in- formation processing systems, 04225778-55244 2022.
13

---

Published as a conference paper at ICLR 2026
Jianbo Shi and Jitendra Malik. Normalized cuts and image segmentation. IEEE Transactions pattern analysis and machine intelligence, 22(8):888 900888900000000 2000.
Konstantin Sofiiuk, Ilia Petrov, Olga Barinova, and Anton Konushin. f-brs: Rethinking backpropa-
gating refinement for interactive segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 8623-8632, 2020.
Konstantin Sofiiuk, Ilya A Petrov, and Anton Konushin. Reviving iterative training with mask guid- ance for interactive segmentation. In 2022 IEEE International Conference on Image Processing (ICIP), pp. p 144.. IEEE, 2022.
Jascha Sohl-Dickstein, Eric A Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. Proceedings of the 32nd International Conference on Machine Learning, pp. 2256-2265 PMLR, 2015.
Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. Advances in Neural Information Processing Systems, 36: 1363-1389, 2023.
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Gonzalez-Franco. Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3554-3563, 2024.
Carl Vondrick, Hamed Pirsiavash, and Antonio Torralba. Generating videos with scene dynamics. Advances in neural information processing systems, 29, 2016.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Yeh, and Greg Shakhnarovich, Score jaco- bian chaining: Lifting pretrained 2d diffusion models for 3d generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 12619-12629, 2023a.
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M. Alvarez. Freesolo: Learning to segment objects without annotations. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp- 14156-14166, 2022. doi: 10. 10L19///CYP22268222222.33333
Xudong Wang, Rohit Girdhar, Stella Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 3124-3134, 2023b.
XuDong Wang, Jingfeng Yang, and Trevor Darrell. Segment anything without supervision. Ad- vances in Neural Information Processing Systems, 37:138731-138755 2024.
Lei Yang, Yan Zi Wei, Yisheng He, Wei Sun, Zhenhang Huang, Haibin Huang, and Haoqiang Fan. ishape: A first step towards irregular shape instance segmentation, 2021. URL https:
7/arxiv. wwtp://rg/abbaa//eeee
15068
Libo Zhang, Lutao Jiang, Ruyi Ji, and Heng Fan. Pidray: A large-scale x-ray benchmark for real- world prohibited item detection, 2022a. httts:/tt:://att /ttp://axwwi.oog///////1111111 107633
Lingzhi Zhang, Shenghao Zhou, Simon Stent, and Jianbo Shi. Fine-grained egocentric hand-object segmentation: Dataset, model, and applications, 2022b. URL
abs/
2208. 03826
Richard Zhang, Phillip Isola, and Alexei A Efros. Colorful image colorization. In Computer Vision- ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part III 14, pp. 649-666. Springer, 2016.
Richard Zhang, Phillip Isola, and Alexei A Efros. Split-brain autoencoders: Unsupervised learning by cross-channel prediction. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1058-1067, 2017.
Canyu Zhao, Mingyu Liu, Huanyi Zheng. Muzhi Zhu, Zhiyue Zhao Hao Chen, Tong He, and Chunhua Shen. Diception: A generalist diffusion model for visual perceptual tasks. arXiv preprint arXiv:2502. 17157, 2025.
14

---

Published as a conference paper at ICLR 2026
Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text- to-image diffusion models for visual perception. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5119-57299599 2023.
15

---

Published as a conference paper at ICLR 2026
A APPENDIX
A EXPERIMENTAL MODELS AND DATASETS
A.1 ADDITIONAL. DETAILS FOR MODELS AND BASELINES
Stable Diffiusion 2 (SD): We finetune Stable Diffusion 2 U-Net end-to-end using our loss. We follow the method in (Garcia al. 2024) which enables deterministic one-step prediction. This has been shown to outperform multi-step stochastic inference with standard diffusion training for other perceptual tasks. To train it in pixel space, we fix the timestep to the highest (999). We replace the input with our image's VAE latent without adding any noise. We set the CLIP embedding to the null condition. We freeze the VAE and finetune the U-net end to end with our instance coloring loss as described above. We show in Appendix B that the choice of timestep has negligible effect on the results as the timestep cross cross-attention within the U-net is finetuned too.
MAE with Decoder (MAE-B/H): We finetune MAE with the decoder end-to-end. In particular, MAE is trained only on images, so it allows us to explore if a strong generative prior (without any additional labels such text or class condition) is enough to learn strong segmentation features. Additionally, we can evaluate if internet-scale pretraining is necessary to generalize. We finetune both the ImageNet- IK pretrained MAE ViT-B (for a direct comparison with DINO and SimpleClick) and the ViT-H (for a rough comparison to SAM and Stable Diffusion). The masking ratio is set to 0%, so no tokens are masked. Our backbones use normalized tokens during pretraining, as this is the standard. We hypothesize this leads to the token artifacts seen in the qualitative figures. It is possible results may improve by using un-normalized tokens.
We train SimpleClick, state-of-the-ee promptable instance segmenter, on the same synthetic datasets that we train our generative models on. This helps us investigate whether our generalization is due to the generative model itself, or simply an effect of training on synthetic data. Our model initializes with an MAE ViT-B encoder, along with a feature pyramid learned from scratch. We finetune using its released training code.
SimpleClick:
We investigate whether a strong discriminative model can suffice for segmentation features without necessarily learning to synthesize images. However, we must pair it with a model that knows how to upsample images of all types from a low-dimensional space so that it will be able generate images of objects unseen in training. To investigate this, join ImageNet- 1K pretrained DINO ViT-B with the Stable Diffusion VAE decoder, and connect them with a single up-conv layer. Additionally, the latent features of images compressed with the Stable Diffusion VAE often look like the image itself (Kouzelis et al.) 2025). Thus, to succeed, all DINO needs to do is generate these object groupings a lower resolution.
DINO + VAE (DINO-B):
Segment Anything (SAM): SAM is a large-scale promptable instance segmentation model super- vised on over a billion masks from very large distribution of data. We use SAM benchmark to evaluate the extent to which our model generalizes across wide variety of images. While in some cases we outperform SAM, we do not mean for this to claim that our models are inherently superior to it. We use the publicly available VIT-H checkpoint and inference library.
Dataset Baselines: We finetune the models (MAE-H and SD) with the same loss and implementa- tion details. For 5 and 10 category models, we computed no loss for any pixels inside the bounding boxes of the objects that are outside the "allowed" categories. We also pruned any images with more than 35% of image area that is "not allowed" any images with allowed masks as found it important to have a smaller amount of high quality data, rather than a larger set of low quality data. Because of limited time, reported results for SD (10 classes) and SD (5 classes) after only 15k iterations of finetuning, rather than the full 30k. We will update our rebuttal draft with results from the full finetuning schedule.
To match their pretraining resolutions, SimpleClick, DINO, and MAE are finetuned at 224x224 resolution, rather than the standard ones described in implementation details.
16

---

Published as a conference paper at ICLR 2026
A.2 EVALUATION DATASETS
COCO We evaluate on the COCO 2017 validation set, as is standard for a large variety of segmentation tasks. To showcase our zero-shot generalization unseen mask categories, we choose a subset of COCO dataset with categories not seen our finetuning and call COCO exe dataset. COCO does not include object types that Hypersim does not have explicit category for, but we have observed to exist in the dataset (i.e. wine glass, teddy bear, potted plant). However, we have seen that our synthetic data does not contain any masks for humans or animals, so COCO,sc includes such images. COCO contains 86% of the images and 64% of the masks in the full COCO 2017 validation set. We provide a full list of categories in Appendix D
DRAM: Humans are able to perceive objects in wide variety of visual media with large amounts of abstraction. Unfortunately, most existing segmentation methods require high amounts of labeled data to generalize art. DRAM is a segmentation dataset that has annotated large variety of art pieces across styles and time periods, including many abstract styles such as impressionism, ink-and-wash, and cubism. We evaluate on the test set.
Egocentric vision is crucial for embodied Al and robotics. However, segmentation in egocentric tasks is challenging because the first-person views often have frequent occlusions, motion blur, and variable lighting, resulting in inconsistent and ambiguous object boundaries. The EgoHOS dataset provides segmentations of a large amount of egocentric images of humans interacting with everyday objects. We evaluate on the test set.
EgoHOS:
Many objects in the real world have fine, intricate structures. We evaluate models' abil- ity to accurately segment these objects using the iShape dataset. iShape is an instance segmentation dataset which contains many occlusions and complex, fine structures such wires or fences. We evaluate on the test set using the PNG Cityscapes style annotations.
iShape:
Humans effortlessly apply their understanding of object shapes even in scenarios never encountered in nature. For instance, TSA agents can quickly identify dangerous items in X-ray images of luggage. To assess our models' generalization this challenging scenario, we employ the PIDRay dataset, which features labeled examples of hazardous objects in baggage. This task is particularly demanding because many dangerous items are small and deliberately concealed within other objects. We evaluate on the "easy" subset of the test set.
PIDRay:
B ADDITIONAL RESULTS AND EXPERIMENTAL DETAILS
50
0.8
UT22
mibu
30
SAM
2.4
53 Uiiited
10
KAE-B
DIN0-8
mmpleChik
0.5
0.2
0.4 0.6
0.5
1.0
Promot
Figure 7: We plot the full precision-recall of the zero-shot edge detection on BSDS500. Our strongest models outperform SAM's precision when recall is low, suggesting their segmenta- tions lie on the exact boundaries of the corre- sponding objects. However, as shown in Figure Sl our models sometimes do not predict objects for certain regions of the image, leading to lower precision at higher recall values.
Figure 6: We evaluate segmentation quality as the number of prompt points increases. Our SD model marginally exceeds SAM 1 prompt point and recovers >82% of SAM's perfor- mance at prompt points. This surprising as learned mask encoder for mul-
do not tiple prompt points, but simply merge similar- ity maps computed individually from each point prompt.
17

---

Published as a conference paper at ICLR 2026
Original
GT
SD
Edges
MAE-H
Edges
SAM
Edges
Figure 8: Delineating edges objects in the scene a fundamentally ambiguous task. While models' outputs do not exactly match the ground truth (neither do SAM's), they represent one interpretation of the "objects" in the scene. Our model tends to include certain objects, such as the clouds or grass, in the background. This emerges without supervision and may be inherent bias from generative pretraining.
Livar
foep
Lincan
smooth &
Norm
mloU
0.2
X
V
30.4
V
X
V
29.3
V
V
Y
X
V
27.7
X
26.2
V
31.6
Table 3: Ablation study using mIoU at a single center point on COCO on our SD model.
SD timestep
COCO exe
COCO
COCO exe
DRAM
EgoHOS
iShape
PIDRay
999 (original)
57.6
38.8
8.5
48.2
40.0
51.4
30.9
4-9
58.8
37.5
6.2
48.9
28.6
50.9
31.5
[=l
57.9
38.4
8.5
47.8
31.3
49.2
28.0
Table 4: We ablate the role of the fixed timestep in results. As shown above, it does not make major difference on the results as the timestep cross-attention is finetuned in the U-net.
We explore if our masks improve with additional prompt points (Figure 6). We average IoU across datasets, excluding COCO M/S (since few small objects are detected in the first place). Remarkably, our SD model achieves over 82% mloU relative to SAM 9 prompt points, despite lacking learned prompt encoder for multi-point prompts or masks from any of the evaluated categories.
exe
Additionally, as shown in Figure 7 our Stable Diffusion variant outperforms SAM in the first quar- ter of the precision-recall curve, and our MAE VIT-H variant matches SAM for the first 20% of recall despite having substantially smaller training resolution when evaluated on edge accuracy on BSDS500. However, visualized in Figure many labeled regions exist at the interface between object and background (e.g., clouds, plants, or rocks). Our model tends to include these regions in the background. As result, no edges detected for some objects in features, and the preci- sion falls for higher recalls. This behavior is fully emergent, as our model has never seen masks of the overwhelming majority of objects present in the BSDS500 dataset. This hypothesis is further
18

---

Published as a conference paper at ICLR 2026
SD
MAE-H model
Augmentation
COCO
COCO
COCO
COCO
COCO
COCO
exc
Original
57.6
38.8
8.5
50.0
23.2
3.5
Solarize
58.6
35.5
7.7
50.8
21.3
1.1
Contrast 2.0x
53.6
36.7
8.6
46.7
19.7
2.4
Contrast 0.5x
61.4
38.4
8.5
49.6
27.6
3.2
Grayscale
56.7
34.0
7.7
45.5
23.0
3.1
Hue +0.3
55.8
34.2
7.9
49.7
21.8
1.4
Hue -0.3
57.0
35.9
8.2
51.0
23.2
3.5
Table 5: We evaluate models' zero-shot accuracy robustness to several image color changes and perturbations. As shown below, our models exhibit strong robustness to all with limited drop in mask quality. Particularly, the solarize augmentation results loosely suggest that our model is grouping based on low-level color/texture and has some sense of high-level grouping.
Last N blocks finetuned
coco
COCO exe
COCOS exc
MAE-H (original, full finetune)
50.0
23.2
3.5
layers (decoder only)
53.5
20.2
3.0
4 layers
45.5
16.3
2.5
layers
40.0
13.0
2.0
layer
36.4
10.6
1.4
layers (linear probe)
19.6
2.7
0.0
Table 6: We explore how many layers need to be finetuned for our method to work. Surprisingly, full finetuning does not yield performance gains over finetuning the decoder. Additionally, just one two layers need to be finetuned for reasonable performance.
supported by how our models with 5 and 10 classes only perform better. Specifically, they have no loss computed on high amount of the background (because we ignore loss for pixels inside the bounding box of unknown-con unknooy-category objects). As a result, our and 10 category models sometimes activate on objects that SD or MAE-H would consider background. Thus, we opt to report AP under 20% recall as the lower precision later the curve not due to inaccurate edges, but rather lack of segmented objects in those regions.
We present an ablation study in Table 3 to examine the effect of each loss component and the VAE normalization using mloU at single center point on the COCO dataset. First, removing the intra- instance variance loss (Cvar) (row causes performance to collapse. This is essential to our method; without it, the model does produce uniform masks. Eliminating the pixel-level separation loss (row 2) prevents the model from learning sharp object boundaries, causing it to slightly overestimate mask boundaries. Eliminating the mean-level loss (row 3) results in reduced mloU well, but instead affects the model's ability to discriminate smaller objects. Replacing the smooth & loss with le loss (row and the normalization of VAE outputs (Norm, row results mloU of 27.7 and 26.2, respectively. Both smooth & loss and normalization of VAE output helps the model converge lower loss values earlier. Overall, the complete model that integrates all these components (row 6) achieves the highest mIoU of 31.6, confirming that each element plays role in obtaining optimal performance, We also ablate the fixed timestep value for our SD model in Table
In Table 5] we explore our models' robustness to a variety of image perturbations. This helps show that our model is robust to changes color and texture, despite never receiving any supervision for this.
We also explore how deep we need to finetune our model to achieve strong results in Table 6 We start with full finetuning and gradually reduce the layers trained to the last N layers in the model, plus the final projection to pixel space, freezing the rest the model. We find that finetuning just the decoder leads to very similar performance to full finetuning. Additionally, we find that just one or two layers are enough to achieve reasonable performance. This suggests that the "core" object grouping information is highly saturated in the decoder, especially the last layers. This makes sense
19

---

Published as a conference paper at ICLR 2026
Class
No Compositionality
MAE-H
SD
SAM
aeroplane (19)
17.5
10.3
19.7
48.6
bird (13)
15.6
14.2
20.3
36.3
bicycle (6)
19.5
28.4
27.6
46.6
bus (46)
12.7
7.5
9.5
56.0
cat (17)
10.7
22.5
29.0
39.1
cow (19)
11.4
11.0
17.3
26.9
dog (18)
11.0
19.0
25.9
34.2
horse (21)
9.2
10.7
17.8
26.2
motorbike (7)
15.5
21.1
29.1
54.0
person (24)
9.2
27.3
31.0
54.1
sheep (19)
14.4
10.0
15.5
27.1
train (52)
30.9
25.0
34.0
38.7
bottle (2)
53.4
28.0
34.4
53.1
car (30)
18.7
10.6
13.6
57.7
potted plant (2)
54.4
36.2
38.4
58.4
TV monitor (1)
55.5
51.3
57.4
66.9
Table 7: Emergent part-based grouping Pascal-Part. We evaluate part-level grouping the Pascal-Part dataset (part masks for PASCAL VOC 2012) by measuring IoU between discovered part groups and ground-truth part masks. "No Compositionality" is baseline computed the IoU between each part mask and its corresponding whole-object mask (e.g.. cat head vs. whole cat), approximating performance with no part-level grouping. Bold indicates classes where MAE-H or SD improves by at least 5 IoU points over the baseline. Gray rows denote classes included in our Hypersim+VKS VK2 training mix.
because 1) since the rest of the model is frozen, only the representations that are available at the last layers will actually have an effect the output 2) these layers are the ones most responsible for synthesizing images, for which understanding object grouping is frozen.
Finally, in Table we quantitatively evaluate emergent part-based grouping on the Pascal-PartChen et al. (2014) dataset, which annotates part-level masks for the PASCAL VOC 2012 set. Bolded values represent classes for MAE-H or SD where our model performed least points better than the baseline.
We compare against baseline "No Compositionality" which calculated by evaluating IoU be- tween every part mask and its whole object mask. This represents the values close to what we would get there was no part-level grouping at all (i.e. predicting the whole cat when ground-trus -ruth is just the cat head). We also report results from SAM to serve high-water mark and provide context for our results.
We signs of part-level grouping for some classes such as bike, cat, cow, dog, horse, motorcycle, and person. This emerges without any supervision on part or whole-object masks these categories, suggesting it is inherent to the generative model.
Classes colored gray are included in our Hypersim+VE VK2 mix; thus we do not expect to see any im- provement them the whole-object supervision may overwrite knowledge of part-level groups.
C IMPLEMENTATION DETAILS
We train our models a node with four RTX6000 Ada GPUs. We train our models with batch size of 2, with gradient accumulation for steps, for an effective batch size of 32. We do this intentionally, as described in (Ke et al. 2024), to mix gradients between images sampled from Hypersim and Virtual Kitti 2. We train our model for 30,000 iterations, which takes about 29 hours for SDv2 and 12 hours for MAE ViT-H. However,
models show signs of overfitting, and performance would likely benefit from additional iterations, but we didn't explore this due to timing constraints. We sometimes struggle with memory constraints when finetuning Stable Diffusion, as
20

---

Published as a conference paper at ICLR 2026
some images in Hypersim have very high number of instances (2000+). Thus, compute the loss on up to 1250 instances at most. We normalize final pixel-level outputs as we observe it improves convergence. We start our learning rate at 6e-5, after 100 steps of warmup. We then decay a cosine schedule so that we end at 20 of our original learning rate. We train at a resolution of x80x640 for Hypersim and 368 1024 for Virtual Kitti 2 for our Stable Diffusion variants. For ImageNet-pretraine models (MAE, DINO, SimpleClick), we resize Hypersim images to x24x224 and randomly crop 224x224 region in Virtual Kitti 2 dataset. We set Asop and Amean to 300 and compute all losses in the range of [0, 255] to weight all terms equally. For Stable Diffusion, we finetune only the U-net and freeze the VAE. We set the text condition to the empty string. For all prompting experiments, we fix the threshold to 253 and use a window size of 9 (for joint bilateral smoothing). We also performed some minor data cleaning prior to training where we removed all images with masks and any scenes with less than 10 objects.
3
D LIST OF OBJECT TYPES IN HYPERSIM AND COCO EXC
For each dataset, we provide a list of labeled object types present, along with the number of objects with that type.
D.1 HYPERSIM
This list includes only objects with instance and class annotations. Certain objects which have instance-level annotations but lack class labels, such as teddy bears or potted plants, were placed into the "unknown" object category.
Total number of objects: 3,693,970
Objects by category: Unknown (1,375,739). books (1,149,313,, chair (334,422), lamp (21 (219, table (102,093). pillow (74,200), window (51,444), picture (46,102), cabinet (38,253), paper (34,420), sofa (30,895), blinds (29,462). clothes (28,917), door (20,410), box (19,769), desk (19,418), floormat (18,879), counter (14,446), bookshelf (12,887), shelves (11,886), sink (10,026), mirror (7,488), bed (6,001), towel (5,250), television (4,367), nightstand (2,803), bathtub (2,556), refrigerator (2,243), curtain (2,066), toilet (1 (1,068), dresser (717), wall (106), whiteboard (100).
D.2 COCO
We excluded object categories that either labeled in our train set or we have observed to appear. For example, have excluded "wine glass" "knife" these would be placed into Hypersim's "unknown" category. We have personally verified all of the objects below to not exist in the subset of Hypersim train on.
Total number of objects: 23,195
Objects by category: Person (11,004), traffic light (637), handbag (540), bird (440), boat (430),
truck (415), umbrella (413), cow (380), banana (379), motorcycle (371), backpack (371), carrot (371), sheep (361), donut (338), kite (336), bicycle (316), broccoli (316), cake (316), suitcase (303), orange (287), bus (285), pizza (285), horse (273), surfboard (269), zebra (268), sports ball (263),
elephant (255), tie (254), skis (241), giraffe (232), tennis racket (225), dog (218), cat (202), train (190), skateboard (179), sandwich (177), baseball glove (148), baseball bat (146), airplane (143),
hot dog (127), frisbee (115), fire hydrant (101), stop sign (75), bear (71), snowboard (69), parking meter (60).
21

---

Published as a conference paper at ICLR 2026
E
ADDITIONAL QUALITATIVE SAMPLES
Image
DINO-B
MAE-B
MAE-H
SD (10 Classes)
SD
SAM
DAC
Figure 9: Qualitative Results on the COCO [Li et al. 2014) dataset. These results are randomly chosen and not cherry-picked.
22

---

Published as a conference paper at ICLR 2026
Image
DINO-B
MAE-B
MAE-H
SD (10 Classes)
SD
SAM
1
Figure 10: Qualitative Results on the DRAM CChenn et al. 2022 dataset. These results are ran- domly chosen and not cherry-picked.
23

---

Published as a conference paper at ICLR 2026
Image
DINO-B
MAE-B
MAE-H
SD (10 Classes)
SD
SAM
Figure 11: Qualitative Results on the EgoHOS (Zhang et al. 2022b) dataset. These results are randomly chosen and not cierry-picked.
24

---

Published as a conference paper at ICLR 2026
Image
DINO-B
MAE-B
MAE-H
SD (10 Classes)
SD
SAM
Figure 12: Qualitative Results on the iShape YYang et al. 2021 dataset. These results are randomly chosen and not cherry-picked.
25

---

Published as a conference paper at ICLR 2026
Image
DINO-B
MAE-B
MAE-H
SD (10 Classes)
SD
SAM
Figure 13: Qualitative Results the PIDRay (Zhang al. 2022a dataset. These results domly chosen and not cherry-picked.
26

---

Published as a conference paper at ICLR 2026
DINO-B
MAE-H (5 classes)
MAE-B
MAE-H (10 classes)
MAE-H
MAE-H (COCO)
MAE-H (ClevrTex)
SD (COCO)
SD (ClevrTex)
SD (5 classes)
SD (10 classes)
SAM
SD
Figure 14: Qualitative comparison of all models on a challenging, in-the-wild image.
27

---

Published as a conference paper at ICLR 2026
DINO-B
MAE-B
MAE-1I
MAE-H classes)
(10 classes)
(ClevrTex)
(COCO)
SD (5 classes)
SD (10 classes)
SD (ClevrTex)
SD (COCO)
SD
SAM
Figure 15: Qualitative comparison of all models on a challenging, in-the-wii wild image.
28

---

Published as a conference paper at ICLR 2026
S
ANC
10
DINO-B
MAE-H (10 classes) MAE-H (5 classes)
MAE-B
MAE-H
MAE-H (COCO) (COCO)
MAE-H (elevrTex))
SD (COCO)
SD (ClevrTexx
SD (5 classes)
SD (10 classes)
SAM
SD
Figure 16: Qualitative comparison of all models on a challenging, in-the-wild image.
29

---

Published as a conference paper at ICLR 2026
DINO-B
MAE-H (5 classes)
MAE-B
MAB-H (10 classes)
MAE-H
MAE-H (COCO)
MAE-H (ClevrTex)
(COCO)
SD (Clevr(Tex)
SD (5 classes)
SD (10 classes)
SD
SAM
Figure 17: Qualitative comparison of all models on challenging, in tn-the-wildd wild image.
30

---

Tublished as a conference paper at ICLR 2026
DINO-B
MAE-H (S classes)
MAE-B
MAE-H (10 classes)
MAE-H
MAE-H (COCO)
MAE-H Tel)
SD (COCO)
SD (elevrTex)
SD (5 classes)
SD (10 classes)
SAM
SD
Figure 18: Qualitative comparison of all models on a challenging, in-the-wild image.
31

---

Published as a conference paper at ICLR 2026
DINO
MAE-B
MAE-I1
MAE-H classes)
MAE-H (ClevyTex)
MAF-H (COCO)
SD classe)
SD (10 classes)
SD (ClevrTex)
SD (COCO)
SD
SAM
Figure 19: Qualitative comparison of all models on a challenging, in- in-the-wit wild image.
32

---

Published as a conference paper at ICLR 2026
DINO-B
MAE-H (5 classes)
MAE-B
MAB-H (10 classes)
MAE-H
MAE-H (COCO)
MAE-H (CleviTex)
SD (COCO)
(ClevrTex)
SI (5 classes)
SD (10 classes)
W SAS
SD
Figure 20: Qualitative comparison of all models on challenging, in tn-the-wil wild image.
33


Reviews:
[{"review_id": "XCPdTkezqY", "reviewer": "Reviewer_qeaH", "summary": "The authors propose a method to adapt image generation models to perform instance segmentations across diverse classes including those unseen during training. Impressive improvements over strong baselines have been shown in the paper. Image generation models are adapted to output segmentation maps in a single step by training on various image segmentation datasets with novel loss functions. Ablations and analysis is also included.", "questions": "Will the pretrained models also generalize to medical segmentation scenarios?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The performance gains are good over baselines trained on the same data. The authors also test on unseen domains such as luggage X-ray images where they observe similar trends. Qualitative comparison is very helpful to understand these claims and quite impressive. The fact that the model performs so well on unseen domains makes the contribution of the novel losses and method design strong.", "weaknesses": "According to evaluation, the Stable Diffusion variant of their model is performing the best. However this a old model and it would be good to see how the performance scales when used with recent DiT based models such as SD3 and Flux.\n\nThe authors hint at part level understanding in Figure 3. It would good to perform evaluation on some part segmentation datasets [1,2] to check generalization of these models to finer granularity levels.\n\nTraining is conducted on a relatively small and limited diversity dataset. It would be good to explore the effect of expanding the dataset, eg. using SA-1B (or a subset). \n\n\n[1] Saha, O., Cheng, Z. and Maji, S., 2022, October. Improving few-shot part segmentation using coarse supervision. In European Conference on Computer Vision (pp. 283-299). Cham: Springer Nature Switzerland.\n\n[2] Vedaldi, A., Mahendran, S., Tsogkas, S., Maji, S., Girshick, R., Kannala, J., Rahtu, E., Kokkinos, I., Blaschko, M.B., Weiss, D. and Taskar, B., 2014. Understanding objects in detail with fine-grained attributes. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3622-3629).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "eg4L7RDpOu", "reviewer": "Reviewer_XWuX", "summary": "This work demonstrates that generative models such as Stable Diffusion and Masked Autoencoders (MAE) can be fine-tuned on a small set of labeled object masks to achieve general-purpose instance segmentation.\nRemarkably, the fine-tuned models exhibit strong zero-shot generalization, accurately segmenting objects of entirely unseen types and styles.\nThey also produce sharper boundaries and handle fine-grained structures with notable precision, behaviors that likely stem from their generative pretraining.\nThese results suggest that generative pretraining inherently encodes a grouping and compositional understanding of the visual world, enabling segmentation capabilities that are more generalizable and human-like across diverse domains.", "questions": "The main concern lies in the model’s difficulty with small-object segmentation.\nGenerative segmentation models, while powerful, can be somewhat uncontrollable, as it is often unclear what structures or priors the model has actually learned.\nThis may explain why small or less prominent objects tend to be omitted in the predictions.\n\nCould the authors elaborate on how controllability might be improved in such generative segmentation frameworks?\nAdditionally, what potential strategies do you envision for enhancing the detection and segmentation of small objects within this generative paradigm?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "### 1. Surprising zero-shot generalization and conceptual novelty:\nIt is impressive that the proposed generative models exhibit strong zero-shot generalization, accurately segmenting objects of entirely unseen types and styles.\nThis suggests that generative pretraining inherently encodes a grouping mechanism that transfers across categories and domains, supporting more generalizable and human-like perception.\nThis is arguably the first convincing demonstration of the potential of generative segmentation models to surpass traditional discriminative approaches in terms of generalization capability.\n\n### 2. Robustness to visual perturbations:\nThe models show high robustness to variations in color, illumination, and texture, indicating that their segmentation ability does not rely merely on low-level visual cues.\nThis robustness highlights a deeper structural or semantic understanding learned through generative pretraining.\n\n### 3. Strong qualitative performance:\nQualitative visualizations (see pages 7–9) show coherent reconstructions and smooth, well-aligned segmentation boundaries.\nThe results are visually compelling and provide strong evidence for the model’s superior perceptual grouping and boundary accuracy compared to discriminative baselines.", "weaknesses": "### 1. Limited performance on small objects:\nAs shown in Table 1, the proposed approach performs well on large objects but struggles significantly with small-object segmentation.\nThis limitation suggests that the model’s generative priors may bias it toward capturing dominant, large-scale structures, leaving fine-scale, small instances underrepresented.\n\n### 2. Imprecise instance coloring and unclear mask extraction details:\nThe color-based instance representation occasionally produces noisy or “dirty” colors, which raises concerns about the consistency and robustness of the predicted masks.\nThe paper would benefit from a clearer explanation of how these colored outputs are converted into binary or instance masks, with more explicit technical detail in the Methods section.\n\n### 3. Limited extensibility to semantic or text-guided segmentation:\nSince the framework relies heavily on color-based encoding, its applicability to semantic or  text-guided segmentation appears constrained.\nThe authors may consider discussing how this approach could be extended to tasks requiring semantic consistency or text/image guidance, which could broaden the impact of the work in future research.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "giUk4uEvtm", "reviewer": "Reviewer_BBSU", "summary": "The paper explores how generative models pretrained for image synthesis can be repurposed for perceptual organization. By finetuning Stable Diffusion and MAE with an instance coloring loss on limited object types, the models achieve strong zero-shot generalization, segmenting unseen objects with high accuracy. Remarkably, they rival or even surpass SAM on fine structures, demonstrating that generative models inherently learn transferable object grouping mechanisms without large-scale supervision.", "questions": "The major weakness of the paper is limited novelty.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1. GenSeg demonstrates that simply finetuning generative models such as Stable Diffusion can yield strong performance in category-agnostic instance segmentation.\n\n2. The paper is clearly written and easy to follow.\n\n3. GenSeg achieves SAM-level performance in certain out-of-distribution segmentation scenarios.", "weaknesses": "1. The novelty of this work is limited. Previous studies [1][2][3] have already shown that generative models can exhibit open-set segmentation capabilities without task-specific training. This paper mainly extends those ideas to category-agnostic instance segmentation.\n\n2. Methodologically, the paper lacks innovation. Finetuning generative models for segmentation is a well-established approach, and the proposed loss function is highly similar to existing techniques [4][5].\n\n[1] Generative prompt model for weakly supervised object localization[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 6351-6361.\n\n[2] Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023: 1206-1217.\n\n[3] Open-vocabulary panoptic segmentation with text-to-image diffusion models[C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023: 2955-2966.\n\n[4] Recurrent pixel embedding for instance grouping[C]//Proceedings of the IEEE conference on computer vision and pattern recognition. 2018: 9018-9028.\n\n[5] Efficient and accurate arbitrary-shaped text detection with pixel aggregation network[C]//Proceedings of the IEEE/CVF international conference on computer vision. 2019: 8440-8449", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lfLPoTNKrY", "reviewer": "Reviewer_P9Gg", "summary": "The paper introduces a novel fine-tuning approach that repurposes generative models such as Stable Diffusion and Masked Autoencoders  for category-agnostic instance segmentation. Despite being fine-tuned only on limited synthetic data (indoor furnishings and cars), these models exhibit strong zero-shot generalization, segmenting unseen object types and image styles -- often surpassing the heavily supervised Segment Anything Model.", "questions": "- Why not use the well known evaluation protocol for assessing quality of click (point-prompt) interactive segmentation? \n\n- Why not compare to pretrained SimpleClick but implement a custom training scheme on synthetic datasets?\n\n- Why SAM and SimpleClick are selected as beaselines? I believe there are some more powerful methods that have been released since; at least SAM2, if not digging deeper [3]. \n\n[3] AdaptiveClick: Clicks-aware Transformer with Adaptive Focal Loss for Interactive Image Segmentation. Jiacheng Lin et al. In TNNLS, 2024.", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Demonstrates good performance on unseen object categories and visual domains despite fine-tuning on a very limited dataset.\n\n- Shows that generative models (Stable Diffusion, MAE) encode transferable perceptual grouping abilities, enabling segmentation without large-scale supervision.\n\n- Achieves competitive results to SAM with a lightweight fine-tuning procedure using fewer resources and annotated masks.", "weaknesses": "- Questionable evaluation protocols\n\nActually, there exists an established experimental protocol for evaluating click-based segmentation methods, that implies testing on GrabCut, Berkeley, DAVIS, SBD, PascalVOC benchmarks. E.g., SimpleClick referred in the proposed paper as a baseline, reports values obtained in this benchmark. But here, the proposed method is not tested in the same setup, instead, an entirely different experimental protocol is used. Namely, SimpleClick was trained on synthetic datasets with limited number of classes and scenarios, and tested far beyond classes used for training. In such an artificial scenario, its low accuracy, otherwise suspicious, can be indeed expected -- as well as superior performance of a generalist model. When tested in the normal evaluation protocol, SimpleClick yields decent performance, though.\n\n- Using diffusion (and even SD!) for click-based segmentation is not novel \n\nThis idea was exploited in the early Conditional Diffusion Network (CDNet) [1] that dates back to 2021. In CDNet, global similarity of features from clicks and potential target regions was used in a diffusion process that spreaded the predicted logits of clicks within locally connected regions. The most recent M2N2 [2] utilizes self-attention maps produced by Stable Diffusion for unsupervised and training-free approach to instance segmentation. None of these works is cited or discussed, which might make a reader to overestimate the novelty of the proposed approach. \n \n[1] Conditional Diffusion for Interactive Segmentation. Xi Chen et al. ICCV 2021.\n\n[2] Repurposing Stable Diffusion Attention for Training-Free Unsupervised Interactive Segmentation. Markus Karmann and  Onay Urfalioglu. In CVPR 2025.\n\n- Results are questionable\n\nThe method was evaluated against trainable SimpleClick, custom pipeline based on DINO and off-the-shelf SAM. SimpleClick were trained according to custom training procedure, which naturally inhibits concerns about whether there were properly trained -- or their potential was not unleashed due to suboptimal hyperparameters, shortened training schedule etc. DINO was again used as a part of a custom pipeline, so similar concerns are applicable. Results achieved with SD, are significantly inferior to those of SAM on medium-sized and small-sized objects in COCO, DRAM, EgoHOS and PIDRay, in other words, in almost all scenarios, which makes me doubt the practical value of the proposed approach. It seems that on real world applications, it barely could compete SAM.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)