TRACE: Your Diffusion Model is Secretly an Instance Edge Detector
Authors: Sanghyun Jo
Ziseok Lee
Wooyeol Lee
Jonghyun Choi
Jaesik Park
Kyungsu Kim
Venue: ICLR 2026 Oral (2026)
Keywords: diffusion
unsupervised instance segmentation
weakly-supervised panoptic segmentation
inference dynamics
attention
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
TL;DR: TRACE turns pretrained diffusion models into annotation-free instance edge generators for instance and panoptic segmentation.

Abstract: High-quality instance and panoptic segmentation has traditionally relied on dense instance-level annotations such as masks, boxes, or points, which are costly, inconsistent, and difficult to scale. Unsupervised and weakly-supervised approaches reduce this burden but remain constrained by semantic backbone constraints and human bias, often producing merged or fragmented outputs. We present TRACE (TRAnsforming diffusion Cues to instance Edges), showing that text-to-image diffusion models secretly function as instance edge annotators. TRACE identifies the Instance Emergence Point (IEP) where object boundaries first appear in self-attention maps, extracts boundaries through Attention Boundary Divergence (ABDiv), and distills them into a lightweight one-step edge decoder. This design removes the need for per-image diffusion inversion, achieving 81× faster inference while producing sharper and more connected boundaries. On the COCO benchmark, TRACE improves unsupervised instance segmentation by +5.1 AP, and in tag-supervised panoptic segmentation it outperforms point-supervised baselines by +1.7 PQ without using any instance-level labels. These results reveal that diffusion models encode hidden instance boundary priors, and that decoding these signals offers a practical and scalable alternative to costly manual annotation. **Project Page:** https://shjo-april.github.io/TRACE.

Published as a conference paper at ICLR 2026
TRACE: YOUR DIFFUSION MODEL IS SECRETLY AN INSTANCE EDGE DETECTOR
Sanghyun Jo';
Ziseok Lee Wooyeol Lee",
Jonghyun Choi?.
Jaesik Park
Kyungsu Kim "Seuul National University, Scoul, Korea
OGQ, Scoul, Korea
ABSTRACT
High-quality instance and panoptic segmentation has traditionally relied on dense instance-level annotations such as masks, boxes, or points, which are costly, in- consistent, and difficult to scale. Unsupervised and weakly-superrised approaches reduce this burden but remain constrained by semantic backbone constraints and human bias, often producing merged or fragmented outputs. We present TRACE (TRAnsforming diffusion Cues to instance Edges), showing that -intag diffusion models secretly function instance edge annotators. TRACE identi- fies the Instance Emergence Point (IEP) where object boundaries first appear in self-attention maps, extracts boundaries through Attention Boundary Divergence (ABDiv), and distills them into a lightweight one-step edge decoder. This design removes the need for per-image diffusion inversion, achieving 81x faster inference while producing sharper and more connected boundaries. On the COCO bench- mark, TRACE improves unsupervised instance segmentation by +5.1 AP, and tag sug-supreeseed panoptic segmentation it outperforms point-supervised baselines by +1.7 PQ without using any instance-level labels. These results reveal that dif- fusion models encode hidden instance boundary priors, and that decoding these signals offers a practical and scalable alternative to costly manual annotation. Project Page:
githab..bb...................... writhaab.o/oie 1o..ooo///R/. TRACE
INTRODUCTION
Panoptic segmentation unifies semantic and instance segmentation and underpins real-world ap- plications, such as autonomous driving (Elharrouss et al. 2021 Zendel et al.) (022). However, achieving reliable instance-level delincation has long relied on dense pixel-wise annotations such as masks, boxes, or points, which are prohibitively expensive, inconsistent across annotators, and fundamentally hard to scale. These limitations motivate the search for annotation-free alternatives that can retain the fine granularity of supervised methods without the cost of labeling.
Recent unsupervised and weakly-supervised approaches (Sick al. 2025 Li al. 2024 attempt to bypass dense labeling. Unsupervised instance segmentation (UIS) eliminates explicit annotation by clustering semantic features from pretrained vision transformers (Caron al. 2021 Oquab al.) 20233. but these models are inherently optimized for semantic similarity across images rather than instance separation within an image. As shown in Fig. 2 existing UIS methods (Wang et al. 2023a Li & Shin, 2024) often merge adjacent objects of the same class or fragment single instances, and rely on heuristic assumptions such as a predefined number of objects. Parallel advances in weakly- supervised semantic segmentation have demonstrated near-supervised performance, achieving up to 99% fully supervised accuracy on VOC 2012 (Everingham et al. 2010) using only image-level tags To et al. 2023 2024a). However, extending this success to the panoptic setup still requires point or box annotations to disambiguate instances. These additional annotations remain costly and error-prone, particularly when objects overlap when annotators inconsistent.
Our key insight is that self-attention maps of rext-to-image diffusion models (Podell et al, 2023 Esser et al. 2024) encode instance-aware cues early in denoising. As shown in Fig. I [a), cross- attention does not reliably separate adjacent objects even with an explicit prompt, whereas self- attention at specific timesteps reveals instance-level structure. During the denoising process, the model transitions from noise to instance-level structure and then to semantic content. This raises
"Equal contribution.
'Corresponding authors: k.pak, kyskim)

---

Published as a conference paper at ICLR 2026
Naise
Image
Image
Noise
Reverse Process
Forward Process
Prompt: "wo Cats"
Prompt: (Nu)
550
Emergence Point
Atemmoo
Attention
Boundary Divergence
/ One-step
Edge Divtillation
(a) Observation: Instance Cues in Diffusion Models (b) TRACE: Extracting Instance Boundaries Figure 1: Emergence and extraction of instance cues in diffusion attention. (a) In reverse pro- cess, cross-attention remains semantic even with the prompt, whereas self-attention specific steps reveals instance-level structure. (b) TRACE selects the instance-emery
step using a temporal di- vergence criterion, extracts non-parann con-parmmtrri edges from self-attention differences, and refines them via a one-step distillation with the diffusion backbone to refine instance boundaries.
central question: can diffusion self-attention itself serve as an annotation-free source of instance- level edge maps?
To answer this, we introduce TRACE (TRAnsforming diffusion Cues to instance Edges), frame- work that decodes instance boundaries directly from pretrained text-to-image diffusion models. As illustrated in Fig, Πb), TRACE first identifies the Instance Emergence Point (IEP) by measuring temporal divergence to select the timestep where the instance structure is most pronounced. It then applies Attention Boundary Divergence (ABDiv) score criss-cross differences in self-attention and generate initial edge maps. At this stage, pixels within the same object exhibit nearly identical self-attention distributions, whereas pixels across different objects diverge sharply; this divergence peaks on true instance boundaries and provides a direct signal for instance edge extraction. To re- duce the computational cost of the per-image forward process test time, these edges are distilled into a one-step predictor that integrates the diffusion backbone with an edge decoder. The result- ing edges are used as boundary priors in downstream segmentation methods (Wang al.) 2023a] Jo et al., 2024a),, guiding the propagation to cleanly separate adjacent objects by splitting merged regions along instance boundaries (Fig.
Our key contributions are summarized as follows:
We observe that self-attention in diffusion models briefly yet reliably reveals instance-level
structure during denoising, unlike common vision transformers (see Tab. 55.
The proposed TRACE unifies key ideas for annotation-free instance boundary discov-
ery: the Instance Emergence Point and Attention Boundary Divergence.
TRACE enables annotation instance and panoptic segmentation: 1) Improves unsuper- vised instance segmentation baselines by +4.4 AP with only 6% runtime overhead; 2) With tag supervision, surpasses point-supervised panoptic models, up to +7.1 1.1 PQ on VOC 2012; 3) As seeds for SAM, outperforms open- vocabulary detectors by up +16.5 PQ (stuff).
Ground Trub
Resdine (MaaCat)
TRACE (Ows)
Bascline TRACE
Maiitte
MudCan TRACE
Prohdere
PrrMergg TRACE
A Pink
0.5
300
3D
COCO COCO
Figure/ 2: Effect of TRACE, (Left) Our instance edges decoded from diffusion self-attention for reconnection of fragmented masks and separation of adjacent objects, with white dotted circles marking corrected boundaries. (Right) Consistent mmm gains over baselines (Wang et al.) 2023a Li & Shin 2024) without instance-level annotations.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Unsupervised Instance Segmentation. Instance segmentation aims to delineate individual objects
and typically requires pixel-level annotations. Early methods (Wang et al. [022 Ishtiak et al. (023) learn pseudo masks from external features (Wang et al. 2021) but require training from scratch and show limited accuracy. Recent approaches, such as MaskCut (Wang et al. (023a), U2Seg NNu
al. (202)), and UnSAM (Wang et al.] 2024b), cluster features from pretrained vision transformers
like DINO (Caron al.) (021). These models strong at semantic grouping across images but are not explicitly designed for separating instances within an image. Therefore, these clustering methods
often rely heuristics such maximum number of instances or confidence thresholds and tend to merge adjacent objects of the same category.
To improve instance separation, CutS3D (Sick et al. 2025) and CUPS (Hahn et al. 2025) incor- porate monocular depth estimators (Ke et al., 2024] Yang et al. 2024) to split objects at different ranges. However, depth-based approaches struggle when neighboring objects lie at similar depth and they degrade on distant or small objects where estimated depth becomes blurry. By contrast, our diffusion-based strategy (TRACE) extracts instance boundaries from self-attention of pretrained diffusion models (Peebles & Xie 2023; [ao et al. 2023: Podell et al.) 2023 Esser et al., (024)). This boundary-centric cue does not assume the number instances, robust to object scale and distance, and refines existing unsupervised pipelines (Wang et al.) [202a Li & Shin 2024) without
any supervision or retraining, achieving up to 29.1% higher performance on COCO compared to depth-based methods (see Tab.
Weakky-supervisee Semantic and Panoptic Segmentation. Panoptic segmentation jointly re- quires semantic masks for "stuff" regions (e.g., grass) and instance masks for "thing" objects (e.g. person), which makes it one of the most annotation-intenive tasks segmentation. To reduce this labeling cost, weakly-supervised panoptic segmentation have been explored. Image-level class tags (Shen et al.) 202I) alone cannot separate instances. Bounding boxes are ill-suited for panoptic supervision because they provide only coarse rectangles for "thing" objects and cannot define the non-overlapping pixel-wise regions required for "stuff" regions. Consequently, point annotations (Fan et al. 2022 Li et al. 2023b) 2024) have become the dominant form of weak supervision. However, points vary across annotators and are often placed near object centers, which produces partial or missed instances and leaves adjacent objects merged (see Fig. 3
Meanwhile, in weakly-supervised se- mantic segmentation. recent tag- supervised approaches (Jo et al. 2024a| Yang et al., 2025a b) show that image tags alone can reach about 95% of fully supervised accuracy on the PASCAL VOC 2012 benchmark, indicating that tags are sufficient for semantics but not for instance sepa- ration. Therefore, we revisit tag su-
Image & GT
Point Annotations
TRACE (Ours)
Point
2. Missed Point
Iuman Blas In Weak Labels
Unblased Instance Edge
Figure 3: Example of human bias in COCO.
pervision and inject instance structure using diffusion priors: TRACE attaches to a tag-supervised
semantic model (Jo et al. 2003 2024a) and converts its pseudo semantic masks into pseudo panoptic masks by supplying instance- aware boundaries from diffusion self-attention. This model-agnostic design uses only image-level tags, cleanly separates adjacent objects, and first surpasses point-
supervised panoptic baselines (Liet 2024) on VOC and COCO benchmarks (see Tab. 2
Diffusion-Driven and vocablocayylary Segmentation. Recent approaches, including DiffCut Couairon al. 2024), DiffSeg (Tian al. 20244). and ConceptAttention (Helbling et al. 2025), repurpose the self- and cross-attention maps of pretrained -ext-to-imaee diffusion models (Peebles) & Xie 2023 [ao al. 2023 [isse et al. 2024 |Podell et al., 2023) for semantic segmentation by analyzing attention at fixed timestep without inversion. In parallel, open-vocabulary segmentation builds on contrastive pretraining (Radford et al. 2021) to map free-form text to visual concepts, enabling text-conditioned masks. Despite progress, such models (Liu et al.) 2024b You et al. 2023 Zhao et al. 2025) typically underperform reliable seeds under multi-tag inputs because they are hard to obtain from captions with limited tag coverage or in scenes containing multiple nearby objects. Compared to them, TRACE yields higher panoptic quality than open- vocabulaay detection when used as TRACE seeds for SAM in Sec. 6
vocabulary segmentation and struggle to produce
3

---

Published as a conference paper at ICLR 2026
Instance Edge Generation via Diffusion Forward
X
Frozen Diffusion
Xn
X
X
Seff-Am
SelF-Ath
Self-
Saft.
SA(X,,)
SA(Xx))
SA(X²)
Instance Edge
E
Attention
Boundary
Divergence
Semantic
Semantic
V Instance Emergence
V Attn-to-Edee Edge
(b) One-step Deff-Distilllation with Edge Decoder
Rec. Image Instance Edge
Input Image
One-step Fine-tuning
I
SelF-Att
Selr-Arth
Gell--Atin
D
Sall
g
Before Training
20% of Training
40% of Training
J Self-Distillation
Figure 4: Overview of TRACE. (a) Diffusion forward locates the instance emergence point t* (IEP) via KL peak and extracts the instance-aware attention SA(Xx+): ABDiv converts it into a pseudo edge map E. (b) One step self distillation at t=0 trains an edge decoder Gs with E, masking uncertain pixels. Training from E closes gaps in fragmented edges (green circles) and yields connected boundaries É. At inference, TRACE predicts E in single pass w/o IEP ABDiv.
3 METHOD
In this section, we outline an overview of TRACE in Fig. for a comprehensive understanding of our framework. Section 3.2 introduces the Instance Emergence Point (IEP), which selects the de- noising step where instance structure is most pronounced. Section 3.3 describes Attention Boundary Divergence (ABDiv), which converts criss-cross self-ateention differences into pseudo edge map. Section 3.4 details one-step distillation that trains edge decoder with the diffusion backbone to produce connected edges and enable real-time inference. Section 3.5 shows how our edges integrate with downstream segmentation models through Background-Guider Propagation (BGP).
3.1 BACKGROUND
Models. Diffusion (Rombach et al. 2022. Podell al., 2023) and flow matching models (Esser al. 2024: Lipman et al. 2022) generate images by learning reverse process from noise to data. Although their objectives differ, both familics use similar transformer backbones with self- and cross-attention. We refer to either diffusion model since our method is compatible with both. In typical text-to-image implementations, a VAE encoder maps an input image into a latent Xa {l), denoising network ey iteratively predicts noise (or velocity) on latents conditioned on an optional embedding c, and a VAE decoder D turns a generated image i D(X))- TRACE reads only self-attention maps from a diffusion model and does not require text prompts.
Self-Attention Collection and Aggregation. For a latent X, E HWxx step f, we form queries Q. = X.Wo and keys K, X,WK. The self-attention for block i (averaged over heads, rows sum to 1) (X,, softmax(Q,K) (TT/) [0, HW HWW ignore cross-attention and do
4

---

Published as a conference paper at ICLR 2026
not provide prompts. To fuse maps from blocks k. ,,..., N that may operate at different spatial sizes Wy (e.g., multi-scale stages in U-Net: for single- scale DiT-style backbones is identity), we upsample to Wmax and average: SA(X,) = +/4 Σx
We implement this with PyTorch forward hooks H. attention blocks; on each forward of the denoising network with inputs (Xo, t), the hook collects {SAA}, ,, performs the aggregation, and returns the map, H(Co, Xo. t) SA(X)., without altering model outputs.
3.2 IDENTIFYING THE INSTANCE- DENOISING STEP
We next ask: at which point of the denoising trajectory does self-attention truly become instance- aware? Early in denoising, self-attention maps are almost indistinguishable from noise. As steps proceed, we observe sharp rise in the maps. This peak coincides with the emergence of clear object boundaries, after which divergence gradually decreases as object shape stabilizes while semanties continue to refine. During inversion, the trajectory unfolds in reverse order: semantic instance + noise.
Leibler (KL. divergence between consecutive
Motivated by our observation, we propose the Instance Emergence Point (IEP) as the timestep t* where this divergence is maximized:
argmax DKL(SA(X,,.)) (re)) SA(X.))
(1)
<E(TT...,TN)
where TO
TN discrete timesteps (with Tke 1prev Tk-1). The self-attention
map at this point, SA(X,-), is denoted as S'nng and serves as our instance-aware representation. Specifically, KL divergence natural choice (Tian al. (024) because each row of SA(X,) is a probability distribution. Unlike mean-squared or absolute differences, KL's log-scale sensitivity amplifies subtle but meaningful variations in high-dimensional self-attention that directly align with boundary emergence (see Tab. 4. In practice, we adopt a fixed inversion stride of 100 steps for efficiency and accuracy. detailed analysis of step size and the distribution of the optimal timestep t" across diffusion backbones is provided in Fig.
3.3 EXTRACTING INSTANCE EDGES FROM CELE-ATTENTION
Left
Right
Left
Right
SA 1.y
SA Fl
SA 12
S AA 1. Top
Down
Top.
Down
(z, th pixel
(i, pixel
SALE J. 2-
SA
SA
SA,
(a) Instance Boundary (ABDiv: 0.114)
(b) Instance Interior (ABDiv: 0.027)
Figure 5: Illustration of Attention Boundary Divergence (ABDiv). Boundary regions (a) exhibit sharp attention divergence between opposite neighbors, whereas interior regions (b) remain stable, producing much smaller ABDiv values.
Neighboring pixels within the same instance exhibit similar self-attention maps, whereas those across instance boundaries differ, shown in Fig. 5 We convert this contrast into edges with Attention Boundary Divergence (ABDiv). a simple non-parametric score that transforms instance- aware self-attention maps into boundary maps without clustering annotations. We apply ABDiv on the instance-aware map SAinst = SA(X,-) identified in Sec. 3.2 For a pixel (i,]), ABDiv aggregates the divergence between opposite 4-neighbors:
ABDiv(SA);, D (SA:)1)) Dku SAig-1).
(2)
By default, ABDiv is computed with 4-neighborhood using opposite pairs (left/rihht and top/bottom) as defined in Eq. 2 An 8-neighborhood extension that adds diagonal pairs achieves the same accuracy while increasing computation by approximately 2x, so we adopt the 4-neighborhood in all experiments.
5

---

Published as a conference paper at ICLR 2026
3.4 ONE-STEP SELF- DELF-DISTILLATION WITH EDGE DECODER
Starting from the ansta self-attention map SAimst S. ;-,) identified in Sec. 3.2 we obtain an initial edge map E by applying ABDiv as defined in Eq. 2 Inspired by aseudo-labellng strategies for weak supervision (Ahn et al. 2019 Jo et al. 2024aj, we adopt reliability- based thresholding to mitigate label noise from ambiguous self-attention signals. Specifically, ternary map using the mean μ and standard deviation a of ABDiv scores: pixels > fr + o are edges 14
define
interior 0, and the intermediate range is marked as uncertain -1. These uncertain pixels are explicitly excluded from the loss computation, which effectively suppresses false positives while maintaining high recall (see ablation in Appendix |.. and Tab. TT).
To replace a per-image IEP+ABDiv computation at inference with a single pass, we fine-tune the diffusion backhone using Low-Rank Adaptation (Hu et al. 2022) and jointly train edge decoder G Beyond efficiency, the decoder also learns to complete fragmented edges, following common
practice in boundary detection (Xie & Tu 2015 Xiao et al., 2018 Su et al. 2023 Let I, E R#x x3 be the original and reconstructed images, E, E (H( 0) 1 be the pseudo edge map of ABDiv and the edge predicted by Gg. Our training objective is C( ()
(0) + Ledge (0, Φ) T| 7112 + DiceLoss(( F,, E). After training, a single forward pass at 1=0 produces a connected and precise edge map and removes the need for IEP and ABDiv during inference, shown in Fig. + Full algorithmic details are provided in Appendix
3.5 SEMANTIC-TO-INTTANI
MASK REFINEMENT WITH INSTANCE EDGES
We now use our instance edges to regularize and complete segmentation masks. Given seg- mentation masks, connected component labeling treats the TRACE edges as separators and assigns unique labels to connected regions that not cut by edges. Inspired by (Ahn et al., 2019). we de- sign the Gudded Propagation (BGP), as shown in Fig. 6. propagate each fragmented mask inside its instance boundaries to close gaps and produce smooth regions. We then iteratively merge overlapping masks whose intersection over union exceeds TBGP = 0.5 until convergence. This produces complete instance masks with our edges. We empirically find that performance re- mains stable over typical choices of TBGP on VOC, so simply use 0.5 in all experiments.
Inpet Imoge
Initill Fragmented Micks
Ourptt
Roudine Sonaon
Af.
My
Meral
TRACK (See. 3)
Blundary-Giiledd Propagation (BGP)
Intevvatlss
Our Edge Figure 6: Illustration of Houndary-Guidedd Propagation. Fragmented masks spread within instance edges, and intersections (yellow) are resolved by edge respecting merging.
Our Propagated Marks
IoU
EXPERIMENTS
Implementation Details. For fair comparison, we follow standard protocols (Wang al. 2023a Li et al. 2023b) and run all experiments on a single NVIDIA A100 GPU. Stable Diffusion 3.5 Large (SD3.5-LL) (Esser al. 2024), our default backbone, performs best overall among five diffusion backbones evaluated (Tab. 5). with VOC and COCO as the main benchmarks and five additional datasets reported in Appendix B Training details and evaluation metrics appear in Appendix C
Unsupervised Instance Segmentation. In Tab. TRACE refines masks produced by existing UIS
methods Wang al. (2023a); Li & Shin (2024) ) Wang et al. (2024b) and consistently improves
performance Apm with gains ranging from +3.6 to +5.3 points. For clarity, we group results into training-free free and fine-tuned methods, where the latter relies on a Mask R-CNN He et al. (2017) trained on pseudo instance masks. In particular, compared to the depth prior (Sick et al. 2025).
TRACE attains higher Apmk on COCO (+2.2/+2.1 on 2014/2017), highlighting the advantage of diffusion-drivenn instance edges. Qualitative results appear in Fig. 17
Weakly-supervise Panoptic Segmentation. We refine semantic masks from tag-supervised meth- ods (Jo al. 2023 2024a) with instance-aware edges to form pseudo panoptic masks and then train a standard Mask2Former (Cheng et al. 2022) following the common evaluation protocol. In partic- ular, DHR (Jo et al.) 2024a) with TRACE surpasses point-supervised counterparts (Li et al. 2023b 2024] by using only image-level tags (see Tab. and Fig. B). indicating that our edges provide the instance geometry that tag supervision lacks. Qualitative examples are provided in Fig. T6

---

Published as a conference paper at ICLR 2026
Table 1: Performance of unsupervised instance segmentation.
(a) Training-fres UIS
(b) Fine-tuned UIS
VOC 2012
COCO 2014
COCO 2017
VOC 2012
COCO 2014
COCO 2017
Method
Apmi
AR
Apmk
AR
Apm
AR
Method
Apm
AR
ApmA
AR
Apm
ART
MaskCut*
5.8
14.0
3.0
6.7
2.3
6.5
CuILER*
11.2
34.2
8.9
25.1
8.7
24.9
+ UnSAM*
14.5
6.9
CmS33
10.9
10.7
TRACE (Ours)
9.7
18.4
7.9
12.6
7.5
9.8
TRACE (Ours)
14.8
45.3
13.1
34.6
12.8
33.9
ProMerge*
5.0
13.9
3.1
7.6
2.5
7.5
ProMerge+*
11.1
33.1
9.0
25.3
8.9
25.1
TRACE (Ours)
18.2
13.1
7.8
11.2
TRACE (Ours)
15.0
43.8
13.3
35.1
13.0
25.8
Table 2: Performance of weakly-supervised panoptic segmentation.
VOC 2012
COCO 2017
Method
Buckbone
Supervision
PQ
PQ
PQ"
SQ
RQ
PQ
PQ
PQ"
SQ RQ
Punoptic FCN
Li
2021
ResNet-50
67.9
66.6
92.9
43.6
49.3
35.0
80.6 52.6
Fasker
Cheng
2022
ResNet-50
M
73.6
72.6
93.5
90.6
80.5
51.9
57.7
43.0
PSPS Fan et
2022
ResNet-50
P
49.8
47.8
89.5
29.3
29.3
29.4
Punoptic FCN
Li
ct al
2021
ResNet-50
P
48.0
46.2
85.2
31.2
35.7
24.3
Poin
20235
ResNet-50
53.8
51.9
90.5
32.4
32.6
32.2
75.1
41.5
EPLD Li
2054
ResNet-50
P
56.6
54.9
89.6
34.2
33.6
35.3
Poin22Mast
ct a1
2023b
Swin-L
P
61.0
59.4
93.0
37.0
37.0
36,9
75.8
47.2
EPLD
Swin-L
68.5
67.3
93.4
41.0
39.9
42.7
(Shen al. 2022
ResNet-18-WS
39.0
37.1
77.7
0.7
30.8
7.8
MARS
ResNet-50
41.4
39.8
85.3
83.0
57.8
11.7
13.3
10.2
58.3
11.8
(UUS
ResNet-50
I
50.4
48.5
88.9
86.6
60.1
29.5
31.1
28.9
62.5
39.3
DHR (Jo al. 2024a]
ResNet-50
45.0
43.3
88.3
83.3
17.5
18.1
69.3
14.8
TTRACE (Ours)
ResNet-50
56.9
55.2
91.0
88.4
63.4
32.8
32.7
32.9
75.5
42.5
-TRACE (Ours)
Swin-l.
I
69.8
68.4
96.2
94.5
71.2
43.1
42.5
43.5
83.8
55.3
MM: Full mask supervision (upper bound), P: One point per instance, X: Image-level tags only instance annotations) Reproduced results using the publicly accessible code. The rest are the values reported in the publication.
ABLATION STUDY
Component Ablation. Table shows how each stage steers TRACE from purely seman- tic toward instance delineation. Because the Instance Emergence Point (IEP) marks the timestep where semantic attention first be- comes instance-aware, it cannot be evaluated on its own: without the boundary scoring of ABDiv there is no measurable edge signal. Ac- cordingly, case (b) applies Attention Bound- ary Divergence (ABDiv) at purely semantic timestep, following prior diffusion approaches (Tian et al. 2024: Couairon et al. 2024), and yields almost gain, confirming that semantic self-attention alone cannot reveal instance edges. Intro- ducing IEP in case (c) pinpoints the denoising step where diffusion self-attention transitions from semantic grouping to instance structure, enabling ABDiv capture instance boundaries. Finally, case (d) adds one-step self-distillation to compress these transient cues into a single-pass predictor, eliminating per-image IEP and ABDiv inference and cutting latency from 3682 ms to just 45 per image (about 81 x faster) while preserving and even strengthening edge connectivity.
Table 3: Effect of key components on COCO 2014 with the UIS baseline (Li & Shin 2024
IEP
ABDiv (See. 3.3
Distill (See. 3.4
(Sec. 3.2
AP
(a)
3.1
(b)
3.2
8.2
Self-Distillation with Reconstruction. During one-step self-distillation (Sec. 3.4, we jointly opti- mize edge prediction and image reconstruction. While the edge loss Ledge compels the student to reproduce the teacher's instance boundaries, edges alone can overfit noisy or incomplete super- vision. Adding reconstruction loss Lree anchors the decoder to global image structure, stabilizing training and suppressing artifacts along low-contrast boundaries. This auxiliary objective yields smoother and more coherent edges and provides measurable gains in both accuracy and perceptual quality (Apmk from 8.9 to 9.4; SSIM from 0.71 to 0.83) without adding inference cost.
Instance Emergence Analysis. Figure 7 evaluates the Instance Emergence Point (IEP) along two axes. In Fig. a),, enlarging the denoising step size reduces the number of self-attention accumu- lation and thus latency, with negligible loss in Apmkk A step size of 100 strikes the best balance, maintaining about 9.4 Apmk with roughly 3 of IEP search per image. In Fig. ((b). the optimal timestep t* clusters tightly across five diffusion backbones, indicating model-agnostic semantic- to-instance transition and supporting a fixed step size without per-ine ver-emage or per-model tuning. Ex- tensive results in Appendix E.2 further confirm the robustness of IEP, demonstrating consistent 1.*

---

Published as a conference paper at ICLR 2026
A Apmk
Tradooff (Ours; 100)
SDIL
Ap
10000
SDXL
Latency (ms)
7500
S03.5-4,
FLUX.1--2v
5000
PIXAta
200
Latency
2500
Instance
09.195)
(199,299)
(299, 399)
4993
(199,599)
(299 699)
(699 (699.799) 7993
13
300
(a) IEP Stride (Step Size)
Semantic
(b) Distribution of Optimal
Figure 7: Analysis of IEP. (a) Accuracy-latent trade-off across different IEP step sizes. (b) Dis-
tribution of optimal timestep t" showing consistent semantic-to-in to-nstance transition.
Image & GT
SD1.5
SDXL
SD3.5-L
SD3.5-L
with IEP
with TEP
with IEP
without IEP
DINOv1 (VIT-B)
DINOv2-G
EVA02-E
CLIP (VīT-B)
LLaVA-13B
Figure 8: Visualization of self-attention maps with PCA.
distributions across datasets and classes, as well the empirical superiority of our KL criterion over alternative metrics (e.g., Entropy and Wasserstein).
KL vs. Other Metrics. To ensure our similarity choice is principled, we compared the Leibler (KI.) divergence with alternative met- rics for both IEP and ABDiv. In Tab. 4 Jensen-Shannon divergence (JSD) achieves the similar AP but requires computing two KL terms against a mixture distri-
Table 4: Similarity metrics for IEP and ABDiv.
IEP Metric
Latencyling
Apm
ABDiv Metric
Latencyfing
Apm
KI. (Ours)
3,082 ms
KL. (Ours)
ms0
9.4
ISD
5.120
ISD
980
MSE (L2)
1,232 ms
3.8
MSE (L2)
425 ms
6.8
MAE (L1)
924
MAE (LI)
412 ms
bution, increasing latency by more than 60%. Mean-squared (L2) and mean-absolute (L1) losses
reduce computation but sharply degrade accuracy, confirming KL as the most effective balance of precision and efficiency.
6 DISCUSSION
Superiority of Generative Diffusion Priors. To verify whether instance-aware cues are specific to diffusion models, we evaluate TRACE across 10 dif- ferent backbones, including diffusion and 5 non- diffusion foundation models (Oquab et al., 2023
Table 5:
Diffusion vs. Non-Diffusion. TRACE fine-tuning results COCO 2014.
Blue rows indicate diffusion backbones.
Method Backbone
Params
Ap
AR
Fang et al. 2024 Liu et al. 2024a Siméoni et al. 2025 Podell al. 2023) (see Tab. 5) Note that
ProMerge
3.1
7.6
Non-diffusion backbones (ABDiv only)
for non-diffusion backbones lacking temporal trajec-
TRACE DINOv2-G
1.1B
2.6
7.7
tories, apply ABDiv (Sec. 3.3) directly their
TRACE EVA02-E
5.0B
3.2
7.9
self-attention maps. Remarkably, even the small- est diffusion model, PixArt-a (0.6B) (Chen et al.)
TRACE DINOv3
7.0B
4.3
8.9
TRACE LLaVA
13B
3.8
8.4
2024], achieves 7.1 Apmkk significantly outperform ing the massive 72B-parameter Qwen2.5-VI. (Bai
TRACE Qwen2.5-VI.
72B
4.1
8.5
Diffusion backbones (IEP ABDiv)
et al. 2025) (4.1 APm*). This confirms that TRACE leverages the unique generative nature diffusion models, where instance boundaries emerge during denoising (IEP; Sec. 3.2). rather than typical seman- tic features found in discriminative multimodal models. Figure X visualizes this distinction: diffusion self-attention tightens along object bound- aries IEP, whereas non-diffusion attention collapses into coarse semantic blobs, failing to separate
+ TRACE SD155
0.8B
6.8
11.2
+ TRACE PixAnt-a
0.6B
7.1
11.8
TRACE SDXL.
2.5B
7.4
12.3
TRACE SD3.5-I.
8.1B
8.2
13.1
TRACE FLUX.1
12B
8.3
13.4
8

---

Published as a conference paper at ICLR 2026
Trputs
Outputs wio SAM
Quiputs w/ SAM
TRACE wi SAM TRACE
Finat Cautraunnce Paopli
Grounding DINO
0.80
Refinement
EPLD
Use instance boses
Imput Trasgee
Puint2Mask
YOLOE SAM
DHR
SAM Refinonen
VOC PO
Input Tryss
only
Best Seed wHo Iessance Labelo
TRACE
Grounded DENO w/ SAM
Uiee diflision
COCO PQ (%)
Panoptic
(A) Qualitasive Comparison of Reskly-angervieed Panoptic Segmentation with TRACE
(b) Quantiative Camparison of PQ
Figure 9: Importance of TRACE. (a) White dotted boxes mark regions where tag-supervised semantic masks are converted into panoptic masks by TRACE, cleanly separating adjacent in-
stances. (b) Quantitative results show TRACE+SAM outperforming open-vocabulary detectors and TRACE+WSS (e.g., DHR) surpassing point-supervised baselines.
GT Instance Mask
TRACE (Ours)
Canny
HED
PiDeNet
DiffusionLdge
Instamee-nware Edge
Gradient Edge
Texture-contran Edge
Texture-contran Edge P.DNNe
Texture Edge DillossnEdgs
Baselino (ProMergc)
Baseline TRACE
Canny
HED
Figure 10: Instance-aware edge comparison with existing edge alternatives.
adjacent instances. Furthermore, within the diffusion family, performance correlates positively with model capacity (SD1.5 FLUUX.1,, demonstrating that TRACE effectively scales with stronger generative priors while remaining model-agnostic in its applicability.
Why Annotation-Free Instance Edges? Recent efforts toward instance and panoptic segmentation often combine open- voccaulla detectors (Liu et al. 20246 You et al. 2023 Zhao et al. 2025) with SAM (Kirillov al., 2023). where the detector supplies instance-leve boxes and SAM refines them into masks. Despite this progress, such pipelines still depend on box annotations and struggle when scenes involve many adjacent objects or ambiguous text prompts. TRACE provides different alternative: it extracts instance edges directly from diffusion self-attention, requiring no instance supervision while offering clean separation of objects (Fig. 9.. These edges are complementary to SAM, since SAM excels at refining seeds into precise instance masks while TRACE provides those seeds. This combination exceeds all supervised open-vocabulary baselines. In addition, when integrated with rag-supervised semantic models, TRACE supplies the missing instance geometry and converts their outputs into complete panoptic masks, outperforming point-supervised methods on VOC and COCO.
(a) Input Image
(b) Ground-Trun Panoptic Mask
(c) Groound-Trutee Instance Edge
R 2
p
nnw
Figure 11: Example ground-truth instance boundary generation. (a) Input image. (b)
Ground-truth panoptic mask. (c) Ground-truth instance boundaries extracted from (b).

---

Published as a conference paper at ICLR 2026
Limitations of Conventional Edge Detectors. Figure 10 tests whether conventional edges can replace TRACE for instance segmentation by swapping each detector's edge map into our BGP pipeline (Sec. 3.5 AP falls from 9.4 (TRACE) to 1.2 with Canny (Canny| 1986 3.7 with HED
Xiie & Tu 2015). 4.1 I with PiDiNet (Su et al. 2023), and 4.3 with Diffusion (Ye et al. 2024).
The gap reflects objective mismatch: these methods are trained to predict RGB intensity- contours (not instance edges), which makes them sensitive to texture and illumination.
Quantitative Evaluation of Instance Boundaries. Standard edge detection benchmarks (e.g. BSDS (Arbelaez et 2010)) prioritize low-level texture or color contrast, which is misaligned with the goal of instance boundary detection. To evaluate instance- aware boundary quality, we construct a new bench- mark from COCO 2014 panoptic masks, defining ground-truth edges (see Fig. M. To assess quality, we report
Table 6: ation on COCO 2014 validation set against ground-truth instance boundaries.
Instance Edge Quality. Evalu-
Method
ODS
OIS
clDice
Canny
0.129
0.202
0.134
HED
0.347
0.443
0.446
pixels between distinct segments
PiDiNet
0.362
0,450
0.574
DiffusionEdge
0.428
0.485
0.576
ODS and OIS metrics (Xie & Tu 2015 Su al. 2023) for boundary precision and elDice (Shit et al.
TRACE (Ours)
0.889
0.899
0.826
2021 for topological connectivity (see details in Appendix F). Table 6 compares TRACE against
representative edge detectors (Canny) 1986, Xie & &Tu 2015| Su et al. 2023] Ye et al. 2024)
TRACE achieves an ODS of 0.889, more than doubling the performance of the strongest baseline (DiffusionEdge, 0.428). Conventional methods suffer from high false positives due to their sensitiv- ity to internal textures, resulting in low precision for instance delineation. In contrast, TRACE effee- tively suppresses non-boundary gradients by leveraging diffusion priors. Furthermore, the superior elDice score (0.826) confirms that TRACE produces topologically connected boundaries, which is a critical property for successfully separating adjacent instances in downstream segmentation tasks.
Limitations. While TRACE demonstrates consistent improvements across 11 real-worl, bench- marks, including autonomous driving (see results in Tabs. T 12 and 13). we identify limitations in specialized domains. First, for tiny instances (Re 0.01% % area) in satellite imagery (Wei et al, 2020 Waqas Zamir et al. 2019), performance degrades due to the spatial compression of the VAE in latent diffusion models. Second, on out-of -attof--isttibuttio medical images (e.g.. histopathology) (Kumar al. 2020 Naylor et al. 2019), the natural-iman image priors of standard diffusion backbones result in misaligned instance boundaries. We provide detailed quantitative results (Tabs. and [5)
and qualitative failure cases (Fig. 15) for these scenarios in Appendix E.4
Computational Overhead. In Tab. TRACE introduces minimal additional cost across different evaluation settings. For training-free unsupervised instance segmentation in Tab. [a), TRACE re- fines each image's 's masks during inference. which increases latency by only about 2%
7
Table 7: Computational Overhead of TRACE.
Phase (Dutuset)
TRACE (SDXL) TRACE (SD3.5-L)
Train (ImageNet)
8 days
10 days
Test (VOC2012)
0.1 hrs
0.2 hrs
Test (COC02014)
2.0 hrs
2.4 hrs
VRAM Usage
20 GB
32 GB
compared to the ProMerge (Li & Shin (024)). In contrast, for weakly-supervised panoptic segmen-
tation (Tab. [b). Tab. 2}. TRACE is used only once during training to refine pseudo instance or panoptic masks before the teacher network (e.g., Mask2Former) is trained, so there is no runtime overhead at inference.
CONCLUSION
TRACE demonstrates that text-to-image diffusion models naturally encode recoverable instance structure. By locating the Instance Emergence Point, extracting boundaries through self-attention, and compressing them into a fast one-step decoder, TRACE delivers sharp and connected instance edges in real time without any prompts, points, or boxes, or masks. Our extensive evaluation across diverse diffusion architectures confirms that this capability is intrinsic to the generative diffusion prior, consistently yielding superior instance boundary precision and topological connectivity com- pared to non-diffusion baselines and conventional edge detectors. These edges act as annotation-free instance seeds that boost both interactive systems like SAM and unsupervised/ pipelines, surpassing point- and box- box-supervised alternatives. Looking forward, the same principle opens opportunitics for video panoptic segmentation, medical imaging, and open- ppccbuooayylary group- ing where text and TRACE can be combined for scalable panoptic perception.
10

---

Published as a conference paper at ICLR 2026
8 ACKNOWLEDGMENTS
This work was partly supported by the KHIDI grant funded by the Korean government (MOHW) 20.RS-2025--22072333, the IITP grants funded by the Korean government (MSIT) INo.RS-2025- 02305581, SALRS-2025-254423555 (AI Star Fellowship-SNU), and No.RS-2021121133 (SNU AD)]. the Research grant from SNU, and the Strategic Hub grant for International Research Col- laboration of SNU.
Kyungsu Kim is affiliated with the School of Transdisciplinary Innovations, Department of Biomed- ical Science, Interdisciplinary Program in Artificial Intelligence (IPAT), Medical Research Center, and Al Institute SNU.
REFERENCES
Jiwoon Ahn and Suha Kwak. Learning pixel-level semantic affinity with image-level supervision for weakly supervised semantic segmentation. In CVPR, pp. 4981-4990, 2018.
Jiwoon Ahn, Sunghyun Cho, and Suha Kwak. Weakly supervised learning of instance segmentation with inter-pixel relations. In CVPR, pp. 2209-2218, 2019.
Shun-ichi Amari and Hiroshi Nagaoka. Methods of information geometry, volume 191. American Mathematical Soc., 2000.
Pablo Arbelaez, Michael Maire, Charless Fowlkes, and Jitendra Malik. Contour detection and hier- archical image segmentation. IEEE TPAMI, 33(5):898 916, 2010.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2.5-vl technical report. arXiv preprint arXiv:2502.1392 2025.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth 2023.
words: A vit backbone for diffusion models. In CVPR, pp.
Luca Barsellotti, Lorenzo Bianchi, Nicola Messina, Fabio Carrara, Marcella Comiaa, Lorenzo Baraldi, Fabrizio Falchi, and Rita Cucchiara. Talking to dino: Bridging self-supervised vision backbones with language for open-vocabulary segmentation. In ICCV, pp. 22025-22035, 2025.
Holger Caesar, Jasper Uijlings, and Vittorio Ferrari. COCO-Stuff: Thing and stuff classes in context. In CVPR, pp. 1209-1218, 2018.
John Canny. A computational approach to edge detection. IEEE TPAMI, PAMI-866):699-69,, 1986. doi: 10. 199.://AAM...1...77777777
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In ICCV, pp. 9650- 9660, 2021.
Junbum Cha, Jonghwan Mun, and Byungseok Roh. Learning to generate text-grounded mask for woen-world semantic segmentation from only image-text pairs. In CVPR, pp. 11165-11174 2023.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-a: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In ICLR, 2024.
Zhaozheng Chen, Tan Wang, Xiongwei Wu, Xian-Sheng Hua, Hanwang Zhang, and Qianru Sun. Class re-activation maps for weakly-supervised semantic segmentation. In CVPR, pp. 969-978, 2022.
Bowen Cheng, Alex Schwing, and Alexander Kirillov. Per-pixel classification is not all you need for semantic segmentation. NewrIPS, 34:17864-17875, 2021.
Bowen Cheng, Ishan Misra, Alexander G Schwing. Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In CVPR, pp. 1290-1299, 2022.

---

Published as a conference paper at ICLR 2026
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In CVPR, pp. 3213-3223, 2016.
Paul Couairon, Mustafa Shukor, Jean-Emmanuel Haugeard, Matthieu Cord, and Nicolas Thome. Diffcut: Catalyzing zero-shot semantic segmentation with diffusion features and recursive nor- malized cut. NeurIPS, 37:13548-1332 2024.
Songhe Deng, Wei Zhuo, Jinheng Xie, and Linlin Shen. Qa-clims: Question-answer cross language image matching for weakly supervised semantic segmentation. In ACM MM, pp. 3572-5586 2023.
Omar Elharrouss, Somaya Al-Maadeed, Nandhini Subramanian, Najmath Ottakath, Noor Al- maadeed, and Yassine Himeur. Panoptic segmentation: arXiv:2111.10250, 2021.
A review.
arXiv preprint
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion English, and Robin Rombach. Scaling rectified flow transformers for high-resolution image synthesis. In ICML, 2024. URL httpt: http://oppeeevvveee httpp:::/ppeeeeeeeee.... net http://wwwweeefoooooddeeoo.. Pot.frrr
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. IJCV, 88(2):303-3388 2010.
Junsong Fan. Zhaoxiang Zhang, and Tieniu Tan. Pointly-supervised panoptic segmentation. In ECCV, pp. 319-336. Springer, 2022.
Ruochen Fan, Qibin Hou, Ming-Ming Cheng, Gang Yu, Ralph R Martin, and Shi-Min Hu. Associ- ating inter-image salient instances for weakly supervised semantic segmentation. In ECCV, pp. 367-383, 2018.
Yuxin Fang, Quan Sun, Xinggang Wang, Tiejun Huang, Xinlong Wang, and Yue Cao. Eva-02: A visual representation for neon genesis. Image and Vision Computing. 149:105171, 2024.
Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pp. 3354-3361, 2012. doi: 10. 10.1109/CVPR..2222.444800
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance seg-
mentation. In CVPR, June 2019.
Oliver Hahn, Christoph Reich, Nikita Araslanov, Daniel Cremers, Christian Rupprecht, and Stefan Roth. Scene-centric unsupervised panoptic segmentation. In CVPR. pp.
2025.
Mark Hamilton, Zhoutong Zhang, Bharath Hariharan, Noah Snavely, and William T. Freeman. Un- supervised semantic segmentation by distilling feature correspondences. In ICLR, 2022. URL
net wwwet/oorrmmiddSa06655
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick. Mask r-cnn. ICCV, pp.
2980, 2017.
Alec Helbling, Tuna Han Salih Meral, Ben Hoover, Pinar Yanardag, and Duen Horng Chau. Con- ceptattention: Diffusion transformers learn highly interpretable features. In ICML, 2025. URL
htttps://gpenreview wttpriiwwwwweeeeeeee www.m//rrmmmmmmmm wtw.orrmmmm--ce000F33333
Edward J Hu, Yelong Shen. Phillip Wallis, Zeyuan Allen-Zhu. Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-rank adaptation of large language models. In ICLR, 2022. URL
https::/
net. wtt.:/eww.eee.ooom.m.eee/eeeeeee
Taoseef Ishtiak, Qing En, and Yuhong Guo. Exemplar-freesolo: Enhancing unsupervised instance segmentation with exemplars. In CVPR. pp. 15424-15433, 2023.
Guangfeng Jiang, Jun Liu, Yuzhi Wu, Wenlong Liao, Tao He, and Pai Peng. Mwsis: Multimodal weakly supervised instance segmentation with 2d box annotations for autonomous driving. In AAAI, volume 38, pp. 2507-2515, 2024.
12

---

Published as a conference paper at ICLR 2026
Sanghyun Jo, In-Jae Yu, and Kyungsu Kim. Mars: Model-agnostic biased object removal without additional supervision for weakly-superrised semantic segmentation. In ICCV, pp. 614-623, October 2023.
Sanghyun Jo, Fei Pan, In-Jae Yu, and Kyungsu Kim. DHR: Dual features-driven hierarchical rebal- ancing in inter- and intra-class regions for weakly-supervised semantic segmentation. In ECCV, 2024a.
Sanghyun Jo, Soohyun Ryu, Sungyub Kim, Eunho Yang, and Kyungsu Kim. TTD: Text-tag self- distillation enhancing image-text alignment in clip to alleviate single tag bias. In ECCV, pp- 341-357. Springer, 2024b.
Sanghyun Jo, Seo Jin Lee, Seungwoo Lee, Seohyung Hong, Hyungseok Seo, and Kyungsu Kim. COIN: Confidence score-guided distillation for annotation-free cell segmentation. In ICCV, pp. 20324-20335, 2025.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. In CVPR, pp. 9492-9502, 2024.
Beomyoung Kim, Youngjoon Yoo, Chae Fun Rhee, and Junmo Kim. Beyond semantic to instance segmentation: Weakly-supervised instance segmentation via semantic knowledge transfer and self-refinement. In CVPR, pp. 4278-4287, 2022.
Sangtae Kim, Daeyoung Park, and Byonghyo Shim. Semantic- aware superpixel for weakly super- vised semantic segmentation. In AAAI, pp. 1142-1150, 2023.
Alexander Kirillov, Ross Girshick, Kaiming He, and Piotr Dollár. Panoptic feature pyramid net- works. In CVPR, pp. 6399-6408, 2019.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan- Lo, et al. Segment anything. In ICCV, pp. 4015 4026,4026, 2023.
Alex Krizhevsky, lla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. NeurlPS, 15:1097-1105, 2012.
Neeraj Kumar, Ruchika Verma, Deepak Anand, Yanning Zhou, Omer Fahri Onder, Efstratios Tsougenis, Hao Chen, Pheng-Ann Heng, Jiahui Li, Zhiqiang Hu, Yunzhi Wang, Navid Alemi Koohbanani, Mostafa Jahanifar, Neda Zamani Tajeddin, Ali Gooya, Nasir Rajpoot, Xuhua Ren, Sihang Zhou, Qian Wang, Dinggang Shen, Cheng-Kun Yang. Chi-Hung Weng, Wei-Hsiang Yu, Chao- Yuan Yeh, Shuang Yang. Shuoyu Xu, Pak Hei Yeung, Peng Sun, Amirreza Mahbod, Gerald Schaefer, Isabella Ellinger, Rupert Ecker, Orjan Smedby, Chunliang Wang. Benjamin Chidester, That-Vinh Ton, Minh-Triet Tran, Jian Ma, Minh N. Do. Simon Graham, Quoc Dang Vu, Jin Tae Kwak, Akshaykumar Gunda, Raviteja Chunduri, Corey Hu, Xiaoyang Zhou, Dariush Lotfi, Reza Safdari, Antanas Kascenas, Alison O' N'Nii,, Dennis Eschweiler, Johannes Stegmaier, Yanping Cui, Baocai Yin, Kailin Chen, Xinmei Tian, Philipp Gruening, Erhardt Barth, Elad Arbel, Itay Remer, Amir Ben-Dor, Ekaterina Sirazitdinova, Matthias Kohl, Stefan Braunewell, Yuexiang Li, Xin- peng Xie, Linlin Shen, Jun Ma, Krishanu Das Baksi, Mohammad Azam Khan, Jaegul Choo, Adrián Colomer, Valery Naranjo, Linmin Pei, Khan M. Iftekharuddin, Kaushiki Roy, Debotosh Bhattacharjee, Anibal Pedraza, Maria Gloria Bueno, Sabarinathan Devanathan, Saravanan Rad- hakrishnan, Praveen Koduganty. Zihan Wu, Guanyu Cai, Xiaojie Liu, Yuqin Wang, and Amit Sethi. A multi-organ nucleus segmentation challenge. IEEE TMI, 19(5))1380-139) 2020. doi: 10.1109/TMI.201 2999999.....
Hyeokjun Kweon, Sung-Hoon Yoon, and Kuk-Jin Yoon. Weakly supervised semantic segmentation via adversarial learning of classifier and reconstructor. In CVPR, pp. 11329-11339 2023.
Dylan Li and Gyungin Shin. Promerge: Prompt and merge for unsupervised instance segmentation. In ECCV, 2024.
Jing Li, Junsong Fan, and Zhaoxiang Zhang. Towards noiseless object contours for weakly super- vised semantic segmentation. In CVPR, pp. 16856-16865, 2022a.
13

---

00.1199/TMI.2018.28657777 Published as a conference paper at ICLR 2026
Jing Li, Junsong Fan, and Zhaoxiang Zhang. Point-supervised panoptic segmentation via estimating pseudo labels from learnable distance. In ECCV. Springer, 2024. URL https:tp///...... :/wps:://wwwww// ecva.
net/papers/eccy 2024 /04e/papess. wwtp/paper..ceeeeeeeeeccca 024... pdf
Qizhu Li, Anurag Amab, and Philip H.S. Torr. Weakly- and semi semi-supervised panoptic segmentation. In ECCV, September 2018.
Ruihuang Li, Chenhang He, Yabin Zhang, Shuai Li, Liyi Chen, and Lei Zhang. Sim: Semantic- aware instance mask generation for bux-supervised instance segmentation. In CVPR, pp. 7193- 7203, 2023a.
Wentong Li, Yuqian Yuan, Song Wang, Jianke Zhu, Jianshu Li, Jian Liu, and Lei Zhang. Point2mask: Point-supervised panoptic segmentation via optimal transport. In ICCV, pp. 572- 581. 2023b.
Yanwei Li, Hengshuang Zhao, Xiaojuan Qi, Liwei Wang, Zeming Li. Jian Sun, and Jiaya Jia. Fully convolutional networks for panoptic segmentation. In CVPR, pp. 214 223, 2021.
Zhiqi Li, Wenhai Wang, Enze Xie, Zhiding Yu, Anima Anandkumar, Jose Alvarez, Ping Luo, and Tong Lu. Panoptic segformer: Delving deeper into panoptic segmentation with transformers. In CVPR, pp. 1280-1289, 2022b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft COCO: Common objects context. In ECCV, pp- 740-755. Springer, 2014.
Yuqi Lin. Minghao Chen, Wenxiao Wang, Boxi Wu, Ke Li, Binbin Lin, Haifeng Liu, and Xiaofci He. Clip is also an efficient segmenter: text-driven approach for weakly supervised semantic segmentation. In CVPR, pp. 15305-15314, June 2023.
Yaron Lipman, Ricky TQ Chen, Heli Ben- Ben-Hamu, Maximilian Nickel, and Matt Le. Flow matching for generative modeling. arXiv preprint arXiv:2210.02747, 2022.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. NeurlPS, 36, 2024a.
Sheng Liu, Kangning Liu, Weicheng Zhu, Yiqiu Shen, and Carlos Femmandez-Granda. Adaptive early-learning correction for segmentation from noisy annotations. In CVPR, pp. 2606-2616, 2022.
Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang. Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In ECCV, pp. 38-55. Springer, 2024b.
Roozbeh Mottaghi, Xianjie Chen, Xiaobai Liu, Nam-Gyu Cho, Seong-Whar Whan Lee, Sanja Fidler, Raquel Urtasun, and Alan Yuille. The role of context for object detection and semantic segmen- tation in the wild. In CVPR, pp. 891-898, 2014.
Koichi Namekata, Amirmojtaba Sabour, Sanja Fidler, and Seung Wook Kim. Emerdiff: Emerging pixel-level semantic knowledge in diffusion models, 2024. URL htttttss:/aaxxi..ooooggs
2441111733
Peter Naylor, Marick Laé, Fabien Reyal, and Thomas Walter. Segmentation of nuclei in histopathol- ogy images by deep regression of the distance map. IEEE TMI, 3882)448-459, 2019. doi:
Quang Nguyen, Truong Vu, Anh Tran, and Khoi Nguyen. Dataset diffusion: Diffusion-based synthetic data generation for pixel-level semantic segmentation. In
A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), NeurIPS, volume 36, pp. 76872-76892. Curran Associates. Inc.. 2023. URL
https: httros//ppooceiinss..eerppp.p..p http://ppoo.pppooe.eeeeeee/aaa// 222.5588222000002222000000008800
14

---

Published as a conference paper at ICLR 2026
Frank Nielsen and Gaëtan Hadjeres. On power chi expansions of f-divergences. arXiv preprint arXiv:1903.058/8. 2019.
Dantong Niu, Xudong Wang, Xinyang Han, Long Lian, Roei Herzig, and Trevor Darrell. Unsuper- vised universal image segmentation. In CVPR, pp. 22744-22754, 2024.
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Russell Howes, Po-Yao Huang, Hu Xu, Vasu Sharma, Shang- Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. ICCV, pp. 4195- 4205, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023. htt.:httts///aatt org/ wwwpo/////////////////.//eeeeeee /207 01 952
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pp. 7748-8763 PMLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10660 10695, 2022.
Shenghai Rong, Bohai Tu, Zilei Wang, and Junjie Li. Boundary-enhanced co-training for weakly supervised semantic segmentation. In CVPR, pp. 19574-19584, June 2023.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedi- cal image segmentation. In MICCAI, pp. 234 241. Springer, 2015.
Lixiang Ru, Heliang Zheng, Yibing Zhan, and Bo Du. Token contrast for suuee-sseeeriieed seman- tic segmentation. In CVPR, pp. 3093-3102, 2023.
Mihir Sahasrabudhe, Stergios Christodoulidis, Roberto Salgado, Stefan Michiels, Sherene Loi, Fab- rice André, Nikos Paragios, and Maria Vakalopoulou. Self-supervised nuclei segmentation in histopathological images using attention. In MICCAI, pp. 393-402. Springer, 2020.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In ICCV, October 2019.
Yunhang Shen, Liujuan Cao. Zhiwei Chen, Feihong Lian, Baochang Zhang, Chi Su, Yongjian Wu, Feiyue Huang, and Rongrong Ji. Toward joint thing-and-stur mining for weakly supervised panoptic segmentation. In CVPR, pp. 16694-16705, 2021.
Suprosanna Shit, Johannes C Paetzold, Anjany Sekuboyina, Ivan Ezhov, Alexander Unger, Andrey Zhylka, Josien PW Pluim, Ulrich Bauer, and Bjoemn H Menze. cldice-a novel topology -ppeeyy-preserving loss function for tubular structure segmentation. In CVPR, pp. 16560-16569, 2021.
Leon Sick, Dominik Engel, Sebastian Hartwig, Pedro Hermosilla, and Timo Ropinski. Cuts3d: Cutting semantics in 3d for 2d unsupervised instance segmentation. In ICCV, 2025.
Oriane Siméoni, Huy V Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
Zhuo Su, Jiehua Zhang, Longguang Wang, Hua Zhang, Zhen Liu, Matti Pietikäinen, and Li Liu. Lightweight pixel difference networks for efficient visual representation learning. IEEE TPAMI, 49551)4449,,14494, 2023.
Junjiao Tian, Lavisha Aggarwal, Andrea Colaco, Zsolt Kira, and Mar Conzalez-Franco. Diffuse attend and segment: Unsupervised zero-shot segmentation using stable diffusion. In CVPR, pp. 3564,3563, June 2024.
15

---

Published as a conference paper at ICLR 2026
Xinlong Wang. Rufeng Zhang, Chunhua Shen, Tao Kong, and Lei Li. Dense contrastive learning for self-supervised visual pre-training. In CVPR, 2021.
Xinlong Wang, Zhiding Yu, Shalini De Mello, Jan Kautz, Anima Anandkumar, Chunhua Shen, and Jose M Alvarez. Freesolo: Learning to segment objects without annotations. In CVPR, pp. 14176-14186, 2022.
Xudong Wang, Rohit Girdhar, Stella X Yu, and Ishan Misra. Cut and learn for unsupervised object detection and instance segmentation. CVPR, pp. 3124-3134 2023a.
Xudong Wang, Ishan Misra, Ziyun Zeng, Rohit Girdhar, and Trevor Darrell. Videocutler: Surpris- ingly simple unsupervised video instance segmentation. In CVPR, pp. 22755-22764, 2024a.
XuDong Wang. Jingfeng Yang, and Trevor Darrell. Segment anything without supervision. In NewrlPS, 2024b.
Yangtao Wang, Xi Shen, Yuan Yuan, Yuming Du, Maomao Li, Shell Xu Hu, James L Crowley, and Dominique Vaufreydaz. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut. JEEE TPAMI, 2023b.
Syed Waqas Zamir, Aditya Arora, Akshita Gupta, Salman Khan, Guolei Sun, Fahad Shahbaz Khan, Fan Zhu, Ling Shao, Gui-Song Xia, and Xiang Bai. isaid: large-scale dataset for instance segmentation in aerial images. In CVPRW, pp. 20.229
Shunjun Wei, Xiangfeng Zeng, Qizhe Qu, Mou Wang, Hao Su, and Jun Shi. Hrsid: A high- resolution sar images dataset for ship detection and instance segmentation. IEEE Access, 8:
00033--202542 2020.
Weijia Wu, Yuzhong Zhao, Hao Chen, Yuchao Gu, Rui Zhao, Yefei He, Hong Zhou, Mike Zheng Shou, and Chunhua Shen. Datasetdm: Synthesizing data with perception annotations us- ing diffusion models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), NewrIPS, volume 36, pp. 54683-54605 54695. Curran Associates, Inc., 2023. URL httpp::/ppooceeddssssss 4d040
nwwrips ec cc/pappp.pppeeee.eeeeee (tapsserrriieeepppe//0/000
fi
Coneerrcenneeeeceee.mmm .ddf]
Tete Xiao, Yingcheng Liu, Bolei Zhou, Yuning Jiang, and Jian Sun. Unified perceptual parsing for scene understanding. In ECCV. Springer, 2018
Jiahao Xie, Wei Li, Xiangtai Li, Ziwei Liu, Yew Soon Ong, and Chen Change Loy. Mosaicfusion: Diffusion models data augmenters for large vocabulary instance segmentation. IJCV, 133(4): 1456-1475, 2025.
Jinheng Xie, Jianfeng Xiang, Junliang Chen, Xianxu Hou, Xiaodong Zhao, and Linlin Shen. C2am: Contrastive learning of class-agnostic activation map for weakly supervised object localization and semantic segmentation. In CVPR, pp. 989-998, 2022.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In ICCV, pp. 1395-1403, 2015.
Jiarui Xu. Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In CVPR, pp. 2955 2266,, June 2023.
Lian Xu, Wanli Ouyang, Mohammed Bennamoun, Farid Boussaid, and Dan Xu. Multi-class token transformer for weakly supervised semantic segmentation. In CVPR, pp. 4310-4319, 2022.
Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In CVPR, pp. 10371-10381, 2024.
Zhiwei Yang, Yucong Meng, Kexue Fu, Feilong Tang, Shuo Wang, and Zhijian Song. Exploring clip's 's dense knowledge for weakly supervised semantic segmentation. In CVPR, pp. 20223- 20232, 2025a.
Ziqian Yang, Xinqiao Zhao, Xiaolei Wang. Quan Zhang, and Jimin Xiao. Ffr: Frequency feature rectification for weakly supervised semantic segmentation. In CVPR, pp. 30261-30270, 2025b.
16

---

Published as a conference paper at ICLR 2026
Yunfan Ye, Kai Xu, Yuhang Huang, Renjiao Yi, and Zhiping Cai. Diffusionedge: Diffusion proba- bilistic model for crisp edge detection. In AAAI, volume 38, pp. 6675-6683, 2024.
Muyang Yi, Quan Cui, Hao Wu, Cheng Yang, Osamu Yoshie, and Hongtao Lu. A simple framework for text-supervised semantic segmentation. In CVPR, pp. 7071-7080, June 2023.
Haoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang. Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang. Ferret: Refer and ground anything anywhere at any granularity. arXiv preprint arXiv:2310.07704, 2023.
Oliver Zendel, Matthias Schörghuber, Bernhard Rainer, Markus Murschitz, and Csaba Beleznai. Unifying panoptic segmentation for autonomous driving. In CVPR, pp- 21351-21360, 2022.
Wenwei Zhang, Jiangmiao Pang. Kai Chen, and Chen Change Loy. K-net: Towards unified image segmentation. NeurlPS, 11110026-103300 2021.
Yuzhong Zhao, Qixiang Ye, Weijia Wu, Chunhua Shen, and Fang Wan. Yoloe: Real-time seeing anything. In ICCV, October 2025.
Bolei Zhou, Aditya Khosla, Agata Lapedriza, Aude Oliva, and Antonio Torralba. Learning deep features for discriminative localization. In CVPR, pp.
2016.
Bolei Zhou, Hang Zhao, Xavier Puig, Tete Xiao, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Semantic understanding of scenes through the ade20k dataset. IJC,, 127:302-321, 2019.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In ECCV, pp- 696-712. Springer, 2022.
Yanzhao Zhou, Zhu, Qixiang Ye, Qiang Qiu, and Jianbin Jiao. Weakly supervised instance segmentation using class peak response. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 3791-3800, 2018.
Lianghui Zhu, Yingyue Li, Jieming Fang, Yan Liu, Hao Xin, Wenyu Liu, and Xinggang Wang. Weaktr: Exploring plain vision transformer for weakly-supervised semantic segmentation. arXiv preprint arXiv:2304.01184, 2023.
17

---

Published as a conference paper at ICLR 2026
Appendix
LLM Usage Disclosure
19
Comprehensive Review of Related Work
19
B.1
Diffusion Models in Segmentation
19
B.2
Open- Vocabulary Segmentation
20
B.3
Instance Segmentation
20
B.4
Additional Analysis of Unsupervised Instance Segmentation
21
B.4.1
Problem A. Adjacent Instance Merging
B.4.2
Problem B. Single Instance Fragmentation
22
Method Details
23
Reproducibility
23
Diffusion Models and Flow Matching Models
23
Self-Attention Accumulation
24
Details of IEP and ABDiv
24
C.S Details of One-step Self- Dell-Distillation with Edge Decoder
25
C.6 Details of Boundary-Guided Propagation
26
D)
Information Theoretic View of TEP
26
D.1
Row entropy and its time derivative
27
D.2
Inter-step KL and Fisher information
27
Comprehensive Analysis and Discussion
28
Ablation Studies on Design Choices
28
In-depth Characterization of IEP
29
E.3
Extended Benchmarking on Diverse Domains
30
Limitations and Future Directions
31
Datasets and Metries for Evaluation
32
18

---

Published as a conference paper at ICLR 2026
A LLM USAGE DISCLOSURE
We used large-language model (LLM) only for minor text editing, such as correcting typos and adjusting wording a formal academic tone. The LLM was not involved in research ideation, experimental design, implementation, or analysis. All scientific content and claims were conceived and written by the authors, who take full responsibility for the entire paper.
B COMPREHENSIVE REVIEW OF RELATED WORK
B.1 DIFFUSION MODELS IN SEGMENTATION
-inatto-mmage diffusion models (Peebles & Xie 2023 Bao et al. 2023 Chen et al. 2024, Podell
et al.) (223)) are trained image-caption pairs to learn complex visual text relationships, with cent versions such as SDXL (Podell et al. 2023) and SD3 (Esser et al. 2024) offering finer con- trol through Taasformer-baseed architectures. These models generate refined images by iteratively denoising latent representations, making them effective for detailed image synthesis. Recently, dif- fusion models have shown potential for segmentation tasks by generating geeudo-maske and aiding unsupervised segmentation. Relevant methods include:
Diffusion-based Segmentation. Recent approaches repurpose the internal representations of diffusion models for segmentation tasks. DiffCut (Couairon et al. (024) and DiffSeg (Tian et al. 2024) apply clustering algorithms, such as normalized cuts or K-means clus- tering, directly to self-attention maps generate class-agnostic semantic masks. However, these methods often require heuristic threshold tuning and struggle to distinguish adjacent instances of the same class. EmerDiff (Namekata et al. 2024) explores the emergence of pixel-level semantic knowledge by aggregating attention features across timesteps and finding semantic correspondences. While effective for semantic segmentation, EmerDiff (Namekata et al.] 2024) focuses on gathering stable semantic signals rather than detect- ing the transient structural boundaries between instances. Consequently, these clustering- and aggregation-basee based methods inherently prioritize semantic grouping (merging same- class pixels) over instance separation. By contrast, TRACE specifically targets the Instance Emergence Point (IEP; Sec. 3.2) and leverages Attention Boundary Divergence (ABDiv; Sec. 3.3 to capture high- gig-frreueeny boundary signals, enabling the precise delineation of individual instances without annotations.
Diffusion-Driven Dataset Generation. Several approaches leverage diffusion models to synthesize training data with pixel-level annotations. DatasetDM (Wu et al. 2023) and Dataset Diffusion (Nguyen al. (023) generate synthetic image-mast pairs train down- stream segmentation networks. MosaicFusion (Xie et al.) (0225) adopts tiling strategy for instance segmentation data: generates single-object images from simple prompts (e.g. "A photo of a cat") and composites them into 2 grid, assigning distinct instance IDs based on grid positions. While effective for data augmentation, this mosaic approach artifi- cially avoids the challenge of segmenting naturally adjacent overlapping objects within a coherent scene. In stark contrast, TRACE is not a data-synthesis pipeline but a decoding framework. We reveal that the early denoising steps of a pretrained text-to-image model already contain rich, recoverable instance-level cues. Unlike MosaicFusion's 's reliance on synthetic composition, TRACE directly extracts precise instance boundaries from sin- gle real image by exploiting the intrinsic Instance Emergence Point (IEP) and Attention Boundary Divergence (ABDiv), successfully separating complex adjacent instances with- out additional training or prompts.
Existing diffusion-based segmentation methods (Couairon et al., 2024 Tian et al. 2024) focus on later timesteps in the inversion process, when image structures are nearly complete. However, this focus limits their ability to identify and separate multiple instances, as instance-speciff information appears in the early timesteps but fades as the model builds the overall semantic structure. This novel use of early diffusion features enables TRACE to capture instance- information and perform instance segmentation effectively.
19

---

Published as a conference paper at ICLR 2026
B.2 OPEN- VOCABULARY SEGMENTATION
Recent methods combining multimodal models (MLLM/VLM) with segmentation frameworks, such as SAM (Kirillov et al. 2023) or its unsupervised counterpart UnSAM (Wang et al. 2024b) have enabled open-vocabulary panoptic segmentation. Numerous approaches utilizing CLI (Rad- ford et al. 2 demonstrated that free-form text prompts can guide segmentation (Zhou et al. 2022
Cha et al. 2023) Do et al.) 2024b] Barsellotti et al. 2025); however, these methods generally underperform compared to specialized segmentation models that rely on precise mask annotations (Cheng et al. 2022). More recent advancements, such as Grounding DINO (Liu et al.) (024b) and Ferret You et al., 2023), have incorporated box annotations and language models to improve de- tection. Despite this progress, these models often struggle to generate accurate instance seeds from multi-tag inputs, failing distinguish visually similar classes. Extending its efficacy beyond un- supervised and weakly-supervisee open-vocabulary frameworks by injecting robust instance-aware boundaries, yielding consistent per- formance improvements across multiple benchmarks (see Appendix E.3 and Tab. 13).
segmentation (Tabs. [ and (). TRACE effectively complements
B.3 INSTANCE SEGMENTATION
Instance segmentation (IS) aims to delineate and label individual object instances within images. Traditional IS approaches rely pixel-Ievel annotations, which can be resource-intensive, partic- ularly for large datasets. More recent work has expanded into unsupervised methods to address scalability concerns. Mcanwhile, panoptic segmentation (PS) methods, designed to handle both semantic and instance segmentation in a unified task, often serve as competitive baselines for in- stance segmentation capabilities. Panoptic models distinguish between "things" (countable objects) and "stuff" (amorphous regions) and can separate multiple instances of the same class. Here, we categorize recent IS and PS methods based on their reliance on annotations.
Fully-supervised Segmentation. Many established methods (Cheng et al. 2022 Li et al. 2022b Zhang al. 202T, Cheng et al. 202T Li al., 202T; Kirillov al. 2019 Xu Mask2Former (Cheng et (002) and Panoptic SegFormer eLi et al. 2022b), require dense, pixel- level annotations(i.ee, masks) to achieve accurate instance segmentation but face scalability issues due to high annotation costs. These models are designed for panoptic segmentation, but serve as baselines for instance segmentation. We refer to these methods as "panoptic models".
2023). ) such as
Point- and Box-Supervised Segmentation. These weakly-supervised methods (Li et al. 2023b
Fan et al. 2022 Li et al. 2018 2023a; [iing et al., 2024) aim to reduce annotation costs by using less detailed but instance-specific labels like points or boxes for each instance. These methods (Li jet al. 2023b Fan et al. 2022) typically operate in two stages: first, they generate assuddopanopple masks from weak annotations, and then panoptic model (e.g., Mask2Former) is trained using these pseudo-masks instead of ground-trutt annotations. Although amnotation effort is reduced, the reliance on manual points and boxes still presents a substantial cost. However, TRACE is annotation free.
Supervisd Segmentation. Tag-supervised instance segmentation uses image-level tags (e.g., "person")) without any instance-level details (e.g., location of each person). While only a single image-level tag is required per image, depending on the number of instances, multiple instance- level annotations may be required. Hence, tag supervision is sigmificantly cheaper than instance- level annotations.
Instance Segmentation with Class Activation Maps (CAM) Zhhou et al. 2016). CAM-
based instance segmentation methods (Kim et al.] 2022 Zhou et al. 2018) leverage heatmap and class tags for rough localization of class-related regions. However, CAM was initially designed for semantic segmentation, and its lack of precision in localizing specific instances limits its effectiveness for instance-level segmentation.
Weakly-supervised Semantic Segmentation (WSS). WSS methods (Yang et al. 2025b
Rong et al. 2023 Kim et al. [0033] K Kweon et al. 2023 Deng et al.
2023 Yi et al. 2023
Jo et al.) 2023 Zhu et al. 2023 Lin et al. 2023 Ru al. 2023 Xu al. 2022 Li al.
2022a
Liu et al. [022] Chen et al. [022] Xie et al. 2022] Fan et al. 2018) use image-level
tags captions and offer promising results in class- segmentation. However, WSS
20

---

Published as a conference paper at ICLR 2026
alone cannot produce instance-level masks without further refinement. TRACE provides the instance-level refinement that can extend WSS methods to tag-supervised PS methods.
We note that the use of class tags is the minimal possible annotation which allows models to focus on specific classes in an image, crucial for evaluating performance in PS tasks. Methods that do not incorporate class tags (e.g., U2Seg (Niu et al.) 2024)) are excluded from our comparisons with PS methods, as they cannot produce specilic class labels. U2Seg attempts unsupervised panoptic segmentation by combining MaskCut (Wang et al. 2023a) for unsupervised instance segmentation with STEGO (Hamilton et al., 2022) for unsupervised semantic segmentation. However, without tag supervision, U2Seg cannot generate correct class labels. generating labels such as "class I" instead "cat 1", which restricts its applicability. Some methods, like JTSM (Shen et al.] (02T), use class tags but demonstrate limited PS performance, further indicating gap in this domain. Remarkably, TRACE+WSS, which uses only tag supervision in the WSS method, outperforms methods using point supervision.
Unsupervised Instance Segmentation (UIS). UIS aims to perform instance segmentation without relying on any annotations. Existing UIS methods be broadly categorized based on their archi-
tectural foundation: DINO-based methods (Niu et al. 2024 Wang et al., 2024a 2023a Li & Shin
2024 like MaskCut (Wang et al., 2023a) and ProMerge (Li & Shin (024) apply feature clustering
with the graph cut algorithm on pretrained self-supervised vision transformen(iee. DINO CCaron
al. 202Th) backbones separate instances. SOLO-based approaches (Wang al. 2022 Ishtiak
et al. 2023) rely on CNN-based (e.g., DenseCL (Wang et al. 2021I) pseudo-mas generation but show poor performance and requires training from scratch, making them computationally expensive. UIS serves as a direct baseline for TRACE, as these methods attempt instance separation without any labels. TRACE stands out from existing instance segmentation literature by taking a fully unsu- pervised, edge-oriented approach instead of clustering features via graph cut. Also, instead of DINO or SOLO-based backbones, TRACE leverages generative diffusion model.
B.4 ADDITIONAL ANALYSIS OF UNSUPERVISED INSTANCE SEGMENTATION
Our method tackles the root causes of problems. Problem A. Adjacent instance merging stems from UIS methods (Wang et al. 2023a) relying on semantic backbones Caron et al. (202T). lacking instance-level distinction. We resolve this using instance-aware diffusion self-attention maps via IEP/ABDiv. Problem B. Single instance fragmentation arises from fixed hyperparameters (T, n) in graph-cut methods (Wang et al.) 2023a), leading to inconsistent granularity. We resolve these issues via non-parametric instance edge & BGP. Figure 20 Figure 21 and Figure T6] provide qualitative validation. Appendix B.4 and Table g give detailed analysis.
Component
Main Role
Contributions to Problem A
Contributions to Problem
IEP
Finds uptmal step for instance boundasies
Seporates adjucent insiacces in icature space
Provides intiial edge information for margging
ABD
Converts self-ane att--atttation maaps odge maps
Derects edge seeds betwees adjseent instances
Provides instance owore edge maps for anerging
Distill
Euhanees connectivity edge mags
Compleses edges between adjacent instances
Redines instance
resolve fragmentztion misks
RGP
random walk progopation
Separates
using refined
Propapates mergos fragmented
Table 8: Component contributions solve (A) adjacent instance merging and (B) single in- stance fragmentation. Each step builds on the previous one, with BGP finalizing adjacent instance separation and fragmentation resolution using the refined edges generated by IEP, ABDiv, and fine- tuning.
B.4.1 PROBLEM A. ADIACENT INSTANCE MERGING
Root Cause of Problem A: Limitations of Existing Backbones. Unsupervised Instance Segmen- tation (UIS) methods (Niu et al., 2024, Wang et al. 2024a| 2023a Li & Shin 2024) depend on pretrained "backbones" (e.g., DINOvI (Caron et 2021) to extract feature maps, which are sub- sequently partitioned into instance masks through techniques like graph cut (Wang et al., 2023b). However, these backbones were originally designed to distill information primarily about semantic segmentation -iin--ddinfiiiigg and classifying foreground objects from the background, rather than dis- tinguishing individual instances within class. This inherent design limitation makes it difficult to achieve true instance separation (c.g., distinguishing two adjacent boats in the top of Fig. 12) using only semantic features.
21

---

Published as a conference paper at ICLR 2026
MILM Backbones Share This Limitation, and More Parameters Do Not Resolve It. When prompted with "Describe this image" in Fig. LLaVA-13B's s self-attention maps only capture se- mantic features, while diffusion models (e.g., SD1.5) separate instances, capability absent in other foundation models. In Tab. S diffusion backbones substantially outperform CLIP/DINO/MLLM counterparts with comparable and fewer parameters.
Ground Truth
Initial Mask
Our Edge Map
Instance-Aware Mask
Problem A. Adjacent Instance Merging
Merging
MaskCut
Ours
Ours+MaskCut
Instance Fragmentation
Problem B. Single
Fragmentation
MaskCut
Ours
Ours+Masker
Figure 12: Additional examples of problem and and our solution.
Our Solution for Problem A. To overcome the limitations of semantic-oriented backbones, we introduce TRACE, framework that leverages the inherent spatial attention in diffusion models to generate instance-aware edges without requiring pixel-level annotations. TRACE harnesses self- attention maps from pretrained diffusion models, which capture instance boundaries through at- tention mechanisms tuned during pretraining. By introducing two novel metrics, IEP and AB- Div, TRACE autonomously identifies and reinforces instance boundaries, ensuring that adjacent instances are effectively separated, even when traditional feature maps fall short. Unlike conven- tional UIS methods (Niu al.) 2024 Wang et al. 2024a 2023a Li Shin 2024) that rely on mantic segmentation backbones, TRACE uses se sell-suerviiee fine-tuning step that enhances edge connectivity, achieving precise instance delineation across classes. This enables TRACE address the issue of adjacent object merging by focusing specifically on instance edges, demonstrating su- perior separation accuracy across challenging datasets (e.g., VOC (Everingham et al., (010)), COCO (Lin et al. 2014)), as shown in our experimental results (see Tab. Tab. (2). Our diffusion-based approach addresses the adjacency issue by focusing on instance-level edge generation rather than relying solely semantic features, making it scalable solution for UIS
B.4.2 PROBLEM B. SINGLE INSTANCE FRAGMENTATION
Root Cause of Problem B: Limitations of Graph Cut. Graph-cut based UIS methods (Wang al. 2023b a Li & Shin 2024) require a hyperparameter, т, which acts as threshold for the initial graph construction. In this approach, the image is represented as graph, where each pixel (or patch) corresponds to a vertex, and each edge represents the degree of similarity (i.e., affinity) between two pixels (or patches). The affinity Ai,j between vertices and j is measured using the cosine similarity SJ,j of their respective feature maps. If St, 2 т, vertices and j are connected by an edge: otherwise, they not connected.
Intuitively, т determines the sensitivity of the UIS method to similarities between feature maps. For instance, if is close to I, only pixels with nearly identical feature maps are considered connected. Conversely, = 0, every pixel is connected to every other pixel. The graph-cut algorithm then partitions the graph into minimizing the number of edges that need to removed. The foreground object identified using heuristic rule, and the corresponding partition is presented instance mask. For the UIS
partitions- foreground partition and background partition- by
22

---

Published as a conference paper at ICLR 2026
MaskCut Resalts with Various Hyperparameter Comhinations
Image
Ground Truth
Ours MaskCut
4
Figure 13: Drawbacks of CuskCut with various n, combinations. Due to the semantic-oriented nature of the DINO backbone used in MaskCut, increasing or decreasing does not result in the successful separation of the two adjacent instances. Rather, changing the hyperparameters leads to fragmentation and detection of false positives in the background.
method to detect at most distinct instances, this process needs to be repeated for iterations, where n is fixed hyperparameter.
The value of impacts the granularity of the instance masks produced by graph-cut based methods (see Fig. 13). fixed often leads to fragmentation of single instances or merging of multiple in- stances, cannot be dynamically adjusted for each image. Additionally, methods that fix iteration counts, n,, cannot detect more than n instances, failing in instance-dense images. This fixed tuning for and n may work on the average case, but does not generalize well across different images.
Our Solution for Problem B. By contrast, TRACE predicts edges between adjacent instances with- tuning hyperparameters (e.g., n., r) related instance count, making it more flexible and robust. edges effectively resolve fragmentation and separate adjacent instances, achieving both conceptual and practical advantages. Since TRACE-generated edges curately delineate each instance, we can use the edge map to construct a transition probability matrix for random walk on the pixel space. This spreads out the masks within the edges (increased IoU be- tween masks of the same instance) while effectively stopping masks from spreading over the edges (restricted loU between masks of distinct instances, even if the instances are adjacent). As result, digerggenerttion approach allows TRACE to handle both adjacent instance merging and single instance fragmentation, enhancing existing UIS methods (Li & Shin 2024
Experimentally. TRACE-generan
Wang et al. 2023a).
C METHOD DETAILS
C.1 REPRODUCIBILITY
Training follows standard diffusion model fine-tuning regardless of the backbone. During the step distillation stage (Sec. (344) LoRA adapters with rank 64 are optimized using Adam with a weight decay of 4e-5. Input images are randomly resized to the native resolution of each backbone, namely 512 x 512 for SD1.5 and 1024 x 1024 for SDXL and SD3.5-L. For UIS and WPS, dataset splits, augmentations, and evaluation protocols strictly follow the corresponding prior works (Jo et al. 2024a] Li et al. 2223bb Wang et al., 2023a) to ensure fair comparison; hyperparameters not specified above are inherited from the respective baselines.
C.2 DIFFUSION MODELS AND FLOw MATCHING MODELS
Diffusion models Rombach et al. 2022 Podell et al. 2203) and Flow Matching models (Esser et al., 2024 Lipman et al. (002) are generative models that learn to produce realistic data by reversing gradual noising process. We use the term diffusion model to refer to both, as our method is compatible with either. Starting from clean image Xo ~
Po. noise progressively added over
time to obtain X,:
= oec,
C~0.1
(3)
Here, Og and o, vary with time t and control how much of the image and noise mixed. Early steps keep the image mostly intact; later steps produce nearly pure noise. This design enables the model learn different levels of structure across time, from coarse to fine.
23

---

Published as a conference paper at ICLR 2026
A neural network eo(X,, is trained to prediet the noise e, minimizing the loss:
O* arg min Exg.te /
(x,4) (X,,
ell
After training, the original image can be approximately recovered by removing the predicted noise:
Xo 1 (X: OrE9- (Xe,t)
Ox
We denote this forward-and-reverss process as Reconstruct(eor. tt.
Flow Matching models follow a similar principle but learn the velocity X, of the diffusion process using a network They minimize Exp.t.e (e,(,,, (á + ozt) with reconstruction given by Xo (vo(X,, Ōze).
C.3 SELF-ATTENTION ACCUMULATION
For noised image X₁ time step I, the self-attention mechanism computes spatial dependencies using weights Wo, WK:
Q X, Kx=XWW E RRHW
xd
(6)
(QcK" E (0, H|JW*H
SA(X.) = softmax
(7)
Vd
where d is the dimensionality of the attention heads, and H, W are the height and width of the self-attention map.
In text-to-image diffusion models, multiple self-attention maps SA SAN are produced at var- ious resolutions Wi..... WN (w, is the width of the ith SA map). We accumulate these maps by upsampling each to the resolution of the highest-resolution map, followed by averaging across all maps:
N
j 82 1
SA =
SA
(8)
N
8h
Wi
ISJEN
Wek
The hook function Xo. extracts self-attention maps evaluated in the forward pass at timestep & given input image No and returns the accumulated attention map in Eq. 8
C.4 DETAILS OF IEP AND ABDIV
Algorithm D (IEP; Sec. 3.2): Identifies the optimal diffusion timestep )" for instance separation by locating the peak in the Kullback-Liilerr (KL) divergence between consecutive attention maps. As derived in Appendix D this peak theoretically corresponds to the point of maximum Fisher information with respect to the noise level, marking the rapid emergence of structural instance cues. Empirical consistency of this metric across diverse models and random seeds is further validated in Appendix E.2
Algorithm IEP: Xo (Image) ++ SAinst
Require: Image Xo. Sequence of timesteps To < T1 Tv, Frozen Diffusion Model eØ- (:..)
1: Initialize: KL-gap+- 0, SA + 0
2: Xp t- Reeonstruut(eer... Xp.
Hook 74
3: SA(X,,, n.) < H(Eg- Xo: To)
4: for = -1 N do
5: Xo <- RReconstruct(een e Xo, Ta)
Hook H
6:
SA(X,,) +- H(Ea-,X0 F
7:
if KL-gapp (KL
SA(X,.) )) then
8:
KL-gap<- (SAX -._.)
SA(X,.))
9: SAmst + normalized SA(X,.)
10: end if
11: end for
12: return SAinst
24

---

Published as a conference paper at ICLR 2026
Transforms instance-aware self-attention maps into secudo-edge maps without annotations. Crucially, this algorithm implements a reliability-basse filtering step: pixels with ABDiv scores in the intermediate range (μ - o, o)) are labeled uncertain (E,, -1) because they typically correspond to ambiguous texture gradients or noise rather than definitive instance boundaries. By explicitly masking these regions, we prevent the introduction of label noise during the subsequent self-distillation phase, ensuring that the edge decoder learns only from high- confidence boundary signals.
Algorithm 2 (ABDiv; Sec. 3.3);
Algorithm 2 ABDiv: SAmma + E (Pseudo Instance Edge)
Require: Self-Attention Map SAinst
1: Initialize: Eg < Hн
2: for (i,j) in [H] x [W] do
3: Eoli,j] < ABDiv SAins))
4: end for
5: μ + mean of Eg
6: σ standard deviation of Eg
1 Eg 1 a]
Edge Pixel
7: E
Eg 4 --
Interior Pixel
(-1) [otherwise]
Uncertain Pixel
8: return E
C.5 DETAILS OF ONE-STEP SELF-DISTILLATHE WITH EDGE DECODER
Fine-tunes the transformer layers of text-to-image diffusion models and trains edge generator Ga from seratch. For the architecture of Gg. we lightweight CNN decoder for simplicity, and in Tab. 9 we show that replacing it with heavier designs such as U-Net or Mask2Former-styl decoders yields negligible gains, indicating that most of the instance- instance-boundary detail already resides in the TRACE edge cues themselves.
Algorithm (Sec. 3.4):
Algorithm One-step Self-Distillation of TRACE
Require: Image Dataset I, Pseudo-Edge Set {Er Hee Edge Generator G4 Pretrained Text-to-ine Text-to-Image Diffusion Model cæ for Epoch in 1... do
2:
L C-0
3:
for 7 I do { Reconstruct T, 0)
4:
Hook H
5:
É Go(Hl(c, I, 0))
Predicted Edge
6:
E + Er
Pseudo-Edge
7:
C+ [|+ 1|-111-1
1||2+ -
DiceLoss E)
8:
end for
9: Backprop on L(),6)
end for
Dice loss is computed as follows:
2 W E., E., WW., F2, + Σ W., 63,
DiceLoss(E,E) =
(10)
and the weighting W.,, 1[E,, # -1] excludes uncertain pixels. We exclude uncertain pixels (where -1) from the dice loss computation via to allow the model to focus on confident edge and interior points.
25

---

Published as a conference paper at ICLR 2026
C.6
DETAILS OF BOANDARY-GUIDEE PROPAGATION
Algorithm 4 UIS/WSS Segmentation Masks with TRACE
Require: Image I
Trained Edge Decoder Gø-
Fine-Tuned Diffusion Model 6p-
Masks Mi.... My from a UIS/WSS Method
1: Í f (erconstructeep... -)
Hook H
2: E <- (H 1,1= 0))
Predicted Edge
3: Identify Connected Components using É and CCL
4: М, Proopagaee Mask M, via BGP
Figure
5: Merge Masks with IoU TBGP
6: Return Final Masks
We apply the random-walk methods proposed in (Ahn & Kwak 2018 Ahn et al. 2019). Here, we outline high-level overview of the propagation technique.
Sparse Affinity Construction: From pseudo-edge map E, construct sparse affinity ma- trix Apase by calculating affinities between each pixel and its neighbors along predefined paths. Paths with edges (indicating boundaries) have low affinities, while edge-free paths have affinities closer 1.
2. Sparse to Dense Affinity:
Convert Asparse into dense matrix Adense by filling in affinities for every pixel pair. For symmetric consistency, a path exists between pixels and j. then both Adense [i, j] and Adese (j. i] assigned the same value.
3. Seed Propagation: Compute the transition matrix T = D-1
dense where D normalizes
8B rows of dense After t iterations, updated masks generated via:
vec MM) = T vec( M, @ (1 É)).
(11)
where (1 - E) prevents edge pixels from influencing neighbors, focusing propagation on non-edge regions.
This method creates more coherent and complete instance masks.
D INFORMATION THEORETIC VIEW OF IEP
Setup and Assumptions. Fix query pixel i and let the i-th self-attention step be the softmax distribution:
exp(1ss)) Zi(i)
Pi (1)
Zi(i) Σ exp((1sis)
(12)
where 8ij J/ k, is time-ivariant' "clean" similarity of i.j (similarity in the clean image) and %t 0 is effective inverse temperature. We assume throughout:
A.1. (Monotone schedule.) 7/ C²,, takes values in [o, 1]. and strictly decreasing along the forward trajectory (noise increases), with Tt=o 1 (clean image) and 10=0000 Rs (pure noise).
A.2. (Bounded logits.) There exists S < oo such that |sk|| < S for all i,j
A.3. (Non-degenerate row.) For a given i, the set of similarity values {sighs contains at least wo distinct values.
High-level Intuition. Under the standard forward diffusion parameterization 2; agzo + ozE, N (O I), the nooiieeo-noise ratio (SNR) at oo? decreases along the forward trajectory as 1. Thus, it is natural and empirically accurate model 7/t strictly decreasing function of / such that
26

---

Published as a conference paper at ICLR 2026
Te SNR(t). At pure noise (e.g., 1000), lower value of n gives higher entropy; pure signal (e.g., t = 0), a high value of "/ yields peaked assignments.
These minimal assumptions suffice for the identities below.
D.1 Row ENTROPY AND ITS TIME DERIVATIVE
For the row entropy Hi(i) -E logpr(i t), classical exponential-family algebra
Amari & Nagaoka 2000) gives:
(H, - Var (11)
(i( = log (1) - 7 (1)
(13)
because
E [s( = Var Varp((-19)(55)) and dyy. log Ze = I (-1)))) By the chain rule, d dhe
d.H((1) dH.(i) t1 -9 Var (-19)ss))))
(14)
dt
dy
Since t < along the forward trajectory (A.1.), Eq. 14 says row entropy increases from the clean end toward the noise end ("slow-fast-slow by the variance of 8, under the current row distribution.
magnitude, as we soon show), with the rate controlled
Endpoint Behavior and Boundedness. By A.2., the variance of st. under pe(: |t) is bounded by:
Var Varp.t-10/s..] = Σ peli [s)j (x0010)5))) S S <
(15)
Hence,
dNa(() dt
S S0
At pure noise. At the noise end, 7t - 0, so (1) regardless of the exact vari- ance. This thcoretical result matches the small temporal change of near-uniform attention observed before IEP.
At pure signal. At the clean image end, a different mechanism makes (dH,(i)/d /df small: if for j arg max, Sij and the top-2 similarity gap A; == Siz* maxjøj" Sij > 0, then (- concentrates on j and Var a (88))) shrinks.
Empirically, we small temporal change both ends and single interior region of rapid change which aligns with our theory.
D.2 INTER-STEP KL AND FISHER INFORMATION
Inter-step KL Peaks Maximal Fisher information. Recall that the Fisher information in 74 is defined by
[
0 0%
logpt )
2 1h =E =E(OO, (1isg) log 2,))
I(+)) = I(x):=
7/2 =E (sis (x 7
=Var,,
As in our implementation, we consider the KI. divergence between consecutive attention rows at steps At and t. For small At, a second-order expansion of the row KL divergence yields:
KL(P--a((- P) ())
2 _ (+ 7- Var [s] +o((Ar)))
(16)
=I(()))
where I(1) is the Fisher information of the one-parameter family pe(j i) exp(11s15) with respect to 7/--
Thus, the Instance Emergence Point t' that maximizes the row-wise temporal KL coincides (to second order) with the point of maximal Fisher information for that row. Averaging rows (as we do operationally when measuring the KI. over full maps) preserves the same interpretation at the
27

---

Published as a conference paper at ICLR 2026
map level. We also remark that any f-divergence has the same local quadratic form with the Fisher information Nielsen & Hadjeres (2019) by a difference of a factor of > O,
Dy(Pi-ae(- ()ppp i)) = cs (Ani)² Varp, [s] +o(( (T,(
explains why KI., JSD, and other f-divergences all produce the same IEP location (up to small higher-order effects), as highlighted in Tab. +
E COMPREHENSIVE ANALYSIS AND DISCUSSION
E.1 ABLATION STUDIES ON DESIGN CHOICES
Sensitivity to Instance-aware Text Prompts. All main results of TRACE are obtained without instance-aware prompts (e.g., A photo of two cats), using only null-text prompt. To assess the potential benefit of such supervision, we conduct an additional experiment in which, during training, exploit the instance annotations in ImageNet (Krizhevsky al., 2012) construct descriptive prompts of the form "A photo of [number of boxes] [class]", and at inference we mirror this setup on COCO (Lin et al. 2014) by building prompts from its instance annotations in the same way while keeping all other components of TRACE fixed. In this setting, Apmk increases slightly from 8.2 (null-text) to 8.3 (instance-count prompt), indicating that explicit instance-aware text information can provide small but marginal gain compared to the overall improvement brought by TRACE itself. Moreover, under the same conditions as Fig. 8)), the diffusion trajectories with null-text and instance-count prompts are almost indistinguishable, and we do not observe any systematic shift in the optimal timestep t". These observations support our claim that TRACE does not rely on instance-awat text supervision: the crucial instance-bounen boundary cues are already encoded in the early diffusion timesteps conditioned on the image alone, with instance-count prompts offering only minor additional refinement.
Influence of Decoder Capacity. A natural question is whether sharper instance boundaries actually require a heavier edge decoder. To isolate the effect of decoder capacity, re- place our lightweight edge decoder Go with two representative alternatives (Ronneberger et al. 2015 Cheng et al. 2022). while keeping the TRACE instance-edge cues (i.e., the accu- mulated multi-scale self-attention maps H((-) fixed. As shown in Tab. upgrading the edge decoder from our 0.1 MB 1-layer CNN to
Table 9: Effect of edge decoder capacity. We replace our lightweight CNN edge decoder in TRACE with heavier alternatives and evaluate performance of unsupervised instance segmenta- tion on COCO 2014.
Method
Params, of Dec.
Apmk
ProMerge
3.1
+TRACE (U-Net)
M4
8.2
+TRACE (Mask2Former)
258 MB
8.4
-TACE
0.1 MB
8.2
34 MB U-Net (Ronneberger et al. 2015) leaves Apmk unchanged (8.2 vs. 8.2), and even a 258 MB Mask2 decoder (Cheng al. 2022) yields only a marginal gain (8.4 APmk). By con- trast, adding TRACE on top of ProMerge (Li & Shin (2024] already increases Apmk from 3.1 to 8.2. These results indicate that sharp, instance-aware boundaries primarily come from the TRACE instance-edge cues themselves; once these cues are available, a minimal I-layer CNN decoder is sufficient, and substantially larger decoders bring negligible additional benefit.
Impact of Uncertainty Masking. In Algorithm [ and Eq. 10 pixels whose ABDiv score falls into the "uncertain" range (assigned value -) are ignored from the loss by a binary mask, so that only pixels labeled as 0 (non- (non-edge/backgroundd or 1 (edge/foregrrond) contribute to the supervision. This strategy follows common practice in -eakly-superviied segmentation (Ahn al. 2019 Jo et al. 2024a), where ambiguous regions are ignored to reduce label noise in pseudo semantic masks. We adopt the same idea for the pseudo instance-edge map F,, using ABDiv as a reliability cue.
Table T0 quantifies the impact of this design. When we threshold ABDiv only at and treat all pixels as confident (i.e., label E.j 1 if ABDiv, > i and E.j = 0 otherwise), the instance segmentation metrics improve over ProMerge (Li & Shin 2024) but remain limited, and the edge precision at ODS drops to 0.572. This large decrease compared to the proposed μ + a scheme (0.852) indicates substantial increase in colsitposiivve edge pixels. In contrast, masking uncertain pixels between μ - a and + a during training not only yields larger gains in Apmk and ART but also improves edge precision by about 1.5x while keeping ODS-recall within 1% of the p-only
28

---

Published as a conference paper at ICLR 2026
Table 10: Ablation on pseudo-labeling schemes for ABDiv (Sec.
3.3
ODS-Precision and ODS-
Recall denote the precision and recall at the optimal dataset scale (ODS) for instance edges, from which the ODS (F-measure) is computed.
Method
Decudo-Labeling for ABDiv
Ap
AR 100
ODS-Prec.
ODS-Rec.
ODS
ProMerge (Li Shin 2024
-TRACE
1
6.4
11.4
0.572
0.963
0.717
TRACE
+
8.2
13.1
0.852
0.950
0.889
500
SD1.5
40000
All Classes
SDL.5
600
SUXL
SDXL
5-L
30000
$03.554
400
ILUX.I-dev
F-UUX.Iddev
PieAme
20000
PixAn-
10000
0
Iostance G99
Instancce AUS
(199.899) 289)
APP7
(399 6999
(699,799)
19 1999
(1992209)
2209 (99)
(499 (99)
(599 699)
(699.799)
099
(AP9,
Semantic
VOC: Distribution of Optimal Person
(a)
Sentantic
(b) COCO: Distribution of Optimal
20000
4000
Car
SDXL
SDXL
15000
3000
FLUX.I-dev
FLUX.I-de
10000
PieAne
5000
1000
Instance
Instance
(99,199)
(199
1209 399)
3399 A997
(409 2997
(599 699)
(6999 (99)
(99, (99,199)
(199,299)
(299, (999
330 4993
(399,599)
(399 (99)
(699,799)
Top-2 COCO Classes: Distribution of Optimal +
Histograms of the optimal timestep 8* across datasets, diffusion backbones, and object categories. (a) VOC 2012 train set. (b) COCO 2014 train set. (c) Top-2 COCO classes (person and car). In all cases, the optimal timesteps t* for five diffusion models concentrate shared instance- aware region, indicating a consistent semantic-toinntance transition and stable IEP behavior across models, datasets, and categories.
Figure 14:
variant. This suggests that excluding uncertain regions effectively suppresses noisy edges without sacrificing recall, leading to a more favorable balance between false positives and false negatives in the distilled edge supervision.
E.2 IN-DEPTH CHARACTERIZATION OF IEP
Evaluation of Similarity Metrics. To assess whether our KI. criterion is a special case or broadly effective choice, we extend the comparison Tab. and evalu- ate additional similarity measures used as IEP scores, summarized Tab. TT Besides symmetric KL and standard regression losses (MSE/L2, MAE/LI), in- clude an entropy-based score and the Wasserstein-1 distance W.. For the entropy-based metric, the absolute entropy difference between two probabil- ity maps p and q (i.e., (p) H (q)l with (p) - pt pt). p.). For the Wasserstein distance, we measure a ID Wasserstein-1 1 distance on the flattened distribution. Empirically. KL achieves the highest instance segmentation performance (Apmk image). Jensen-Shanne (5,120 ms), offering no practical advantage over KL. Entropy difference and Wasserstein distance are clearly inferior in Apink (5.1 1 for entropy, 6.2 for W,) despite comparable or higher computational cost. We note that JSD can be written as JSD(p, g) = (L[ + ]L (L(gllm) with m = 1 {p q), and is equal to the mutual information between samples and binary index variable, this ablation also serves as a mutual cuutual-inforatioonn information-styy variant of our IEP score. Overall, these results support our choice of KI. the primary IEP metric: it provides the best trade-off between accuracy and effi-
Table 11: Similarity metrics for IEP.
Metric
Latency/ing
VOC
COCO
KL (Ours)
3,082 ms
9.4
8.2
JSD
5,120 ms
9.4
8.1
MSE (L.2)
1,232 ms
3.5
MAE (L.I)
924 ms
3.5
3.3
Entropy
2,070 ms
5.1
4.1
W
3,434 ms
6.2
5.0
9.4) with moderate latency (3,082 ms divergence (JSD) attains the same Apmk but is substantially slower
29

---

Published as a conference paper at ICLR 2026
Table 12: Performance of unsupervised instance segmentation on multiple benchmarks. To ensure a fair comparison with existing UIS models (Wang et al. 2023b|a a fine-tuned on the ImageNet train set (Krizhevsky et al.) 2012). we also fine-tune our diftusion model (SD3.5-L (Esser et al,]
2024) and edge generator on the ImageNet train set (Krizhevsky et al. 2012). This allows to
generate TRACE-based instance edges from the ImageNet-trained model and apply them directly
to other seven datasets (Everingham et al. 2010 Lin et al., [014] Cupta et al. 2019 Geiger et al.
2012
Shao et al. 2019; Kirillov et al. 2023) without further fine-tuning.
VOC
2012
COCO
2014
COCO
2017
IVIS
KITTI
Ohjects365
SA-IB
Average
Method
APm
ARqu
APmA
ARqu
APm
ARiu
APm
AR
APmA
AR
APm
AR
AP
AR
Ap
AR
TokenCul
6.1*
10.6*
2.7
4.6
2.0
4,4
0.9
1.8
0.3
1.5
1.1
2.1
1.0
0.3
2.0
3.6
MaskCut
5.8*
14.0*
3.0
6.7%
2.3
6.5
0.9
2.6
0.2
1.9
1.7
4.0
0.8
0.6
2.1
5.2
ProMerge
5.0*
13.9
3,1"
7.6
2.5
7,5"
1,1"
3,4"
0.2"
1.6*
2.2
6.1
1.2
0.8"
2.2
5.8
TRACE
9.4
18.2
8.2
13.1
7.8
11.2
2,5
4.7
1.2
2.6
4.3
9.5
2.0
1.6
5.1
8.7
Reproduced results using publicly accessible code for a fair comparison. Thee rest are the values reported in the publication.
ciency, while entropy- and Wasserstein-based alternatives underperform and the mutual-information variant (JSD) yields similar AP at higher latency.
Distributional Consistency of t². To further characterize the distributional patterns of IEP (Sec. 3.2) beyond Fig. b), we extend the analysis to different datasets and object categories, as shown in
Fig. 14 Figure 14 a) reproduces the VOC 2012 train-set distribution of optimal timesteps t" used in Fig. (b), while Fig. 14 b) shows the corresponding distribution on the much larger COCO 2014 train set (82,783 2,,78 images). Across all five diffusion backbones, the COCO histograms closely match those from VOC, with 1* consistently concentrating in the same instance-aware range, indicating that the semantic-to-instane
transition is stable across datasets and scales. In Fig. 14 c), we further restrict the analysis of IEP to images containing two most frequent classes in COCO and plot the per-class distributions of t*. Although there mild class-dependent shifts, the instance-awart timesteps for all four classes lie predominantly in the noise-like instance regime rather than the semantic regime, and they remain largely concentrated between 30-60% of the diffusion trajectory, consistent with the global patterns observed in Fig. b).
E.3 EXTENDED BENCHMARKING ON DIVERSE DOMAINS
Generalization to Diverse UIS Benchmarks. To rigorously evaluate the generalization capability of TRACE across varied domains, we extend our unsupervised instance segmentation (UIS) exper- iments to seven benchmarks: COCO 2014 (Lin et al.
2014), COCO 2017 (Caesar et al. 2018).
LVIS (Gupta et al., ) KITTI (Geiger et al. 2012). Objects365 (Shao al.] 2019), SA- 1B (Kir-
illov et a 2023), and VOC 2012 (Everingham et al. 2010). We compare TRACE (with SD3.5-L
backbone (Esser et al. 2024 against three representative UIS methods: TokenCut (Wang et al,) 2023b), MaskCut (Wang et al., 2023a). and ProMerge (Li & Shin 2024). For fair comparison, we strictly reproduce these baselines using their official public checkpoints and evaluation proto- cols (see Appendix F for details). Crucially, to align with the standard UIS protocol where models (Wang et al. 2023b a are typically fine-tuned on the ImageNet train set (Krizhevsky et al. 2012), also fine-tune our diffusion backbone and edge generator on ImageNet. This allows us gen- erate TRACE-basee instance edges using the ImageNet- trained model and apply them directly to downstream datasets without any target-specific adaptation. As shown in Tab. 12] TRACE con- sistently outperforms all baselines across all datasets, achieving an average AP p of 5.1, which corresponds to a 2.3x improvement over the strongest baseline (ProMerge). Notably, on the chal- lenging LVIS dataset (Gupta et al.) 2019), which features a long-tail distribution, TRACE more than doubles the performance (1.1 2.5 Apml), indicating superior handling of rare and diverse objects. Similarly, in the dense scenes of Objects365 (Shao et al. 2019). TRACE achieves a significant boost (1.7 + 4.3 APm*), proving its efficacy in separating heavily occluded instances where traditional feature clustering often fails. These results confirm that the instance-aware cues extracted from diffusion priors capture fundamental structural boundaries rather than -ataset-specifie semantics. Consequently, this demonstrates that the refined instance edges produced by TRACE are robust and generalizable, effectively bridging the gap between semantic grouping and instance separation even in shoo transfer scenarios.
30

---

Published as a conference paper at ICLR 2026
Enhancement of Open- Sieen-Vocauulary Segmentation Frameworks. While most of our experiments fo- cus on closed-set benchmarks, additionally eval- uate TRACE in open-vocabulary segmentation set- tings by attaching it to recent open-vocabulary mod- els. Specifically, we integrate TRACE with TTD (Jo) ctt al. 2024b) and Talk2DINO (Barsellotti et al.
Table 13: Performance of open-vocabulary segmentation. We attach TRACE to two open-vocabulary segmentation methods and evaluate on five benchmarks.
Method
VOC
Context
Stiff
ADE
City
TTD
61.1
37.4
23.7
17.0
27.9
TRACE
64.9
40.3
24.3
18.3
31.3
2025) and measure performance on five standard npen-vocabulary segmentation benchmarks: VOC 2012 (Everingham et al.] (2100), Pascal Context (Mottaghi et al., 2014), COCO-Stuff (Caesar et al. Cityscapes (Cordts et al. 2016). As shown in Tab. 13 refined instance boundaries from TRACE consistently improve open- -renvoobbulaay segmentation quality. On top of TTD (Jo et al. 2024bb) TRACE yields gains of 0.6-3.8 points across datasets (e.g.. from 61.1 to 64.9 on VOC and from 27.9 to 31.3 on Cityscapes), and on top of Talk2DINO (Barsellotti et al. 2025 it further improves performance by 1.6 points (e-g., from 65.8 to 68.4 on VOC and from 38.1 to 40.2 on Cityscapes).
TalK2DINO
65.8
42.4
30.2
22.5
38.1
TRACE
68.4
44.8
32.1
23.9
40.2
2018), ADE20K (Zhou et al. 2019), and
These results indicate that the instance-aware cues extracted by TRACE transfer beyond closed-set UIS and WPS (Tabs. D and 2 and provide consistent benefits for open-vocabulary segmentation, including real-world scenes, by sharpening object boundaries while preserving the underlying open- vocabulary recognition capability of the backbone models.
E.4 LIMITATIONS AND FUTURE DIRECTIONS
Challenges in Satellite Imagery (Tiny Instances). While TRACE substantially improves instance segmenta- tion on natural-image benchmarks (see Tabs. [ad 2 we observe a clear limitation on datasets dominated by very small objects (i.e., instances occupying only about 0.01 % of the image area). Table [4 reports results on two satel- lite benchmarks such as HRSID (Wei et al.) (020)) and iSAID (Waqas Zamir et al. 2019). evaluated on their of- ficial test sets using Apmaax When added on top of Mask (He et al.] 2017). TRACE
Table 14: Performance on satellite benchmarks. For a fair comparison, we evaluate instance segmentation perfor- mance HRSID and iSAID test sets
in terms of Apmack
Method
HRSID
iSAID
Mask R-CNN
65.4
25.6
+RACE
(-5.1)
20.2 (-5.4)
R-CNN
leads to degradation of 5.1 A HRSID (65.4 50.3) and 5.4 Apmask iSAID (25.6 - 20.2). We attribute this failure mode to the resolution loss inherent in latent diffusion models: before denoising, all images are encoded by a VAE into low-resolution latent grid (up 16x spatial downsampling), as shown in Fig. 4 which severely compresses tiny structures. As consequence, closely packed small objects tend to share blurred or merged boundaries in latent space, and the de- coded instance-ede cues from TRACH cannot reliably separate individual instances in high-density satellite scenes. Qualitatively (see Fig. [5ja)), we often observe multiple nearby targets fused into single instance mask. Addressing this limitation likely requires diffusion backbones with higher- resolution latents or hybrid schemes that combine TRACE with high-resolution, task-specific feature extractors for small-object regimes.
Applicability to Medical Imaging (Out-of- Distribution). TRACE is built on text-to-image diffusion models trained on natural images, which raises concerns about its behavior distribution domains such as histopathology images. In Tab. [5 we evaluate two cell instance segmenta- tion methods, SSA (Sahasrabudhe et al., 2020) and COIN (Jo et al.) 2025), on the MoNuSeg (Kumar et al. 2020) and TNBC (Naylor et al. 2019) test sets using PQ. Adding TRACE on top of these backbones consistently degrades performance: PQ
Table 15: Performance medical bench- marks. For a fair comparison, we evalu- ate panoptic segmentation performance on MoNuSeg/TNBC test sets in terms of PQ.
out-of-
Method
MoNuSeg
TNBC'
SSA
0.185
0.253
+ +TRACE
0.148 (-0.037)
0.209 (-0.044)
COIN
0.536
0.540
+ TRACE 0.439 (-0.097)
0.426 1-0.114)
drops from 0. to O. 148 on MoNuSeg and from 0.253 to 0.209 on TNBC for SSA, and from 0.536 to 0.439 (MoNuSeg) and 0.540 to 0.426 (TNBC) for COIN. We hypothesize that this limitation stems from a domain mismatch between vatural-image diffusion priors and medical imagery: the diffusion backbones representations tend to emphasize color and texture patterns that do not align with cell boundaries. Consequently, incomplete instance edges predicted by TRACE often undersegment cells, as seen
trained on photographs, not histopathology slides, and their latent
31

---

Published as a conference paper at ICLR 2026
Input Image
Ground Truth Mask
TRACE (Ours)
Input Image
Ground: Truth Mask
TRACE Ours)
Original Resolution
dx Zoom
(a) Satellie Benchmarks (eg.. HESID)
Medical Benchmanks (eg., MoNuSog)
Figure 15: Failure cases on satellite and medical benchmarks. Two examples illustrate key limi-
tations of TRACE on tiny objects and cuttof-distribuutin domains.
in Fig. b)). These results indicate that directly applying TRACE to highly out-of-distribution medical images can be harmful, and suggest that future extensions should either adapt the diffusion prior to the medical domain (e.g., via domain specific diffusion training) or combine TRACE with medical-speciffe feature extractors and supervision.
F DATASETS AND METRICS FOR EVALUATION
Datasets for Unsupervised Instance Segmentation. We evaluate our TRACE seven bench- marks including COCO2014 (Lin et al.
2014) and COCO2017 (Caesar et al. 2018). LVIS (Gupta al. 2019). KITTI GGiggrr et 22122). Objects365 (Shao al. [2199), and SA-1B (Kirillov al. 2023 COC02014 (Lin et al. (2044) and COCO2017 (Caesar et al. 2018) are standard datasets for object detection and segmentation. COCO2014 has 80 classes, 83K training images and 41K vali- dation images. COC02017 is composed of 118K and 5K images for training and validation splits respectively. For results on COC02014 and COCO2017, we use the images in the validation split. instance masks, making it more challenging for segmentation. We test
LVIS has densely-annotate our performance on the validation set containing 245K instances on 20K images. For KITTI and Objects365, we evaluate on 7K images and subset of 44K images in the val split, respectively. Lastly, for SA-1B, we assess on a subset of 1 1K images, which come with 100+ annotations per image average.
Datasets for Weakly-supervised Panoptic Segmentation. Pascal VOC (Everingham et al. 2010) consists of 20 "thing" and "stuff" categories. It contains 1 IKK images for training and 1.4K images for validation. COCO2017 (Caesar et al. 2018) has 80 "thing" and 53 "stuff" categories, which is challenging benchmark for PS. We validate TRACE on the 5K images in the validation dataset.
Datasets for Instance Edge Evaluation. To directly evaluate instance-edge quality, we construct an instance-aware edge benchmark from the COCO 2014 validation set (Lin al.) 2014)). For each of the 41K validation images, we start from the ground-truth panoptic segmentation masks and extract instance boundaries by labeling pixels that lie on the borders between distinct panoptic segments as positive edges, and all remaining pixels as background, as shown in Fig. [1] The resulting bi- nary instance-edge maps are used as ground truth for all our edge-quality experiments (ODS/OIS and connectivity metrics (Shit et al. 202T)) real images, enabling a direct comparison between
TRACE and conventional edge detectors Xie & Tu (0155); Su et al. (2023) under an instance-awan aware boundary supervision rather than generie low-level contour annotations.
Evaluation Metrics for Edge Quality. To evaluate the quality of instance boundaries, we follow the standard evaluation protocols widely adopted in general edge detection methods such as HED (Xie & Tu 2015) and PiDiNet (Su et al. 2023). We report the standard F1-score (or F-measure, defined PrPlectstonr Recall
Precision Recall using two standard metrics: Optimal Dataset Scale (ODS) and Optimal Image Scale (OIS). ODS computes the Fi-score using a global threshold that is optimal across the entire test dataset, providing a measure of generalizability. OIS selects the optimal threshold for each individual image to maximize the F1-score, reflecting the best possible performance per image.
32

---

Published as a conference paper at ICLR 2026
Furthermore, since pixel-wise metrics often fail to capture the topological correctness of thin bound- ary structures, we additionally employ the clDice (centerline Dice) metric (Shit et al. 2021). Unlike the standard Dice coefficient, elDice calculates the overlap between the skeletons (centerlines) of the predicted edges and the ground truth. This allows for a more robust evaluation of topological connectivity and structural preservation of the instance edges.
Evaluation Metries for Instance Segmentation. We evaluate the performance comparison of UIS methods with and without TRACE based on average precision (AP) and average recall (AR) on Pascal VOC and MS COCO dataset. Precision and Recall are defined as
TP
TP
Precision = TP + FP
Recall +P FN
(17)
where TP, FN, FP are short for True Positive, False Negative, and False Positive, respectively. Intu- itively, a high precision means the method has a low rate of making false positives, but does not imply that all the positives were found. A high recall means the method has low rate of making false negatives.
AP measures the precision across different recall levels. A high AP means the method finds most objects(recal)) while minimizing false positives (precision). On COCO, AP is calculated over dif- ferent IoU thresholds and object sizes, then averaged. For Pascal VOC, mAP used, averaging AP across classes at a single loU threshold (0.5). COCO uses mAP@ I:U=005:00.55 meaning it averages AP across ten IoU thresholds(0.50, 0.55, .... 0.95).
AP precision(r)dr
(18)
m.A.ccco mAPo.0 -mAP.55 0.53 +.... m.AP0.95
(19)
10
Average Recall (AR) is calculated as the area under the recall-threshold curve. Like AP, it is aver- aged over multiple loU thresholds. AR reflects how well the method recalls true objects rather than balancing precision and recall. In COCO, AR is reported as AR @ 100, AR @ 100 (small), AR @ 100 (medium), and AR 100 (large), corresponding to the average recall with up to 100 detections per image across different object sizes.
Evaluation Metrics for Panoptic Segmentation. We report evaluation results on the standard evaluation metrics of panoptic segmentation task, including panoptic quality (PQ), segmentation quality (SQ) and recognition quality (RQ). PQ is defined for matched segments (IoU >> 0.5 between predicted and ground truth masks) and combines SQ and RQ:
Σ E(p.tDET IoU gt)
PQ = [TP| FP + F.5]
(20)
Segmentation quality (SQ) measures the accuracy of segment boundaries, focusing only on segments that were correctly identified and ignoring false positives and false negatives. SQ is defined as:
SQ E(ngheTT loU(p,g() gt)
(21)
IPP]
Recognition quality (RQ) measures the model's ability to correctly classify instances, accounting for both precision and recall for detected instances. RQ combines recall (finding all objects) with precision (only detecting true objects), focusing on instance recognition rather than segmentation quality. RQ is defined as:
TP
RQ TP + FP FN.
(22)
Table 16: Evaluation Segmentation Metrics on Pascal VOC and COCO datasets.
Metric
Intuitive Meaning
AP
Balance between precision and recall
AR
Ability to recall true objects
PQ
Combined segmentation quality and recognition quality
SQ
Accuracy of segmentation shapes
RQ
Correct classification of instances
33

---

Published as a conference paper at ICLR 2026
Image
Ground Truth
Point2Mask
Ours+DHR
Figure 16: Qualitative results in WPS with ours (TRACE+DHR (Jo et al. 2024al) and point- supervised Point2Mask (Li et al.) 2023b) on the VOC2012 (Everingham et 2010) validation set. (Ours) We trained a Mask2Former using ResNet-50 backbone with pseudo panoptic masks generated from TRACE+DHR, The samples in this figure are the outputs from Fosk2Frrmer trained with our masks.
34

---

Published as a conference paper at ICLR 2026
Ground Truth
TRACE (Ours)
ProMerge
Ours+ProMergg
MaskCut
Ourst-MaskCut
Figure 17: Qualitative results UIS with ours (TRACE+ProMerge, TRACE+MaskCutt and exist-
ing methods (ProMerge (Li & Shin 2024), MaskCut (Wang et al. 2023a)) the COC02014 (Lin et al. 2014) validation set.
35

---

Published as a conference paper at ICLR 2026
Ground Truth
TRACE (Ours)
ProMerge
OurstProMerge
MaskCut
OurstMaskCut
Figure 18: More Qualitative results in UIS with ours (TRACE+ProMerge. TRACE+MaskCut) and existing methods MProMerge (Li & Shin 2024), MaskCut (Wang et al. 2023a)) on the COC02014 (Lin et al. 2014) validation set.
36

---

Published as a conference paper at ICLR 2026
Ground Truthh
TRACE (Ours)
ProMerge
OurstProMerge
MaskCut
OurstMaskCut
Figure 19: More Qualitative results in UIS with ours (TRACE+PPoMMere. TRACE-MasKCud and existing methods (ProMerge (Li & Shin 2024), MaskCut (Wang et al. 2023a)) on the COC02014 (Ln et 2014) validation set.
37

---

Published as a conference paper at ICLR 2026
Ground Truth
TRACE (Ours)
ProMenge
OurseProMerge
MaskCut
OurktMa
La Z
O
Figure 20: Qualitative results in UIS with ours (TRACE+ProMerge, TRACE+MaskCutt and ex-
isting methods (ProMerge (Li & Shin 2024), MaskCut (Wang al. 2023a on the COC02017 CCasarr et al. 2018) validation set.
38

---

Published as a conference paper at ICLR 2026
Ground Truth
TRACE (Ours)
ProMerge
Ours-ProMergge
MaskCut
Ourkt-aaskuut
Figure 21: Qualitative results in UIS with ours PTRACE++ProMerg,, TRACF*MaskCur and ex- isting methods (ProMerge (Li & Shin (024], MaskCut (Wang et al. 2023a]) on the VOC2012 (Everingham et al. 2010) validation set.
39


Reviews:
[{"review_id": "FNQf9xQQdL", "reviewer": "Reviewer_inKu", "summary": "This paper proposes TRACE (TRAnsforming diffusion Cues to instance Edges), a framework for decoding instance boundaries from the self-attention of pre-trained text-to-image diffusion models. It aims to solve the problem of difficult instance separation in unsupervised and weakly-supervised segmentation.", "questions": "1. Although the authors claim that the IEP timestep distribution is \"model-agnostic,\" the results in Table 5 indicate that TRACE's final performance is strongly correlated with the chosen diffusion model backbone. For example, SD3.5-L (8.1B parameters) achieves an $AP^mk$ of 8.2, while SD1.5 (0.8B parameters) only reaches 6.8. The authors should discuss in more detail the method's generalizability across model scales and architectures, and whether this dependency might limit its application in resource-constrained scenarios.\n2. In Section 3.4, it is ambiguous when describing the one-step self-distillation. The authors do not explicitly state how the \"uncertain\" pixels (value -1) are handled (or \"excluded\") within this loss function. This lack of critical implementation detail could hinder reproducibility. It does not sufficiently explain the specific impact of this threshold choice on the results (edge connectivity, false positives, false negatives), nor why $\\mu+\\sigma$ is robust across various scenarios. This step directly impacts the quality of the distillation supervision.\n3. The paper compares TRACE with alternatives like Canny/HED/PiDiNet/Depth-based methods in tables/figures, showing TRACE's advantages. However, it lacks: a) Edge quality metrics (e.g., Precision/Recall/F1, ODS/OIS) on standard edge detection benchmarks (like BSDS, or COCO edge annotations if available); and b) Quantitative connectivity/topological metrics for the output edges. Currently, the quality of the edges is primarily demonstrated indirectly through downstream segmentation metrics.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1.The idea that instance-level structural information (not just semantic information) is hidden in the self-attention (rather than cross-attention) of pre-trained diffusion models is interesting. \n2. As a plug-and-play module, TRACE shows consistent and significant performance improvements when integrated into various existing segmentation frameworks.", "weaknesses": "1. It seems that the proposed method depends on specific diffusion model backbones. The discussion on detailed generalizability across model scales and architectures is required. Whether this dependency might limit its application in resource-constrained scenarios.\n2. The paper compares TRACE with alternatives like Canny/HED/PiDiNet/Depth-based methods in tables/figures in Fig. 10, but it lacks the direct edge quality evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JpphBX32IS", "reviewer": "Reviewer_rCqE", "summary": "The authors observe that self-attention in diffusion models encodes instance-level structure during denoising. Based on this observation, the authors propose to decode instance boundaries directly from pre-trained text-to-image diffusion models. This is achieved by identifying the Instance Emergence Point, extracting boundaries through Attention Boundary Divergence, and distilling them into a lightweight one-step edge decoder. Experimental results on unsupervised instance segmentation and weakly-supervised panoptic segmentation demonstrate the effectiveness of the proposed method.", "questions": "See the questions mentioned above. Given the current status of the paper, I am leaning towards borderline accept and hope the authors could address my concerns during the rebuttal.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tThe proposed method is well motivated. The authors start from the interesting observation that instance-level structure emerges from self-attention in diffusion models, which motivates the authors to decode instance boundaries from diffusion models to enable annotation-free instance and panoptic segmentation.\n-\tThe paper is generally well-written and easy to follow.\n-\tThe experiments are extensive and the results seem promising.", "weaknesses": "-\tSome related works are missing. For example, [1, 2] are diffusion-based instance/semantic segmentation methods that also leverage the attention maps directly from diffusion models without relying on any label supervision. The authors should discuss these works as well.\n-\tThe experimental settings mainly focus on the closed set. Given the open-set nature of diffusion models, I am curious about the effectiveness of the proposed method for open-vocabulary segmentation.\n-\tHow to measure the quality of instance boundaries? Currently, it somewhat manifests through the downstream segmentation performances. Would it be possible to have a metric to directly evaluate instance boundaries before downstream training?\n-\tWhat are the failure cases of the proposed method? Adding more analysis on failure cases will add more insights to the paper.\n\n**References:**\n\n[1] MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation. In IJCV, 2024.\n\n[2] EmerDiff: Emerging Pixel-level Semantic Knowledge in Diffusion Models. In ICLR, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "otRTmvQQGk", "reviewer": "Reviewer_KrZ9", "summary": "This paper proposes **TRACE** (TRAnsforming diffusion Cues to instance Edges), a novel framework revealing that diffusion models inherently encode *instance-level boundary information* within their self-attention maps.  \nThe authors identify the Instance Emergence Point (IEP)—the timestep where object structures first appear—and introduce Attention Boundary Divergence (ABDiv) to extract instance edges without any annotation.  \nThrough one-step edge distillation, TRACE achieves real-time inference and demonstrates strong results on unsupervised and weakly-supervised segmentation tasks.", "questions": "1. How consistent is the identified IEP across diffusion timesteps, model scales, or datasets?  For example, does the same timestep correspond to “edge emergence” across all classes and diffusion models?  \n2. Could the KL divergence criterion for IEP detection be replaced or augmented by other measures (e.g., entropy, mutual information)?  \n3. How sensitive is the distilled edge decoder to the choice of teacher diffusion backbone? Could it generalize across unseen models or domains (e.g., medical, satellite images)?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "1. **Novel Insight** – The discovery that diffusion models internally encode instance boundaries is original and insightful.  \n2. **Elegant Design** – The approach leverages self-attention statistics (KL divergence) without requiring extra supervision or training labels.  \n3. **Strong Empirical Results** – TRACE significantly outperforms state-of-the-art methods like MaskCut and ProMerge on COCO and VOC datasets.  \n4. **Interpretability** – The visualization of attention evolution (IEP phenomenon) provides an interpretable window into diffusion model internals.  \n5. **Efficiency** – The one-step distillation achieves ~81× inference acceleration, demonstrating practical usability.", "weaknesses": "1. **Limited Theoretical Explanation** – While IEP is clearly defined using KL divergence peaks, the underlying cause of such emergence remains described qualitatively. A lightweight theoretical or statistical discussion (e.g., how noise levels influence attention entropy) could enhance the argument.  \n\n2. **Negative Analysis** – The paper focuses on successful examples; adding a few failure visualizations or robustness analyses (e.g., under cluttered scenes or overlapping objects) would provide a more balanced evaluation.  \n\n3. **Fine-Grained Analysis of IEP** – The phenomenon of IEP is intriguing, but its quantitative behavior (e.g., variance across images, layers, or diffusion steps) is not deeply examined. A more detailed statistical characterization—such as distributional patterns or consistency metrics—could enrich understanding and further substantiate the discovery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IFlepXwcXi", "reviewer": "Reviewer_JkD6", "summary": "TRACE shows that text-to-image diffusion models already encode sharp instance boundaries in their self-attention maps at a special denoising step (the Instance Emergence Point, IEP). By harvesting these “secret” edges with a simple, KL-based Attention Boundary Divergence (ABDiv) and distilling them into a one-step edge decoder, TRACE delivers: +5.1 AP on unsupervised COCO instance segmentation, +1.7 PQ on tag-supervised panoptic segmentation (no boxes/masks), 81× faster than per-image diffusion inversion\nAll without a single manual instance label.", "questions": "see weakness", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. First to treat diffusion self-attention as a free instance-edge oracle.\n2. Introduces IEP + ABDiv, two lightweight, non-parametric cues that any diffusion backbone yields.\n3. Exhaustive ablations (Tab. 3, 4, 5) and 5 backbones; KL beats MSE/MAE by > 5 AP.\n4. Clear pipeline figure (Fig. 4); every symbol defined; 81× speed-up quantified.\n5. Annotation-free edge seeds outperform point-supervised panoptic models (Tab. 2).", "weaknesses": "1. Only one 1-layer CNN decoder is tried; U-Net, light transformer, or multi-scale may sharper edges.\n2. Paper shows success cases (Fig. 2, 9); no figure of merged/fragmented predictions when IEP fails.\n3. TRACE claims “no prompt needed”, but null-text vs. descriptive prompt may shift t* or edge quality.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)