Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
Authors: Danfeng Li
Hui Zhang
Sheng Wang
Jiacheng Li
Zuxuan Wu
Venue: NeurIPS 2025 poster ()
Keywords: controllable image generation
text to image generation
Primary area: deep_learning

Abstract: Despite recent advances in diffusion models, top-tier text-to-image (T2I) models still struggle to achieve precise spatial layout control, *i.e.* accurately generating entities with specified attributes and locations. Segmentation-mask-to-image (S2I) generation has emerged as a promising solution by incorporating pixel-level spatial guidance and regional text prompts. However, existing S2I methods fail to simultaneously ensure semantic consistency and shape consistency.
To address these challenges, we propose Seg2Any, a novel S2I framework built upon advanced multimodal diffusion transformers (*e.g.* FLUX). First, to achieve both semantic and shape consistency, we decouple segmentation mask conditions into regional semantic and high-frequency shape components. The regional semantic condition is introduced by a Semantic Alignment Attention Mask, ensuring that generated entities adhere to their assigned text prompts. The high-frequency shape condition, representing entity boundaries, is encoded as an Entity Contour Map and then introduced as an additional modality via multi-modal attention to guide image spatial structure. Second, to prevent attribute leakage across entities in multi-entity scenarios, we introduce an Attribute Isolation Attention Mask mechanism, which constrains each entity’s image tokens to attend exclusively to themselves during image self-attention.
To support open-set S2I generation, we construct SACap-1M, a large-scale dataset containing 1 million images with 5.9 million segmented entities and detailed regional captions, along with a SACap-Eval benchmark for comprehensive S2I evaluation.
Extensive experiments demonstrate that Seg2Any achieves state-of-the-art performance on both open-set and closed-set S2I benchmarks, particularly in fine-grained spatial and attribute control of entities.

Seg2Any: Open-set Segmentation-MMaskttoommag
Generation with Precise Shape and Semantic Control
Danfeng Li Hui Zhang4.2- Sheng Wang' Jiacheng Li Zuxuan Wu
"Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Shanghai Collaborative Innovation Center of Intelligent Visual Computing
HiThinkk Research
https://seg2an: gi cithub.15 10
IT Louning Tower of Pisa. 2) Grart wall. 3) Eiftel Tower. 4J Tij Makal.
I)'S in deep Nue. 2) orange.
Flccalgelieeo lspestty will
Stnweberry piush. Cablage
5) z in nurcunise. 4) yellow. 5)
faaree 21 Woodes dalknel fealuring
stll Gerlic 2D style Banane
AA' pink. in deep pupple Py'y" in lnverder.
drawers woves
brigait slyle, Scallion In. cartac: style
buskel oontainig
Guzumber bwllion styl
dressed with Seage bemboo Nindd. etc
4
tr
ec 2Amy
Il Deep Blue cae puzt. 2) Velucle bendight 3) Red fur paint. While car yeiat, 5) While car zeer minoe Red text "PORSCLL"" 7) Green number -92". 3) Yallow lost "PRK"" 9) Black seas wing.
I) Red and Black spademain mask. Deep green
hooded jariset with red drawstrikg. 3) Linkt gry inner
cicches 4) Black spont shorts. Red light flling ponf 5) Red and whitl AJ sneekers.
92
Figure We propose Seg2Any, a novel segmentation mask mask-to-image to-inage generation approach that achieves strong shape consistency and fine-grained attribute control (e.g. color, style, and text).
Abstract
Despite recent advances in diffusion models, top-tier text-to-image (T2I) models still struggle to achieve precise spatial layout control, i.e. accurately generating entities with specified attributes and locations. commentttioonmmss-toor generation has emerged as a promising solution by incorporating pixel-level spa- tial guidance and regional text prompts. However, existing S2I methods fail simultaneously ensure semantic consistency and shape consistency. To address these challenges, we propose Seg2Any, novel S2I framework built upon advanced multimodal diffusion transformers (e.g. FLUX). First, to achieve both semantic
(S2I)
"quual contribution. Corresponding author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

and shape consistency, we decouple segmentation mask conditions into regional semantic and high- tigh-freuuenyy shape components. The regional semantic condition is introduced by a Semantic Alignment Attention Mask, ensuring that generated entities adhere their assigned text prompts. The high-frequency shape condition, representing entity boundaries, is encoded as an Entity Contour Map and then introduced as an additional modality via multi-modal attention to guide image spatial structure. Second, to prevent attribute leakage across entities in multi- entity scenarios, we introduce an Attribute Isolation Attention Mask mechanism, which constrains each entity's 's image tokens to attend exclusively to themselves during image self-attention. To support open-set S2I generation, we construct AM, large- scale dataset containing 1 million images with 5.9 million segmented entities and detailed regional captions, along with a SACap-Eval bench- mark for comprehensive S2I evaluation. Extensive experiments demonstrate that Seg2Any achieves state-of-the-art performance on both open-set and closed-set S21 benchmarks, particularly in fine-grained spatial and attribute control of entities.
Introduction
Text-to-image (T2I) generation [34, 4, 9] has been widely adopted in various applications due to its powerful generative capabilities. However, T2I models struggle achieve precise layout control (i.e. precisely generate entities in specified attributes and positions) solely through text prompts.
Layout-to-image generation has been proposed and designed to generate images based specified layout conditions, including spatial locations and descriptions of entities. These layout conditions
come in various forms, such as bounding boxes [22, 60, 44, 19, 54, 25], depth maps [62, 63, 61], segmentation masks [47, 24, 53]. etc. This paper focuses on
(S2I) generation, where segmentation masks dictate the spatial locations of entities, and text descriptions specify their semantic content, thereby enabling the most fine-graines control over the images.
Existing S2I methods be mainly divided into two categories: 1) Methods that integrate seg- mentation masks as additional conditional inputs, such as ControlNet [56], ControlNet++ [20] and T2I-Adapter [26]. These methods often fail to align regional textual descriptions with their respective regions, resulting in semantic inconsistency in the generated images (see Figure 2 (a)). II) Methods based the masked attention mechanism, which restricts each text embedding attend solely to the respective image embeddings (e.g. FreestyleNet [47], PLACE [24] and EliGen [53]). Although these methods achieve semantic alignment, they fall short in precise shape preservation, as shown in Figure 2 (b) and (c). We attribute this shape inconsistency to the loss of spatial information when segmentation masks are compressed into the latent space. Notably, compared to the UNet [34] archi- tecture with 8x downsampling, recent ad advanced DiT [28] architectures employ more aggressive 16x downsampling, which further amplifies the loss of spatial information.
To address these challenges, we propose Seg2Any, a novel S2I framework built upon advanced multimodal diffusion transformers (e.g. FLUX [18]). Seg2Any mainly relies on two key innovations: I) Semantic-Shape Decoupled Layout Conditions Injection. We decouple segmentation mask conditions into two components: the shape condition (high-frequency shape information) and the regional semantic condition (low-frequency semantic information). For the injection of regional semantic conditions, we employ Semantic Alignment Attention Mask. This mechanism tightly binds each entity's 's image tokens to its corresponding text prompts, ensuring that the generated image is semantically consistent with the input descriptions at the entity level. For the injection of shape condition, we propose Sparse Shape Feature Adaptation to integrate key spatial structure into the model efficiently. Conventional approaches [56, 20] assign fixed colors distinct categories in the semantic segmentation maps, which is only suitable for closed-set S2I generation and cannot be generalized to open-set scenarios. In contrast, we introduce an Entity Contour Map as our category- agnostic shape representation, which consists of entity contours extracted from segmentation masks. This representation is inherently sparse, preserving essential shape details compact and efficient manner. Following OminiControl [37], we integrate the condition tokens with text and noisy image tokens into a unified sequence, allowing them interact directly through multi-modal attention [9]. As shown in Figure (d), our approach achieves both semantic and shape consistency simultancously.

---

Semantic label
Real image
(a] ControlNet++
(b) FreestyleNet
(c) PLACE
(d) Ours
Figure 2:
Comparison in terms of shape and semantic consistency. Semantic inconsistency is annotated by blue boxes, while shape inconsistency highlighted with red boxes, which reveal inconsistency in the number of vertical bars on the railings. In contrast, our approach achieves both shape and semantic consistency.
III Attribute Isolation via Image Sell-Attention Mask. Multi-instanee generation often suffers from attribute leakage, where the visual attributes of entity may affect others. To mitigate this problem, we introduce an Attribute Isolation Attention Mask strategy that prevents cross-entity information leakage by ensuring that image tokens associated with each entity are isolated from those of others.
Current dense caption datasets [33, 51, 31, 8, 50] are often limited by their closed-set vocabularies and coarse-grained descriptions, which constrain their effectiveness in training S2I models. Recent advances in open-source vision language models (VLMs), such as Qwen2-VL-725 [43], have significantly reduced the performance gap with close-source VLMs like GPT-4V [1], making feasible to create large-scale and richly annotated datasets. Leveraging the capabilities of Qwen2-VL. VL- 72B, we construct Segment Anything with Captions Million (SACap- IM), large-scale dataset derived from the diverse and high-resolution SA-1B dataset [16]. SACap- IM contains million image-text pairs and 5.9 million segmented entities, each comprised of a segmentation mask and detailed regional caption, with captions averaging 58.6 words per image and 14.1 words per entity. We further present the SACap-Eval, a benchmark for assessing the quality of open-set S2I generation.
Through comprehensive evaluations on both open-set (SACap-Eval) and closed-set (COCO-Stuff, ADE20K) benchmarks, Seg2Any consistently outperforms prior SOTA models, particularly in fine- grained spatial and attribute control of entities.
To summarize, contributions as follows:
1. We propose Seg2Any, a framework that enables precise control over shape and semantics while preventing attribute leakage in open-set S2I generation.
2. We construct SACap- IM, large-scale open-set dataset with 1M images and 5.9M regional annotations, along with SACap-Eval, an open-set benchmark for evaluating S2I generation.
Seg2Any achieves state-of-the-ar performance on both open-set (SACap-Eval) and closed-set (COCO-Stuff, ADE20K) benchmarks.
Related Work
2.1 Text-to-Image Generation
Text-to-image (T2I) generation [34, 4, 9, 42, 38] has undergone significant advancement in recent years. Motivated by advances in large-scale transformer architectures, the Diffusion Transformer (DiT) [28] was introduced. Building upon this foundation, recent models like SD3 [9] and FLUX [18] further propose the Multimodal Diffusion Transformer (MM-DiT), which treats text an independent modality and incorporates flow matching objectives, achieving state-of-the-ard results. To integrate additional condition images (e.g., canny maps, depth maps, subject references) into MM-DiT, OminiControl [37] introduces a novel controllable framework. It concatenates condition image, text, and noisy image tokens, and then employs speciscci LoRA [13] modules handle various conditions within unified pipeline while maintaining minimal trainable parameters.

---

2.2 Layout-to-Image Generation
Layout-to-Imas -mmag (L2I) generation is a task that synthesizes images guided by spatial layout conditions and entity-level textual descriptions. Existing L.21 methods can be categorized according to the type of layout condition they employ. These include: Bounding box-based approaches [22, 60, 44, 19, 54, 55, 25, 11, 10, 36, 7. 58, 48] typically use rectangular spatial constraints guide the image generation process, offering a coarse-grained layout control. Depth map-based approaches [62. 63, 61] utilize depth maps to achieve une-ggrined spatial control similar to segmentation masks, For instance, 3DIS [62] divides muuli-instance generation into two stages: it first generates a depth map via text-to- depth model, then uses a pre-trained
model to synthesize images with
attribute control. DreamRenderer [61], a training- free approach based on pre-trained controllable T2I models (FLUX.I-Depp)
[18] and FLUX. I-Canny [18]), identifies that middle layers in the FLUX model are responsible for instance-level rendering while shallow and deep layers capture global context. Consequently, it applies a hard image self-attention mask only to the middle layers to prevent attribute leakage. However, as training-frre free approach, directly applying the attention mask severely impairs overall visual harmony.
mask-based approaches 24,,24 53, 27, 15, 2, 52] focus on pixel-level layout control. For example, FreestyleNet [47] uses binary attention weights in the cross-attention module that assigns a value 1 to allow text tokens to bind to corresponding image regions and 0 to prevent attention from unrelated areas. However, this approach requires downsampling the segmentation masks to align with the lower-resolution latent features, thereby sacrificing spatial detail. To alleviate the above issue, PLACE [24] introduces a layout control map that softens the attention weights. Instead of hard binary assignments in FreestyleNet, it calculates the area proportions of different entities within the receptive field of every image token, yielding soft attention weights. Yet, this approach struggles to mitigate spatial detail loss in MM-DiT architectures, where 16x downsampling (compared to 8x in U-Net) leads to more severe degradation of spatial information. The work most similar to ours is EliGen [53], which is also built on FLUX. Unlike approach, it is trained on datasets with bbox annotations and only supports loose position control through scribble-styl masks, whereas approach enables both strict and loose mask position control. Additionally, it relies solely on a masked attention mechanism without injecting explicit spatial information.
Segmentation
3 Methodology
3.1 Problem Definition
We define the instruction y for a segmentation -aann--aakk-oaaa tomim model as a composition of global text prompt and N entity-level text prompts with corresponding binary masks:
y = [Po: (p1. m1). .....
: (p., mz), ..... (pN,my/ i E [I,
(I)
where po denotes the global textual description, while each p: represents the entity-specific textual prompt with its corresponding binary segmentation mask m; for the i-th entity.
3.2 Semantic-Shape Decoupled Layout Conditions Injection
As shown in Figure 3, we decouple segmentation mask-based layout conditions into complementary semantic and shape components. For semantic information, we employ a Semantic Alignment Attention Mask (Section 3.2.1) mechanism that binds text prompts to their corresponding image regions. For shape information, we adopt the Sparse Shape Feature Adaptation (Section 3.2.2) to efficiently encode and integrate spatial layout conditions into the model. 3.2.1 Semantic Alignment Attention Mask
To ensure semantic alignment between textual and visual modalities, we introduce Semantic A Alignment Attention Mask, denoted as Msem-align- which governs the interactions between textual and visual tokens. We denote the text token indices of the global caption as To. and those of the i-th entity's regional caption as T, (i = 1,...,N), Likewise, To and I, correspond to the image token indices of the background and the i-th entity, respectively. The Moss-aliga is then defined as follows:
1, if E Tr, k E Tr (text-text)
1, T & I
(text-image)
M Mscm-atign [g. k] =
1, if TUT
(image-text)
i E [0, NJ,
(2)
1. k UN
(image-image)
0, otherwise

---

Image
Entity Contour Map
Segmentation Masks
Seg2Any
Globel Caption
Regional Captions
extract
contours
A yellow cat near red ball
yellow cat
A red ball
Image Tokens
Condition Tokens
(b) Attention Masks
T5 Text Encoder
VAE
Shallow Deep Layers
Text Tokens
drop
Linear
Linear
Linear
QKV Concat
Mil-Attention
Middle Layers
MLP
MLP
MLP
Text
Image
Condition
Semantic Alignment
Text
Image LoRA
2 7
Foreground Image Token
Background Image Token
LoRA
Attention Mask
Candition
Regional Text Token
Global Text Token
Attribute Isolation
LoRA
Dropped Condition Token
Condition Token
Attention Mask
Figure 3: (a) An overview of the Seg2Any framework. Segmentation masks are transformed into Entity Contour Map, then encoded condition tokens via frozen VAE. Negligible tokens are filtered out for efficiency. The resulting text, image, and condition tokens are concatenated into unified sequence for MM-Attention. Our framework applies LoRA to all branches, achieving S2I generation with minimal extra parameters. (b) Attention Masks in MM-Attention, including Semantic Alignment Attention Mask (Section 3) and Attribute Isolation Attention Mask (Section 3.3).
where and k are the query and key indices, respectively. As illustrated in Figure 3 (b). this attention mechanism guarantees that each generated entity adheres closely to its text prompt. Meanwhile, all image tokens attend to each other ensure globally coherent visual synthesis. The global caption tokens To are allowed attend to all image tokens, providing contextual global guidance.
3.2.2 Sparse Shape Feature Adaptation
Condition Image Representation. In open- set S2I generation, the common practice of using fixed, class-specific colors to represent semantic segmentation maps is inherently limited. To address this limitation, we use an Entity Contour Map to effectively encode shape information.
Starting with a set of binary masks [m for N distinct entities where each m, E (0, Hx we extract the contour of each mask as Contour(m;) E {0, 1] These contours are then merged into single binary map:
Cgray (z, v) max Contour(m:)(a (z) 1S15N
The resulting grayscale map Cgray E H& is then further converted to an RGB image by duplicating the gray channel across all three channels, resulting in our Entity Contour Map C E | x x3
This sparse shape representation offers advantages in framework, semantic information is already integrated through the Semantic Alignment Attention Mask mechanism (Section 3.2.1). This approach eliminates the need for dense, per-pixel semantic labels to indicate region occupancy. Due to its efficiency and sparsity, we adopt this representation as the shape encoding method our work.
Minimal Condition Image Control. Inspired by OminiControl [37], we treat the Entity Contour Map-u type of image-basee based condition-at as independent modality and leverage LoRA [13] to

---

minimize training overhead. As shown in Figure 3 (a), the Entity Contour Map is encoded into condition image tokens by a frozen VAE encoder, concatenated with text and noisy image tokens to form the joint input sequence, which directly participates in multi-modal attention. Furthermore, the condition tokens share the same 2D position indices with the noisy image tokens under the RoPE encoding, which helps preserve spatial alignment.
Notably, unlike OminiControl, which applies LoRA only the condition branch. We apply LoRA to all branches (as shown in Figure 3 (a)). The condition branch is trained with LoRA to seamlessly incorporate the Entity Contour Map. Meanwhile, the image and text branches are also trained using LoRA to ensure precise alignment between the generated entities and the regional text prompts. This approach modifies the linear layers each DiT block across all three branches as follows:
cond
Wig + Bcond-cond:
Winey
Wing + Bimg Aimg:
W Wixt + BrestAsxx
where Acond: Amg. A R and Beond Bing: Biext [Rdr are the low-rank adaptation matrices for each respective branch (r < d). Here, Wing. Wiei Badd denote the original weight matrices of the linear layers.
Shape Guidance Strength Modulation. To adjust the influence of condition image tokens, the attention mechanism is adapted by incorporating a bias term [37], defined as:
QK
Attention (Q K, V) softmmx
+ Bias(())
(5)
vd
where Q, are computed from the concatenation of text tokens, noisy image tokens, and condition tokens. The bias matrix Bias(y) E R + +2am)) x (Lea)) with Liext and Limg denoting the number of text and related tokens respectively, is given by:
Inea Lusu
O O Lima
O X Lime
Bias(r)
Lo x Lesst
O Ling x Lonn
log(?) Lome Ling
(6)
OTeane x Tress
log(7) Time Tring
O X Tius
The factor 7 E (0, 1] serves to modulate the strength of shape conditioning. As y approaches 1, log(7) tends to 0, and the condition tokens retain full influence, enforcing strict adherence to the shape guidance. In contrast,
approaches 0, log(7) tends to -oo, which suppresses the contribution of condition tokens, thus making scribble-style segment masks feasible for flexible control.
Condition Image Token Filtering. Given the sparse nature of the Entity Contour Map, where numerous areas exhibit low or zero values, we discard tokens that are entirely composed zero values, as they provide no shape information. This results in a significant reduction in tokens without compromising shape details. The token filtering process illustrated in Figure (a).
3.3 Attribute Isolation Attention Mask
A critical challenge in multi-instance generation is attribute leakage where visual attributes from one entity transfer to others (as illustrated in Figure 4). To address this, we introduce the Attribute Isolation Attention Mask, defined as Matr-isolnte.
1, if E T,, E (text-text)
1, if Ta E (text-image)
Mattr isslltte [q, ki]
1, if E
(image-text)
,e [0,
(7)
1, if l E I, (image-image)
0. otherwise
Unlike the Semantic Alignment Mask (Eq. 2). the Attribute Isolation Mask operates with stricter constraints, as illustrated in Figure 3 (b). First, it prevents cross-entity visual information interaction by restricting each entity's image tokens to attend only to themselves. Second, it restricts the global caption tokens (To) from attending any foreground image tokens. This complete separation of entities effectively prevents attribute leakage. Notably, background image tokens are still permitted to attend to all image tokens, ensuring environmental coherence.

---

B
W/O
H
Aurrbute Isolation
Attention Mask
Training-free
Attribute Isolation
Attention Mask
B
Truining-based
G
H
Attribute Isolation
M paople
Atlention Mask
Figure 4: Visualization results of different attribute isolation strategies. In Column 1, 20 colored circular badges labeled A to T are required be generated in raster order. The results show that our Attribute Isolation Attention Mask effectively prevents attribute leakage between entities. Columns 2-4 demonstrate that direct application of the mask without training leads to visual inconsistencies, manifesting unnatural shadows and reflections. In contrast, the training-base approach proposed large-scale dataset achieves both strong attribute control and high visual coherence.
Building insights from DreamRender [61] that the middle layers (20-38 layers) of the 57-layer FLUX architecture are dedicated to processing the visual features of individual instances, we apply the Attribute Isolation Attention Mask to these middle layers. In practice, observe that training- free rigid attention constraints often introduce visual artifacts, as demonstrated in Figure Instcad, through training our proposed large-scale datasets, deeper layers learn to refine holistic image quality, achieving an optimal balance between visual harmony and attribute control.
3.4 SA Dataset
To address the lack of large-scale and fine fine-grained datasets for S2I generation, construct SACap- 1M, containing million image-text pairs and 5.9 million segmented entities with detailed descriptions. We propose automated pipeline for data annotation and filtering: Image Filtering. We select images from the high-resolution and wide-ranging SA-1B [16] dataset. Initially, we remove images that are excessively large small in size, as well those with extreme aspect ratios. Subsequently, we apply the LAION-Aestheties predictor [35] to filter out low-quality images with an aesthetic score below II) Entity Extraction. The SA-1B dataset provides accurate, category-agnosti for each image. However, each image contains average over 100 masks, many of which are nested. To filter masks, we retain only top-level masks by removing those that are contained within top-level masks. Additionally, we discard masks whose area is smaller than 1% of the total image area. Finally, exclude images whose number of remaining masks falls outside the range of to 20. III) Regional and Global Caption Annotation. We employ the open-source (wwe22-V--722 [43] model generate captions for each entity and the global image, yielding an average of 58.6 words image and 14.1 words per entity. See supplemental materials for more details.
masks
7

---

Experiments
4.1 Experiment Setup
Baselines. Our method is compared with prior state-of-the S2I approaches, including FreestyleNet [47] and PLACE [24]. Controllable T2I models, such ControlNet [56], Control- Net++ [20], and UniControl [29]. are also considered. Additionally, recent FLUX-based models (DreamRender [61], 3DIS [63], and EliGen [53]) are included in the comparison. Notably, Dream- Render and 3DIS are training-free methods that require depth maps as input instead of segmentation masks. For evaluation of these models, we use Depth Anything V2 [49] to predict depth maps from real images as guidance.
Training datasets and Evaluation Benchmarks. Experiments are conducted on both open-set and closed-set segmentation datasets. For the open-set scenario, we utilize SACap-IM, which consists of I million images accompanied by 5.9 million regional captions. Evaluation for this setting is performed on AACap-Eval. benchmark curated from subset of SACap- 1M, comprising 4,000 prompts with detailed entity descriptions and corresponding segmentation masks, with an average of 5.7 entities per image. For closed-set scenario, [59] and COCO-Stuff [3]. Following prior works [47, 24), for ADR20K and COCO-Stuff, regional captions are assigned the semantic class names of each segment, and no global caption is provided.
select two widely used datasets: ADE20K
Implementation Details. Our experiments are conducted based on FLUX. 1-dv.. The LoRA modules are applied to all linear layers of each block in DIT, with the LoRA rank set to 64, resulting in approximately 594M additional parameters. Across all datasets, our model is trained for 20,000 steps with a batch size of 16, using the A optimizer and a fixed learning rate of 0.0001, The training resolution is set to 1024 1024 for the SACap-IM dataset and 512 x 512 for ADE20K and COCO-Stuff. All experiments are conducted on NVIDIA H100 GPUs
Evaluation Metrics. For closed-set S2I generation, we report both FID and MloU. FID reflects the visual fidelity of the generated images, while MIoU measures semantic and layout consistency, For MIoU calculation, we use Mask2Former [6] for ADE20K and DeepLab [5] for COCO-Stuff to predict semantic segmentation, as done in ControlNet++ [20]. For open-set S2I generation, following CreatiLayout [54], we evaluate S2I quality on SACap-Eval from three perspectives:
Class-agnostic MIoU. We use the ground-truth segmentation masks as mask prompts, and then employ SAM2 [32] to predict class-agnostic segmentation masks for the generated images. The predicted masks are compared with the ground truth to compute the class- class-agnostic MIoU, which measures shape consistency.
Region-wisg quality. Image regions are cropped based on ground-truth segmentation masks and evaluated with Qwen2-VL--2BB [43] in Visual Question Answering (VQA) manner to measure both spatial and attribute accuracy. Specifically, spatial accuracy is evaluated by checking whether each entity appears within the correct region, while attribute accuracy considers whether its color, text, and shape match the provided descriptions.
Global-wied quality. We assess overall visual quality and global caption fidelity using multiple metrics, including the scoring models IR score [46] and Pick score [17], as well as commonly used metrics such CLIP score and FID.
4.2 Quantitative Results
Fine-Grained open-set S21. Table presents the quantitative results on the SACap-Eval benchmark. We report multiple metries covering class-agnostic shape consistency, region-wise, and global-wise qualities. Compared to previous methods, Seg2Any achieves significant improvements across most criteria. Notably, our approach reaches aggnosic MIoU of 94. 94.90, nearly matching the real image upper bound (96.03 class-agnostic MloU), which demonstrates excellent shape consistency. In terms of region-wise assessment, our method excels in region-wise spatial localization and precise attribute control, effectively mitigating attribute leakage. The qualitative results presented in Figure 5 further support this observation. For global quality, our method achieves strong overall performance thanks to the FLUX model and high-quality training data.
8

---

Table 1: Quantitative comparison on the SACap-Eval benchmark. Bold and underline represent the best and second best methods, respectively.
Class-agnostic MloU f
Region-wiss Quality
Global-wise Quality Shape f Texture f IR [46]
Method
Spatial f Color
Pick [171 1 CLIP f FID 1
Real Images
96.03
97.04
93.87
91.66
92.50
FreestyleNet [47]
74.59
42.34
40.08
25.07
40.07
-1.66
18.13
19.66
46.20
PLACE [24]
84.30
79.05
49.40
52.00
57.96
-1.11
19.71
24.69
17.81
EliGen [53]
51.38
83.62
76,94
78,91
77.21
0.49
22.60
27.57
19.69
3DIS [63]
82.09
88.25
85.80
85.87
89.20
0.53
21.81
28.03
15.36
DreamRender [61]
82.32
88.69
87.73
86.76
89.19
0.43
21.58
28.21
13.71
Seg2Any (ours)
94.90
93.89
91.52
88.15
90.12
0.44
21.66
27.87
15.53
Regional prompts
Seg masks
Ours
DreamRender
3DIS
EliGen
PLACE
A person wosttg jocket and paldingg
& Thes aedsiingaa yellow net loyale
Two people paddiing kayak os outer
T.The io deer with cloads coeild
2.A wisdows
dozee
3 Tree dalilel wise
Trem eu riglt als
S.A. sornac pand with small suliboott.
buildlags.
L imsge shows a Mile dky wils soaterss dlodds
portoo bilight stject Liuu
eelegroand
Izeel: Big tovente Lodion, with dnstertive gettor crchitectore lerge
Langeny hemp with wedlin brige bodeggash
beyledk
coentetop
with bulk is uisk und faucst. Woodar drswas with caaaa lancka.
C.A. stuts shee oven with glaes
top
Figure 5: Qualitative comparisons on SACap-Evar Seg2Any accurately generates entities exhibiting complex attributes such as color and texture, surpassing previous approaches.
Coarse-Grained Closed-set S21. Table 2 shows quantitative results for closed-set S2I generation the COCO-Stuff and ADE20K datasets. On COCO-Stuff, Seg2Any achieves the highest MloU. On ADE20K, Seg2Any reaches 54.46 MloU, approaching the upper bound of real images (54.41 MIoU). Although Seg2Any does not surpass PLACE (60.20 MIoU) on ADE20K. it outperforms all other baselines, demonstrating strong overall performance across datasets. Seg2Any exhibits higher FID scores compared PLACE and FreestyleNet, which we attribute to the domain gap between the base model FLUX and the real images in ADE20K and COCO-Stufi
4.3 Ablation Study
We conduct ablation study to evaluate the individual contributions of our key components on the SACap-Evas benchmark (see Table 4). Introducing explicit shape condition via Sparse Shape Feature Adaptation (SSFA) significantly improves the class-agnostic MIoU metric over using only the Semantic Alignment Attention Mask (SAA), confirming the effectiveness of preserving shape details. Further, adding the Attribute Isolation Attention Mask (AIA) notably improves region-wiss quality, indicating better prevention attribute leakage. The training-fres free version degrades global quality; however, after training Table 4 also shows that incorporating Condition Image Token Filtering (CITF) leaves performance almost unchanged; its impact computational cost is further analyzed in the supplemental materials.
large-scale dataset, the overall image quality improves significantly.
On COCO-Stufi and ADE20K, we observe a similar trend: SSFA substantially improves MloU, and CITF shows marginal impact performance. (sce Table 3).

---

Table 2: Quantitative evaluation of S2I models on COCO-Stuul and ADE20K.
Table Ablation results on COCO-Stuff and
ADE20K. We conduct ablation experiments on SAA (Semantic Alignment Attention Mask),
COCO-Stur Stuff
ADE20K
Method
MloU 1 FID J M FID
SSFA (Sparse Shape Feature Adaptation), and
CITF (Condition Image Token Filtering).
Real Images
40.87
54.41
FreestyleNet [47]
42.42
15.12
44,42
28.45
Methods
COCO-Stuff
ADE20K
PLACE [24]
42.23
14.95
60.20
24.51
GLIGEN [22]
23.78
33.02
SAA SSFA CITF MloU
FID 4 MloU
FID +
ControlNet [56]
27.46
21.33
32.55
33.28
/
43.48
20.57
44.85
33.14
UniControl [29]
25.44
46.34
/
/
45.50
19.06
54.11
32.37
Controlnet++ [20]
34.56
19.29
43.64
29.49
V
/
45.54
19.90
54.46
32.89
Seg2Any (ours)
45.54
19.90
54.46
32.89
Table 4: Ablation results on the SACap-Eval benchmark. MIoU" denotes class-agnostic MloU. The ablation focuses on SAA (Semantic Aligmment Attention Mask), SSFA (Sparse Shape Feature Adap- lation), AIA (Training-Free Attribute Isolation Attention Mask) AIA (Training-Baal Based Attribute Isolation Attention Mask), and CTTF (Condition Image Token Filtering).
Methods
Region-wise Quality
Globul-wise Quality
MIoU
SAA SSFA AIAi AIA CITF
Spatial
Color
Shape
Texture
Pick
CLIP
FID
89.39
93.36
89.26
85,84
87.86
0.47
21.83
28.09
15.30
94.16
93.43
89.16
85.61
88.53
0.44
21.81
27.98
15.20
/
94.46
92.37
92.16
87.77
90.23
0.27
21.39
27.79
16.11
94.94
93.36
90.08
87.40
88.93
0.39
21.57
27.97
17.35
94.90
93.89
91.52
88.15
90.12
0.44
21.66
27.87
15.53
Conclusion
We propose Seg2Any, a novel wawpenntttioommss--tt--o--- generation framework that achieves fine- grained layout control by decoupling spatial layout and semantic guidance. Through the integration of sparse entity contours and modai-modal masked attention, Seg2Any simultaneously ensures shape preservation, semantic alignment, and robust attribute control. We further introduce the large- scale SACap-IM dataset and SACap-Eval benchmark to foster open-set S2I research. Extensive experiments validate that Seg2Any achieves new state-of-the-art performance, particularly excelling in generating entities with detailed descriptions.
Limitation. Seg2Any faces resource constraints when generating images with a large number of entities, each accompanied by detailed descriptions. Additionally, large-scale dataset relies vision-language models for regional captioning, which inevitably introduces annotation noise that may impact sotpeennttiionmma--tooiiii generation performance.
6 Acknowledgment
This work was supported in part by the National Natural Science Foundation of China (Grant 62472098).
References
[1] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Igge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint 2xXx:23033008744 2023.
[2] Omri Avrahami, Thomas Hayes, Oran Gafni, Sonal Gupta, Yaniv Taigman, Devi Parikh, Dani Lischinski. Ohad Fried, and Xi Yin. Spatext: Spatio-textual representation for controllable image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18370- 18380, 2023.
10

---

[3] Holger Caesar. Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1209-1218. 2018.
[4] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Zhongdao Wang, James T Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of diffusion transformer for photorealistic text-to- -itttt--mmge synthesis, In International Conference on Learning Representations, 2024.
[5] Liang- Chiih Chen, George Papandreou, Florian Schroff,, and Hartwig Adam. Rethinking atrous convolution for semantic image segmentation. arXiv preprint 1706 05587, 2017.
[6] Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1290-1299, 2022.
[7] Omer Dahary, Or Patashnik, Kfir Aberman, and Daniel Cohen-Or. Be yourself: Bounded attention for multi-subject lext-to-image generation. In Europeun Conference on Computer Vision, pages 432-448. Springer, 2024.
[8] Xueqing Deng, Qihang Yu, Ali Athar, Chenglin Yang. Linjie Yang. Xiaojie Jin, Xiaohui Shen, and Liang-Chich Chen. Concnut-pancap: Joint panoptic segmentation and grounded captions for fine-grained understanding and generation. arXiv preprint arXiv:2502.02582 2025.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In International Conference Machine Learning, 2024.
[10] Yutong Feng, Biao Gong, Di Chen, Yujun Shen, Yu Liu, and Jingren Zhou. Ranni: Taming text-to-image diffusion for accurate instruction following. Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pages 4744-4744 2024.
[11] Yuchao Gu, Yipin Zhou, Yunfan Yc, Yixin Nic, Licheng Yu, Pingchuan Ma, Kevin Qinghong Lin, and Mike Zheng Shou. Roictrl: Boosting instance control for visual generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23658-23667, 2025.
[12] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5356- 5364, 2019.
[13] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. International Conference Learning Representations, 1(2):3, 2022.
[14] Siteng Huang, Biao Gong, Yutong Feng, Xi Chen, Yuqian Fu, Yu Liu, and Donglin Wang. Learning disentangled identifiers for cectoo-customizzed text-to-image generation, In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 7797-7806, 2024.
[15] Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Wox Bon Ha, and Jun-Ya Yan Zhu. Dense text-to-image generation with attention modulation. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 7701-7711, 2023.
[16] Alexander Kirillov, Fric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4015-4026, 2023.
[17] Yuval Kirstain, Adam Polyak, Uriel Singer, Shahbuland Matiana, Joc Penna, and Omer Levy. Pick-a- pic: An open dataset of user preferences for text-to-image generation. Advances in Neural Information Processing Systems, 36:36652-36663, 2023.
[18] Black Forest Labs. Flux. https: /tpp::///iitiiie httpp////ggtaa...cmbaaaattt--ooe wwt.:///aaaaat-rrreeaaaaaaaaaaaa 2024.
[19] Yuseung Lec, Tachoon Yoon, and Minhyuk Sung. Groundit: Grounding diffusion transformers via noisy patch transplantation. Advances in Neural Information Processing Systems, 37:58610 586366 2024.
[20] Ming Li, Taojiannan Yang. Huafeng Kuang, Jie Wu, Zhaoning Wang. Xuefeng Xiao, and Chen Chen. Controlnet++: Improving conditional controls with efficient consistency feedback. In European Conference Computer Vision, 2024.
11

---

[21] Yu Li. Xingyu Qiu, Yuqian Fu, Jie Chen, Tianwen Qian, Xu Zheng, Danda Pani Paudel, Yanwei Fu, Xuanjing Huang, Luc Van Gool, et al. Domain-rag: Retrieval-guided compositional image generation for cross-domain few-shot object detection. arXiv preprini wwXiv:2506.08872, 2025.
[22] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang. Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded extt-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22511 22511-22521, 2023.
[23] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects context. In European Conference on Computer Vision, pages 740-755. Springer, 2014.
[24] Zhengyao Lv, Yuxiang Wei, Wangmeng Zuo, and Kwan- Yee K Wong. Place: Adaptive layout-semantic fusion for semantic image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9264-9274, 2024.
[25] Yuhang Ma. Shanyuan Liu, Ao Ma. Xiaoyu Wu, Dawei Leng, and Yuhui Yin. Hico: Hierarchical controllable diffusion model for layout- ayrouttt-iimaee generation. Advances in Neural Information Processing 2024.
Systems,
[26] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAl conference on artificial intelligence, volume 38, pages 4304, 2024.
[27] Marianna Ohanyan, Hayk Manukyan, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Zero- painter: Training-frce layout control for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8764-8774, 2024.
[28] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 4195-4205, 2023.
[29] Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang. Yingbo Zhou, Huan Wang. Juan Carlos Niebles, Caiming Xiong. Silvio Savarese, al. Unicontrol: a unified diffusion model controllable visual generation in the wild. In Advances in Neural Information Processing Systems, pages 42961 -42992, 2023.
[30] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovic, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang. Aaron Marquez. Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IFEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7141-7151, 2023.
[31] Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Eric Xing, Ming--Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13009-13018, 8, 2024.
[32] Nikhila Ravi, Valentin Gabeur, Yuan- Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson. et al. Sam 2: Segment anything in images and videos. arXiv preprint (tt.:2400.0071111 2024.
[33] Zhongwei Ren, Zhicheng Huang, Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26374-26383, 2024.
[34] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffuusin models. In Proceedings of the IEFF/CV Conference Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[35] Large scale Artificial Intelligence Open Network (LAION). Laihn-aesthetics-v2. https://githhn com/ christ chuhhmann/ -mmproodd-asstthtii--eeeeee prediiicceppeiiooo..o.o 2022.
aesthetic-
[36] Takahiro Shirakawa and Seichi Uchida. Noisecollage: A layout-aware text-to-image diffusion model based on noise cropping and merging. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8921-88888888 8930, 2024.
[37] Zhenxiong Tan, Songhua Liu, Xingyi Yang, Qiaochu Xue, and Xinchao Wang. Ominicontrol: Minimal and universal control for difffusion transformer. 2025.
12

---

[38] Rui Tian, Mingfei Gao, Mingze Xu, Jiaming Hu, Jiasen Lu, Zuxuan Wu, Yinfei Yang. and Afshin Dehghan. Unigen: Enhanced training & test-time strategies for unified multimodal understanding and generation. arXiv preprint arXiv:2505.1400 14682, 2025.
[39] Shuyuan Tu, Yueming Pan, Yinming Huang, Xintong Han, Zhen Xing, Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Stableavatar: Infinite-length audio-driven avatar video generation. arXiv preprint arXiv:2508, 08248, 2025.
[40] Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng, Qi Dai, Chong Luo, and Zuxuan Wu. Stablcanimator: quality identity-preeervng human image animation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 21096-21 106, 2025.
[41] Shuyuan Tu, Zhen Xing, Xintong Han, Zhi-Qi Cheng. Qi Dai, Chong Luo, Zuxuan Wu, and Yu-Gang Jiang. Stablcanimator++: Overcoming pose misalignment and face distortion for human image animation. arXiv preprint arXiv:2507. 15064, 2025.
[42] Junke Wang. Zhi Tian, Xun Wang, Xinyu Zhang, Weilin Huang. Zuxuan Wu, and Yu-Gang Jiang. Simplear: Pushing the frontier of autoregressive visual generation through pretraining, sft, and rl. arXiv preprint arXiv:2504. 11455, 2025.
[43] Peng Wang, Shuai Bai, Sinan Tan, Shijic Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model's 's perception of the world at any resolution. arXiv preprint arXiv:2409. 12/91, 2024.
[44] Xudong Wang, Trevor Darrell, Sai Saketh Rambhatla, Rohit Girdhar, and Ishan Misra. Instancediffusion: Instance-level control for image generation. In Proceedings of the IEEF/CVF Conference Computer Vision and Pottern Recognition, pages 6232-6242, 2024.
[45] Jialian Wu, Jianfeng Wang. Zhengyuan Yang. Zhe Gan, Zicheng Liu, Junsong Yuan, and Lijuan Wang. Grit: A generative region-to-text transformer for object understanding. In European Conference on Computer Vision, pages 207 224, Springer, 2024.
[46] Jiazheng Xu, Xiao Liu, Yuchen Wu, Yuxuan Tong, Qinkai Li, Ming Ding, Tang, and Yuxiao Dong. Imagereward: Learning and evaluating human preferences for text-to-image generation. Advances in Neurul Information Processing Systems, 36: 1611531-15935 2023.
[47] Han Xue, 7hiwu Huang, Qianru Sun, Li Song, and Wenjun Zhang. Freestyle layout-to-image synthesis. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 14256- 14266, 2023.
[48] Binbin Yang. Yi Luo, Ziliang Chen, Guangrun Wang, Xiaodan Liang, and Liang Lin. Law-diffusion: Complex scene generation by diffusion with layouts. Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 22669-22679, 2023.
[49] Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. Advances in Neural Information Processing Systems, 37:21875-21911, 2024.
[50] Zuyao You, Junke Wang, Lingyu Kong, Bo He, and Zuxuan Wu. Pix2cap-coco: Advancing visual comprehension via pixel-level captioning. arXiv preprint arXiv:2501. 2025.
[51] Yuqian Yuan, Wentong Li, Jian Liu, Dongqi Tang, Xinjie Luo, Chi Qin, Lei Zhang, and Jianke Zhu. Osprey: Pixel understanding with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28202-28211 2024.
[52] Yu Zeng, Zhe Lin, Jianming Zhang, Qing Liu. John Collomosse, Jason Kuen, and Vishal M Patel. Scenecomposer: Any-level semantic image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages -22478, 2023.
[53] Hong Zhang, Zhongjie Duan, Xingjun Wang. Yingda Chen, and Yu Zhang, Eligen: Entity-level controlled image generation with regional attention. arXiv preprint (xXx:255010009) 2025.
[54] Hui Zhang, Dexiang Hong, Tingwei Gao, Yitong Wang, Jie Shao, Xinglong Wu, Zuxuan Wu, and Yu-Gang Jiang. Creatilayout: Siamese multimodal diffusion transformer for creative layout-to-image generation. 2025.
[55] Hui Zhang. Dexiang Hong, Maoke Yang, Yutao Cheng, Zhao Zhang. Jie Shao, Xinglong Wu, Zuxuan Wu, and Yu-Gan Gang Jiang. Creatidesign: A unified multi-conditiona diffusion transformer for creative graphic design. arXiv preprint 2xXiv25055.19144 2025.
13

---

[56] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-lo-image diffusion models. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 3836-3847, 2023.
[57] Shilong Zhang. Peize Sun, Shoufa Chen, Minn Xiao, Wenqi Shao, Wenwei Zhang, Yu Liu, Kai Chen, and Ping Luo. Gpt4roi: Instruction tuning large language model on regionof-interes 2024. URL
https://openreview. net/forum.
[58] Guangeong Zheng, Xianpan Zhou, Xuewei Li, Zhongang Qi, Ying Shan, and Xi Li. Layoutdiffusion: Controllable diffusion model for layout to -iaaage generation. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22490-2222 22499, 2023.
[59] Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through adc20k dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 633 641, 2017.
[60] Dewei Zhou, You Li, Fan Ma. Xiaoting Zhang, and Yi Yang, Mige: Multi-instance generation controller for text-to-image synthesis. In Proceedings of the IEFF/CVF Conference on Computer Vision ond Pattern Recognition, pages 6818-6825 2024.
[61] Dewei Zhou, Mingwei Li, Zongxin Yang, and Yi Yang. Dreamrenderer: Taming culti-inssaace attribute control in large- scale text-to- -mage models. 2025.
[62] Dewei Zhou, Ji Xie, Zongxin Yang, and Yi Yang. 3dis: Depth-driven decoupled instance synthesis for text-to-image generation. International Conference Learning Representations, 2025.
[63] Dewei Zhou, Ji Xie, Zongxin Yang, and Yi Yang. 3dis flux: simple and efficient multi instance generation with dit rendering. arXiv preprint arXiv:2501.055: 301131, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we introduce novel segmentation-mast image generation approach and describe the contributions in the Abstract and Introduction section (see the Abstract and Introduction).
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations of our work in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-specfiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer:
[NA]
Justification:
This paper does not include experimental results related to theoretical aspects.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification:
We detail the experimental setup in Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily new algorithm, the paper should make it clear how to reproduce that algorithm. (b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
16

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [No]
Justification: Our data and code are not open-source yet. However, we have provided sufficient details of method and all the required hyperparameters in the main paper and supplementary material to enable the reproduction. We plan to open-source the data and code in the future.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ttpps.ccppuulll//// wwt.cc/pppcccppeeeeeeecceec /etcc//ggdee//.eeeee (ooeeSubisss: wwwsssssssiiooooooooooooossossio for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification: We have specified all the training and test details necessary to understand the results (See Section 4).
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification:
Due to extensive computation and time cost, we do not report error bars.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not show in tables figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: In this paper, we provide detailed information about the experimental resources, including GPU configurations used in our studies (See Section 4).
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https http:s:///euueee http///eerpppp.eeee httpp//ccc.//hihceceeeeeeeeeeeee
Answer: [Yes]
Justification:
The study presented in this paper conforms to the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (c.g.. if there special consid- eration due to laws or regulations in their jurisdiction..
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
See supplementary material.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
18

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g.. gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question:
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [No]
Justification:
This paper does not address issues related to this aspect.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All creators and original owners of the assets used in our paper, such as code, data, and models, have been properly credited (See reference). We have explicitly mentioned the licenses and terms of use for each asset and have ensured full compliance with these terms throughout our research.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
19

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [No]
Justification: The paper has not released new assets yet.
Guidelines: The datasetssetccodd/moddel include details about training, license, limitations, etc.
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question:
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve experiments research related human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with human subjects.
20

---

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification:
The core method development this research does not involve LLMs.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: fnttps://netti /ttp///een...peeeeeee wwt.e/cceeeceeecc//e/aaa for what should or should not be described.
21

---

A More Details of Condition Image Token Filtering
We analyze the computational cost of Condition Image Token Filtering (CITF) three benchmark datasets: COCO-Stufi ADE20K, and EACap-IM. For each dataset, we consider five settings to comprehensively analyze the computational cost: without condition tokens, representing the lowest computation. II-IV) CITF applied with the minimum, maximum, and average numbers of retained condition tokens, respectively. V) without CITF, i.e., token filtering, representing the highest computation. For each configuration, we report the resulting average image generation time and multppy-accumulate operations (MACs).
Average Generation Time per Image
Computational Complexity
Method
Greani-: Decruass is. wis CIT
Method
Creen::: Decrusse vs. CIT Redt 12 tncwase Tooe
Cood
Rect Inchesee Dios
Cond
Cood
30
Tolons)
17.5
(Max
30
CIFF
15.0
Time (seconds)
25
12.5
20
10.0
+0.
ma 15.37
-7.9%
15
5.0
-3.9
10
16.1
7.5
7.2 7.5
11.44 11.03
11.4 11.14
4.5
0.0
SACop
Figure 6: Impact of Condition Image Token Filtering (CITF) on computational cost.
Figure illustrates the results. All experiments are conducted on a single NVIDIA H100 GPU with 32 sampling steps. Each image uses 5 regional text prompts (50 tokens each) and one global prompt (512 tokens). As shown, CITF leads to a notable reduction both inference time and MACs when the entity contour maps sparse. When more tokens retained, computational savings decrease. Overall, CITF provides a simple yet effective mechanism to reduce inference overhead, particularly in cases with sparse entity layouts.
flowers
leaves
adorable orange bear
text Seg'
text 22' in yellow bold
Seg 2 ANY
in red bold
quitar
text 'Aay"
campfire
green dragon
green dragon
spraw ling tree
anime girl
weathered gr piano
Figure 7: Qualitative results with shape guidance strength (7 = 0.2), demonstrating flexible scribble- style control.
22

---

B More Details of Shape Guidance Strength Modulation
We empirically find that adjusting the strength hyperparameter Y Shape Guidance Strength Modulation serves as a "free lunch", enabling scribble-style control without incurring any additional training overhead. our experiments. we set ? 0.2, which provides sufficient flexibility for scribble-style segment masks while still maintaining semantic alignment. The qualitative results are shown in Figure 7.
C More Ablation Study Results
C.1 Ablation on AIA
To further verify the effectiveness of the proposed Attribute Isolation Attention Mask (AIA), we conduct an ablation experiment by removing the AIA module from Seg2Any. As illustrated in Figure 8, the absence of AIA leads to attribute leakage across entities.
Segmentation masks
Without AIA
With AIA
[) Siamese Tab Orange cat. 4) Sphynx cat.
1) Orange slice, Apple sliee,
3) Dragon fruit slice. 4) Apple slice. 5) Kiwi slice.
I) LEGO lion, 2) Ice penguin.
3) Flaming cagle.
Figure 8: More qualitative ablation comparisons on the Attribute Isolation Attention Mask (AIA). Without AIA, attribute leakage occurs across entities, e.g., fur texture is transferred between cat breeds (row 1), the red peel of dragon fruit replaced by kiwi's green peel (row 2), and the LEGO style spreads to other entities (row 3). In contrast, with AIA, such attribute leakage is effectively eliminated.
23

---

C.2 Ablation on SSEA vs. PLACE Attention
To ensure a fair comparison between the proposed Sparse Shape Feature Adaptation (SSFA) and PLACE attention [24], we reimplemented PLACE attention [24] under the same FLUX architecture, while keeping all other training setups fixed. Specifically, we use learning rate of le-4, a batch size of 16, and train for 20k steps on NVIDIA H100 GPÜs. As shown in Tables 5 and 6, our model surpasses this PLACE attention variant by (+2.55%) MIoU on ADE20K and (+3.36%% class-agnostic MIoU on -vaa,- confirming the superiority our design in a strictly fair comparison.
Table 5: Comparison with PLACE attention under the FLUX model on COCO-Stuff and ADE20K.
ADE20K
Method
MloU f FID
MIoU
FID L
SAA
43.48
20.57
44.85
33.14
PLACE attention
44.56
20.10
51.56
32.22
SAA+SSFA
45.50
19.06
54.11
32.37
Table 6: Comparison with PLACE attention under the FLUX model on SACap-Eval.
Class-agnostic MloU f
Region-wise Quality
Global-wise Quality
Method
Spatial
Color
Shape
Texture
IR f Pick f CLIP
FID
SAA
89.39
93.36
89.26
85.84
87.86
0.47
21.83
28.09
15.30
PLACE attention
90.8
93.03
89.12
86.36
88.08
0,51
21.90
27.93
15.81
SAA+SSFA
94.16
93.43
89,16
85.61
88.53
0.44 21.81
27.98
15.20
Figure 9 presents the synthesis results the ADE20K dataset using SAA, PLACE attention, and SAA+SSFA, all trained under the same FLUX architecture. It can be seen that SAA+SSFA generates shapes that best conform to the provided segmentation masks.
Segmentation masks
SAA
PLACE attention
SAA+SSFA
LIAT
Figure 9: More qualitative ablation comparisons on the Sparse Shape Feature Adaptation (SSFA). All methods are trained under the same FLUX architecture.
D More Qualitative Results
We provide additional qualitative results Figure to illustrate the effectiveness of our approach. Seg2Any shows strong capability in generating high-quality images that faithfully adhere to detailed entity descriptions, enabling flexible and precise control over complex visual attributes such as color, texture, and shape. Figure 11 presents the qualitative results of Seg2Any in densely crowded scenes.
24

---

L person wearlig a cap and
1.A metocress ridee weating
1.A small green and yallow
yillow guometnic shape on dafi
graphic t-short is holding and sating
protective geer and riding yellow
Christmas frve with orsaments.
Jour:
sandwnch in bnghily colord indoor
dist like.
LEG0 Santa Claus figure with
2. yelow geomettic shape on dark
setting
2. yellow and w white dut bike with
red sult and bat holding browa bed.
Σoor:
2. The porzon is holding a sandwich
the number "19" on it. being ridden
3.A colortul Lego Chaistmas house
3.A large metallic, thrse- dimensional
with a scsame saed hun.
bs pesson protective gear.
wett green doot, red and white
1000 on a dark floor
A ned water buttle placed un the lucke sext the person.
3. A motocross nider on yellow dirt
deccraaoon,, and small tree neartyy
A.A large metallic, three dimenstonal oge on a dark floor
bika, weartng protective genr,
4. rod and yellow slrigh with two
1. local gres testures while and checkered wall with framed poster or sign. at top right comnr
ragrared air against a backsmp
manneer figures next
A lame, metallic logo on dark polished floor with teetumd walls on cithur side.
of green traes.
The toxt "One" is written on the wall.
A tall wooden wantrode with muiltpple compartments and doors, providing storage space bedrocm.. 2.A neetly made wiblte hedding and pillows posttoned) noar lame window with platform
bottle of ove pistachio cream maggoolia shower with watst droplets
L heimet with straps and visor placed on the desk. 2 rulee with cnessuremests on or board posiiioned behind aptop the desk, A black and white
The toxt "Step" is written on the wall.
A.A. bottle Dove pistachio cream & magnolla shower gel with water droplets on 3.A.
yellow panel
wall-monnted light fexture frosted glass shape and black
3.A
vellow purf or sponge
gold
extending light freture block wall.
handli,,
Cola
bedfsprend pillows.
purple pumine
items, relescupe:
light calored with lighting pllaced dessomative Large
coffoliating placed
paper bundwritten nottes pen.
mounted light frosted class shape black metal
drupless, bottles
-sptup
displaying desk
extending from. wall.
magnolia
word "Blueberry"
building image mounted
gel, loofah sponge:
surmonddd various
druplets, likely
blue
sign "Chue Step"
counter shower
smell components
blinds, allowing
buth
calendar showing
light
A
one step
Dove
21
dolvedd
Figure 10: More qualitative results using the Seg2Any method.
25

---

Figure 11: Qualitative results Seg2Any densely crowded scenes.
E More Details on Datasets and Benchmarks
E.1 Comparison with Existing Dense Caption Datasets
Table presents detailed comparison between our SACap-IM IM dataset and several recent dense caption datasets. Unlike previous datasets such as PixeLLM-MUSE [33], Osprey [51], COCONut- PanCap [8] and Pix2Cap-COCO [50] which are constructed upon closed-set label spaces (e.g.. LVIS [12] and COCO [23]). our proposed SACap-IM dataset provides open-set segmented entities, enabling much broader generalization and flexibility open-set segmentation-man mask-to-image to-image genera- tion. Compared to I.LMMM-Grranl [31], also derived from SA-1B [16], our SACap-1M achieves significantly higher segmentation mask density and caption granularity by employing the advanced open-source Dwen2-VL-722 [43] vision- vision-language model to generate more accurate and fine-grained regional captions.
Table Comparison of dense caption datasets. Note that "Avg. Words" indicates the word count per regional caption, and "Avg. Masks" denotes the average number of masks per image.
Name
Image Source
Image Number Annotated by
Avg. Words Avg. Masks
PixcILM-MUSE [33]
LVIS [12]
246K
GPT-VV (J]
11.3
COCO [23]
Osprey [51]
724K
GPT-4V [1]
38.7
PACO-LVIS [30]
GPT4Rol [57]
GLaMM-GranD [31]
SA-IB [16]
11M
GRiT [45]
5.8
GPT-4V [1]
COCONur-PanCap [8]
COCO [23]
118K
16.6
13.2
and human conrection
GPT-VV [1]
Pix2Cap-COCO [50]
COCO [23]
20K
22.94
8.14
and human correction
SACap-IM IM (ours)
SA-IB [16]
IM
Owen2-VIL-728 [43]
14.1
5.9
E.2
Data annotation pipeline
We construct our dataset from the SA-IB [16] dataset through multi-stage filtering and annotation process:
Image Filtering. Images filtered based size and aspect ratio, keeping those with width and height between 1000 and 3000 pixels and aspect ratios from 0.6 to 1.8. further ensure visual quality, images with LAION-Aestheties [35] score below 5 excluded.
II) Entity Extraction. While SA-1B provides numerous accurate, class-agnostic segmentation over 100 per image many nested and redundant. To mitigate this redundancy,
26

---

O
person wearting motorcycle gear next
adventure motoreycle overlooking scenic lindscape.
mountain range with green vegetation and rocky terrain.
Flltering
motorgycle with storage boxes parked gravel road.
image size limit
aspect ratio
The sky is clear and blue, indiiatingg good weather conditiuns for outdoor activities
aesthetic scoring model
gracel mad with motoreyele parlod surrounded by scomic sconiclandscy landscape with mointaiis in the --
VLM Region caption
What is in <|box_start>> Provide descrpttoon of this local area within 10-20 words.
VLM
Global caption
Generate description the image
standing next BMW R125505 motorrycle gravel
The shows
Select top-level mask
rider wearing protective gear, including helmet, jacket, and pants. The mutortycle is equipped with side cases and tup case, suggesting
riding. background, long-distance travel adventure (blue siyy, indicatinga scente outdoor location.
mountains
Figure 12: An overview of the data annotation pipeline.
retain only the levelevel masks, which generally correspond to the primary objects in the image, and discard any masks fully enclosed by others (as shown in Figure 12). Additionally, masks occupying less than 1% of the image area considered too small and removed. Finally, only images containing between and 20 valid masks are retained for further annotation.
III) Regional and Global Caption Annotation. We utilize the vision-language model Qwen2- VL-72B [43] for both regional and global captioning. For regional captioning, we incorporate the bounding box coordinates of each segment into the text prompt (see Figure 12), enabling the model to produce accurate and context-aware descriptions of each local region. This approach yields detailed annotations, with average of 14.1 words per entity and 58.6 words per image, supporting fine-grained segmentation- to-image generation.
Finally, we present SACap-1N M, large-scale layout dataset containing 1 million image-text pairs and approximately 5.9 million segmented entities. As illustrated in Figure 13, most images contain a moderate number of masks, while images with a large number of masks constitute the long tail of the distribution. Figure 14 shows some examples from SACap-IM.
Training Set Mask Distribution
Test Set Mask Distribution
20.0%
17.8%
17.5%
15.0%
15.0%
Frequancy (8g)
Frequency (95)
12.9%
10.2%
10.0%
7.8%
5.%%
2,5%
1.2%
LEN 81% ATh
Masik per Image
Moskk per Image
Figure 13: The distribution of the number of segmentation masks per image across the training and test sets.
27

---

A pirk perfume that meis "My Sceat" and has
A startless man mounii white basing glores
small embiem
and shorts white trim
atrig of peads
beige sarfico.
gairol besing shorts trime
4 grwn partially
side
and lest.
image
sbirtiess bover
tim
asasuuar sontings with parple beads
?
prepartig Bhrow punch.
black
white parr--lle baads aaronged on surfice.
A
"December diglaying
person werting hemmet and protective gear
yollow moliorycla
tisnt witeel
rhiin wrepped
molarevets protectio
Carczmns
seluet zidtng aspbail rood with the A grass road.
decopatd Christrae
ecnamnett burlap
pellow located
figure standing next
CD 2 Develle
worder Christmas
Grussy nost roa..
A.A. bottle Pezsi Mioe
several visible the distacee. 2 drucure 3.A.
the table A Mask dispocable
lid. a. shoce of loead dark spread
bue shirt
A tra toead in vegetabies,
white backppak back stzdding
Inxluding broad. A of bread placed the plate food. 7.A with vegetalliss and
driak
strps
G black heeaphooes with custacoed headband, and
Jookcag fonwards
B.A
7.A with short dark heedphhoes eirglanses, The cllarrand ploe with visiiie clouds
sroden table with try contaiting food, drink, and usenels outdoues stone A ioone pareed wallowey vieliie behind the sabie and beach.
L. Lust grean troe with denae fnllage stand naar 2.A tee with leaves stanck near the byiiding A madiomall Chirana- style building with oninge mot and nol pillers silusted nsara body water 4 white boa-- structuro with tatioaate dasigts, feoaing the weter naar the buiising. & Tarrr are troes bebind the butidigg A traditooal Chite sbile building with an orange roof and rdd pillars
L bastball player in vhiito jenny and black shorts white show with blue greprring mosive boll, 2 hendfelll player
the
in with the nember her 67, jumping and solding ball. handhell player
white
csses with number 7. jumping and stampting block the ball. A blus and grean socmbooard with taxt and sumbers dispayed oz A hize wall with white but and loges likzly part of
inbletingg the water 7.A cales body waver intlwdingg the sumondingg atructunss and tiaes.
sports facilit or arnna. & gaay wall with poctongutaa window passi.
The sky is Hiue some wille clouds pealtered turoogoout
LAA succer player weeting az orange jettes and
black and path.
white shorts stands oz the feid.
bacopeck welking
A suscer player wertnng white jarsey and blhe
brown and binds jechet, blin jeirs 3.A. wwerng black ond sosef, and hat wilking weating brosen coot and bine joass walking show. S.A pesson weartng red facket and hat walking oz snow A snowy landscape with five people wallinng acm woacaa vanter clothnng and canyiia barkspacks A row of bnildings with multiple windows. lially rosidential or conmenciaal dructinss pysiest soowyy beckdopp.
stors standing on the fildd
bockgerk, wilking
3 soceer player woaring yellow uniffrm
sizeding on Thii rld.
snow. 4
& wabeor plaper ishie and blus uniforn
standing os the Exid.
S.A soceergles player lying on the ground with lis hurds anvening Its face.
& wacerr player waining patito jerway and blue sheers is standing on the field.
L woodes sloce with light colored wood platiss
LA woocen door with doconatie parels and
and sncoth fansh A wooten tabe or desk
doodkoah. 2 wooden cabinet with plass doors.
with a smooch surface and ligit browa color 3.A.
wooden cainett with a mimer above it and
bagg thow piloor on the toach. A.A. while pillon
light fecture 2a the wall 4 larg mimor with on
Large windows with sieer curtuns and mooten
kmo frome mounted ahove the vantyy S.A
hindss aloncing naturi light to serve the room,
feceestive light fistur with testured glas studle
cozy licing 2o0z with vooden table while
hangs from the celling
decorative backroinah
sota, and lazz windows hise betrooms visile bedhgened the bvd. B.A. biue wall with white shxlf and osling
with intiicato patterss and statue of boce on
denugt open door. 7.A
hpp. 7. white light owisch panel multpple pwitches mousted the well. &t worden varily with back countertop iyg sink, and dscerstive minor abowwit éecomatio floor tie with guomeric pottor fruduring swiths and
dorrway leednng to nouther
feetrmi modeem light fixturm and decorative comice ilong wigus eailing light ficture wilh wams unber glow.
Figure 14: Examples from the SACap- M dataset.
28

---

E.3 SACap-Eval Benchmark
We construct SACap-Eval, benchmark derived from SACap- 1M, designed assess the quality of segmentation-mass masiioo---moooo---ooooo mass-to-image generation. The benchmark comprises 4,000 samples, with an average 5.7 entities per image. Evaluation conducted from two perspectives: Spatial and Attribute. Both aspects are assessed using the vision-language model Qwen2- VI.-72B [43] via visual question answering manner.
Spatial Score. For each segmentation mask, we first crop the corresponding region from the image and then prompt the VLM to determine whether the target entity is located within this area, allowing responses of either "Yes" or "No". The spatial score is obtained by computing the ratio of "Yes" answers the total number of entities.
To compute the attribute score, the region specified by each segmentation mask is cropped from the image, after which the VLM determines whether the entity inside this area satisfies the described attributes. Each attribute (e.g. color, shape, or texture) examined separately using visual question answering, and the score is calculated in the same manner the spatial score.
Attribute Score.
F
Broader Impacts
The proposed method for segmentation -as---aaso-ttt tomiaag synthesis has potential applications in con- trollable image generation [21, 14], video generation [40, 41, 39], and the construction of image segmentation datasets. However, like other generative models, its misuse could result in the pro- duction of misleading or inappropriate content. Our approach may also inherit biases present in the training datasets, potentially reinforcing certain stereotypes. Responsible usage and further investigation into fairness and robustness are important, but comprehensive analysis is beyond the scope of this work.
29


Reviews:
[{"review_id": "j86wVt30EN", "reviewer": "Reviewer_tBGK", "summary": "This paper Seg2Any framework to achieve open-set semantic-to-image generation with improved semantic consistency and shape consistency. It proposes Semantic-Shape Decoupled Layout Conditions Injection and Attribute Isolation Attention Mask to improve model performance. In addition, it builds a large-scale datasets SACap-1M with images, open-set segmentation mask annotations, image-level captions and regional captions.", "questions": "1. According to the Ablation study in Table 3 and Table 4, the Condition Image Token Filtering (CITF) seems to only sightly improve the class-agnostic MIoU while negatively affecting other metrics. Statistically, the overall contribution of CITF is negative, but authors claimed \"Table 4 ... Condition Image Token Filtering (CITF) leaves performance unchanged\" and \"CITF does not affect performance (see Table 3)\". More explainations are needed here.\n2. Figure 3 column 4 does not support the statement in caption: \"direct application of the mask without training leads to visual inconsistencies, manifesting as unnatural shadows and reflections\". Shadows of the plane in Training-free AIA and Training based AIA results are both unatural, even worse than W/O AIA.\n3. What is the definition of 'attribute' in the term 'attribute leakage'? Authors claims that failures in Figure 3 Row 1 are caused by 'attribute leakage', but it seems that the biggest disadvantage of W/O AIA w.r.t Training-based AIA is the text generation.  One of the  most impressive advanatages of Seg2Any in Figure 4 is also good text generation. However, all these experiments can not prove the contribution of AIA, because it is entirely possible that this improvement comes from other modules. Stronger evidence is needed.\n4.  The AIA prevents one entity token attending to other entity tokens, background tokens and global text token. Wouldn't this strategy undermine the global perception and long-term dependences among entities?\n5. Authors should provide attentive map or visualize some results in ablation study section. In particular, it is better to show how the results of attentive map changes with respect to different configurations of key components.\n6. Different key components improve performance in different aspects and the final determined version did not achieve the best results in most of the indicators. Authors should clarify their reasons and dicuss the balance and interply of different components.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. Good writing;\n2. High novelty\n3. Abundant content\n\nWeakness:\n1. Relatively poor persuasion of ablation study\n2. Lacking visualization of ablation study", "was_revised": true, "final_justification": "4.Borderline accept", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.\n# Response\n## Response to questions\n> Q1: According to the Ablation study in Table 3 and Table 4, the Condition Image Token Filtering (CITF) seems to only sightly improve the class-agnostic MIoU while negatively affecting other metrics. Statistically, the overall contribution of CITF is negative, but authors claimed \"Table 4 ... Condition Image Token Filtering (CITF) leaves performance unchanged\" and \"CITF does not affect performance (see Table 3)\". More explainations are needed here.\n\nThank you for pointing out the lack of rigor in our previous statements regarding the Condition Image Token Filtering (CITF). We agree that it is imprecise to claim that CITF \"does not affect performance\" or \"leaves performance unchanged.\" Upon closer inspection of Tables 3 and 4, there are indeed minor quantitative influences in certain metrics. \nWe acknowledge that overall metrics like FID, CLIP, IR, and Pick may not fully reflect human perceptual quality. **We will provide more visual examples in the final version to show that CITF does not noticeably impact image quality.**\nOur main motivation for using CITF is to accelerate training and inference while maintaining similar performance, as shown in Figure 1 of the supplementary material. We will revise to clarify these points and highlight both the efficiency gains and the minimal effect on generative quality.\n\n> Q2: Figure 3 column 4 does not support the statement in caption: \"direct application of the mask without training leads to visual inconsistencies, manifesting as unnatural shadows and reflections\". Shadows of the plane in Training-free AIA and Training based AIA results are both unatural, even worse than W/O AIA.\n\nThank you for the careful inspection. We agree that the shadow under the plane in Figure 3 column 4 is too subtle to clearly demonstrate the benefit of training the Attribute Isolation Attention Mask (AIA). **In the final version we will replace this instance with a more salient example.**\n\n> Q3: What is the definition of 'attribute' in the term 'attribute leakage'? Authors claims that failures in Figure 3 Row 1 are caused by 'attribute leakage', but it seems that the biggest disadvantage of W/O AIA w.r.t Training-based AIA is the text generation. One of the most impressive advanatages of Seg2Any in Figure 4 is also good text generation. However, all these experiments can not prove the contribution of AIA, because it is entirely possible that this improvement comes from other modules. Stronger evidence is needed.\n\nThank you for highlighting the insufficiency of our evidence.\nWe define \"attribute\" as instance-level visual properties including color, texture, style and shape. In Figure 3 column 1, the letter on each badge is also an attribute, as explicitly requested by the prompt \"circular badges labeled A–T\".\n**Owing to NeurIPS 2025 policy we cannot include new figures in the rebuttal, but we will add four new study cases in the final version to prove the contribution of AIA:**\n1. Three animals—Lego-style lion, ice-style penguin, flame-style eagle—where without AIA the Lego texture leaks to the penguin and eagle.\n2. Nineteen fruits. Without AIA, the red/green apples swap colors and the kiwi incorrectly takes on dragon-fruit hues.\n3. Four cats—Maine Coon, orange tabby, calico, Sphynx—where fur color and ear/eye shapes are mixed without AIA.\n4. Three people wear wool and leather. Without AIA, the wool texture appears on the leather jacket.\n\n>Q4: The AIA prevents one entity token attending to other entity tokens, background tokens and global text token. Wouldn't this strategy undermine the global perception and long-term dependences among entities?\n\nYour concern is well-taken but is already mitigated in our design.\nThe Attribute Isolation Attention Mask (AIA) is applied only to the middle layers (layers 20–38) of the 57-layer FLUX model, which are responsible for instance-level rendering. All other layers retain full self-attention among entity, background, and global-text tokens, thereby preserving global context and long-range dependencies.\nIndeed, the realistic shadows and reflections shown in Figure 3 (columns 2–4) empirically confirm that our training-based AIA does not compromise long-term dependences among entities.\n\n> Q5: 5. Authors should provide attentive map or visualize some results in ablation study section. In particular, it is better to show how the results of attentive map changes with respect to different configurations of key components.\n\nThank you for this helpful suggestion. **Owing to NeurIPS 2025 policy we cannot include new figures in the rebuttal, but the camera-ready version will add visualizations of the attention maps for each key component:**\n1. SSFA (Sparse Shape Feature Adaptation). We visualize the attention scores between Entity-Contour-Map tokens and noisy-image tokens. High scores appear for tokens at identical spatial indices, confirming spatial alignment under SSFA.\n2. AIA (Attribute Isolation Attention Mask). Using the \"three animals (Lego-style lion, ice-style penguin, flame-style eagle)\" example mentioned in Q3, we show that at layer 20 the cross-entity attention values drop dramatically once AIA is enabled, directly evidencing attribute leakage suppression.\n3. Training-based vs. Training-free AIA. For the Figure 3 column 3 case, layer-39 attention maps reveal that the training-based AIA assigns markedly higher attention scores between the landmark and its water reflection than the training-free variant, evidencing restored global coherence in deeper layers.\n\n> Q6: Different key components improve performance in different aspects and the final determined version did not achieve the best results in most of the indicators. Authors should clarify their reasons and dicuss the balance and interply of different components.\n\nThank you for highlighting the sub-optimal metrics in the final configuration. In the final version, we will incorporate the following explicit justifications for each component:\n- SSFA yields a substantial +4.7% gain in class-agnostic mIoU (shown in Table 4).\n- AIA improves the average of four region-wise metrics by 1.4 % in Table 4 and effectively prevent attribute leakage. We will add the four new cases mentioned in Q3 to make this benefit concrete.  \n- Training-based AIA is essential for recovering global coherence, as evidenced by realistic reflections and shadows.  \n- CITF is retained solely for efficiency. Standard metrics (FID, CLIP, IR, Pick) imperfectly capture perceptual quality. We will provide extra visual results confirming CITF leaves image fidelity almost unchanged."}, {"review_id": "nO30pt0WnJ", "reviewer": "Reviewer_q31Y", "summary": "This paper introduces Seg2Any, a new framework for generating images from segmentation masks that gives users precise control. It tackles a key problem where existing methods either get the object's appearance wrong (semantic inconsistency) or fail to match the provided shape accurately (shape inconsistency). Seg2Any's main idea is to decouple these two problems. It uses the outlines of the masks to guide the shape and special attention mechanisms to ensure each object matches its text description and that attributes like color don't leak between objects. A major contribution is also the creation of a new, large-scale dataset called SACap-1M to support this kind of detailed, open-set generation.", "questions": "The Attribute Isolation Attention Mask is applied only to the middle layers (20-38) of the FLUX model. What is the reasoning behind this specific layer range? How does performance change if you apply it to fewer or more layers?\n\nYou mention that the shape guidance strength γ can be modulated for flexible, scribble-style control. The examples in the paper showcase excellent results with precise masks. Could you provide a visual example of the model's performance with a looser, scribble-style input?\n\nThe paper acknowledges that using a VLM to create the SACap-1M dataset inevitably introduces some annotation noise. Have you done any analysis on the nature or rate of this noise, and how might it be affecting the model's training and final performance?\n\nThe results shown are very impressive. Could you discuss or show some common failure cases for Seg2Any? For example, does it struggle with an extremely high number of objects, very complex overlapping shapes, or abstract text prompts that are hard to visualize?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "**Strengths:**\n\nThe core idea of decoupling shape and semantic control is elegant and addresses a clear failure point in prior work.\n\nCreating and sharing a large, high-quality dataset (SACap-1M) and benchmark is a significant contribution to the community.\n\nThe technical solutions are novel, especially the use of an Entity Contour Map for shape and an Attribute Isolation Attention Mask to prevent visual attribute bleeding.\n\nThe experimental results are very strong and convincing, showing clear improvements over many recent state-of-the-art models in both quantitative metrics and visual examples.\n\n**Weaknesses:**\n\nThe framework is quite complex, with multiple custom components (two attention masks, a new condition type, etc.) that need to work in concert.\n\nThe method's performance on one of the closed-set benchmarks (ADE20K) is strong but still beaten by a prior method (PLACE) on the MIoU metric.\n\nThe new dataset relies on a VLM for automatic captioning, which, as the authors acknowledge, can introduce label noise.", "was_revised": true, "final_justification": "The rebuttal effectively addressed key concerns:\n\n* Complexity clarified as modular with each component addressing distinct failure modes.\n* ADE20K gap explained; fair reimplementation of PLACE shows Seg2Any outperforms under matched settings.\n* Dataset noise acknowledged but shown to have limited impact via qualitative checks and robust results.\n\nSome minor concerns remain (noise quantification, deployment complexity), but the strong novelty, results, and clarifications justify a higher score.", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.\n# Response\n\n## Response to weaknesses\n\n>W1: The framework is quite complex, with multiple custom components (two attention masks, a new condition type, etc.) that need to work in concert.\n\nThank you for the feedback. We respectfully clarify that Seg2Any is concise: it introduces only four lightweight components: Semantic Alignment Attention (SAA), Attribute Isolation Attention (AIA), Sparse Shape Feature Adaptation (SSFA), and Condition Image Token Filtering (CITF). In addition, it requires only two inputs: free-form text and segmentation masks. **Each component serves a distinct, non-redundant purpose**:\n- SAA guarantees that every regional caption binds to its mask without extra trainable encoders.\n- AIA prevents attribute leakage across entities by masking cross-entity self-attention in mid-layers. This is critical when multiple entities share similar textures or colors.\n- SSFA injects high-frequency boundary information via an Entity Contour Map, yielding a pronounced +4.7% absolute gain in class-agnostic MIoU in Table 4.\n- CITF removes zero-value tokens from the Entity Contour Map, thereby shortening the token sequence and accelerating both training and inference, as illustrated in Appendix Figure 1.\n\n>W2: The method's performance on one of the closed-set benchmarks (ADE20K) is strong but still beaten by a prior method (PLACE) on the MIoU metric.\n\nThank you for the comment. The official PLACE training recipe has not been released. Its paper reports using the ADE20K dataset augmented with additional text–image pairs, which may explain the gap. To ensure a fair comparison, we conducted an ablation experiment: we reimplemente only PLACE’s core attention mask under FLUX.1-dev, keeping other hyperparameters fixed: learning rate 1e-4, batch size 16, 20k steps, 4 NVIDIA H100 GPUs, etc. \n\n**Table A: Supplementary experiments reimplement PLACE within the FLUX architecture on COCO-Stuff and ADE20K. Bold represents the\nbest.**\n\n|                                     | COCO Stuff MIoU↑ | COCO Stuff FID↓ | ADE20K MIoU↑ | ADE20K FID↓ |\n| ----------------------------------- | ---------------- | --------------- | ------------ | ----------- |\n| PLACE attention                     | 44.56            | 20.10           | 51.56        | **32.22**   |\n| ours (equal to Table 3 row 3) | **45.54**        | **19.90**       | **54.46**    | 32.89       |\n\n**Table B: Supplementary experiments reimplement PLACE within the FLUX architecture on the SACap-Eval benchmark. Bold represents the\nbest.**\n\n|                                    | class-agnostic MIoU | Spatial Accuracy | Colo Accuracy | Shape Accuracy | Texture Accuracy | IR↑      | Pick↑     | CLIP↑     | FID↓      |\n| ---------------------------------- | ------------------- | --------------------- | ------------------- | ------------------- | --------------------- | -------- | --------- | --------- | --------- |\n| PLACE attention                    | 90.8                | 93.03                 | 89.12               | 86.36               | 88.08                 | **0.51** | **21.90** | **27.93**     | 15.81     |\n| ours (equal to Table 4 row 5)        | **94.90**           | **93.89**             | **91.52**           | **88.15**           | **90.12**             | 0.44     | 21.66     | 27.87     | **15.53**     |\n\n**As shown in above Tables A and B, our model surpasses this PLACE attention variant by +2.9% MIoU on ADE20K and +4.1% MIoU on SACap-Eval, confirming the superiority of our design in a strictly fair comparison.**\n\n\n>W3: The new dataset relies on a VLM for automatic captioning, which, as the authors acknowledge, can introduce label noise.\n\n We provide indirect evidence that the impact is limited.\n1. Qualitative inspection: Appendix Figure 5 shows some hard samples of SACap-1M annotations, where Qwen2-VL-72B produces accurate, fine-grained descriptions.\n2. Downstream robustness: Figure 3 column 1, Figure 4, and Appendix Figure 6 demonstrate that Seg2Any still achieves precise attribute and spatial control, indicating that the model tolerates residual noise.\n\n## Response to questions\n\n>Q1: The Attribute Isolation Attention Mask is applied only to the middle layers (20-38) of the FLUX model. What is the reasoning behind this specific layer range? How does performance change if you apply it to fewer or more layers?\n\nThanks for this insightful question. \nThe choice of layers 20–38 is motivated by DreamRenderer, which identifies these middle layers as primarily responsible for instance-level rendering, whereas shallower (1–19) and deeper layers (39–57) mainly encode global scene context. To verify this, we conducted a systematic case study with varying layer ranges:\n- Extending AIA into layers 1–19 or 39–57 disrupts global image quality, yielding noticeable artifacts.\n- Restricting AIA to only layers 20–29 or 30–38 markedly weakens attribute isolation.\n\nWe will include these ablations in the final version.\n\n>Q2: You mention that the shape guidance strength γ can be modulated for flexible, scribble-style control. The examples in the paper showcase excellent results with precise masks. Could you provide a visual example of the model's performance with a looser, scribble-style input?\n\nIn fact, Appendix Figure 2 (bottom row) already presents qualitative results obtained with $\\gamma = 0.2$, where the input consists of coarse, scribble-style strokes rather than precise masks. Under the scribble-style input, Seg2Any still generates entities that respect the rough spatial layout while allowing natural shape variations, confirming the flexibility of our modulation strategy.\n\n>Q3: The paper acknowledges that using a VLM to create the SACap-1M dataset inevitably introduces some annotation noise. Have you done any analysis on the nature or rate of this noise, and how might it be affecting the model's training and final performance?\n\nWe appreciate the reviewer's concern regarding annotation noise. Quantitatively estimating the noise rate across 5.9 M region captions is non-trivial. Instead, we provide indirect evidence that the impact is limited.\n1. Qualitative inspection: Appendix Figure 5 shows some hard samples of SACap-1M annotations, where Qwen2-VL-72B produces accurate, fine-grained descriptions.\n2. Downstream robustness: Figure 3 column 1, Figure 4, and Appendix Figure 6 demonstrate that Seg2Any still achieves precise attribute and spatial control, indicating that the model tolerates residual noise.\n\n\n>Q4: The results shown are very impressive. Could you discuss or show some common failure cases for Seg2Any? For example, does it struggle with an extremely high number of objects, very complex overlapping shapes, or abstract text prompts that are hard to visualize?\n\nThanks for prompting a deeper discussion of failure cases.\nDue to the NeurIPS 2025 rebuttal policy, we cannot include new visuals here. All examples below will appear in the final version.\n1. Extreme object density.\nExtensive tests with 19 overlapping fruits across nine categories, 12 cars of distinct colors, and 5 heavily occluded football players demonstrate Seg2Any’s strong robustness under high overlap. However, pushing the limit further, we evaluate a crowded cinema scene containing roughly 50 people and observe that the model begins to merge adjacent masks, thereby revealing a practical upper bound on the number of instances.\n2. Highly abstract prompts.\nFor abstract prompts like \"a fantastical world split into three distinct zones: desert, ice, and volcano.\" Seg2Any yields unnatural transitions at mask seams, likely because SACap-1M contains few surreal, zone-abrupt scenes."}, {"review_id": "RApo9Hs9F7", "reviewer": "Reviewer_T7Yi", "summary": "Seg2Any tackles open-set mask-to-image generation by disentangling shape and semantics: a Semantic Alignment Attention Mask precisely links each text prompt to its corresponding mask region, while an Entity Contour Map provides high-fidelity shape guidance via sparse condition tokens. An Attribute Isolation Attention Mask further prevents attribute leakage among multiple instances, and a lightweight Condition Image Token Filtering step drops zero-information patches to cut compute without harming quality. Trained on the newly released SACap-1M dataset (1 M images with 5.9 M mask-level captions) and evaluated on the SACap-Eval benchmark plus COCO-Stuff/ADE20K, Seg2Any sets or matches state-of-the-art results in both open- and closed-set settings, achieving superior shape consistency and attribute control while remaining efficient.", "questions": "Please see the weaknesses.", "limitations": "yes", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 4, "strengths_and_weaknesses": "### Strengths\n\n- The qualitative results are impressive, and Seg2Any performs on par with other strong baselines.\n\n- The proposed SACap-1M dataset and SACap-Eval benchmark will be valuable resources for the community.\n\n- The open-set mask-to-image generation setting is interesting.\n\n### Weaknesses\n\n- Both the Semantic Alignment Attention Mask and the Attribute Isolation Attention Mask resemble existing techniques for layout-controllable generation [1, 2]. How does your Attribute Isolation Attention Mask differ from MIGC’s [3] attention-mask design?\n\n- In the introduction section, the authors claim that the $16\\times$ downsampling ratio amplified the loss of spatial information. My question is, how do authors mitigate this problem? In my opinion, the attention mask and condition tokens encoded by VAE are still sparse and lack spatial detail.\n\n- The paper lacks qualitative ablation results for CITF. Because automatic metrics may not fully reflect human perception, I recommend adding visual comparisons to illustrate CITF’s impact.\n\n- Could authors further clarify the practical value of the mask-to-image generation task? In real-world applications, semantic masks can be difficult for users to obtain.\n\n[1] MultiBooth: Towards Generating All Your  Concepts in an Image from Text\n[2] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance\n[3] MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis", "was_revised": true, "final_justification": "The authors have fully addressed all of my concerns in their rebuttal, and I am satisfied with their clarifications. While some aspects could be further strengthened in future work, the current submission meets the standards for acceptance.\n\nScore: 4 Borderline Accept", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.\n\n# Response\n## Response to weaknesses\n\n>W1: Both the Semantic Alignment Attention Mask and the Attribute Isolation Attention Mask resemble existing techniques for layout-controllable generation [1, 2]. How does your Attribute Isolation Attention Mask differ from MIGC’s [3] attention-mask design?\n\nThe key distinction lies in the architectural target: whereas MIGC [3] and prior works [1,2] devise masks for U-Net’s cross-attention and self-attention, our Semantic Alignment Attention Mask (SAA) and Attribute Isolation Attention Mask (AIA) are designed for the multi-modal attention in the DiT framework.\nMoreover, MIGC applies two attention masks (instance-intra and instance-inter) within each layer. In contrast, Seg2Any assigns distinct attention masks to different layers.  Our attention mask strategy aligns with FLUX’s internal hierarchy: layers 20–38 specialize in instance-level rendering, so AIA is applied only there, while layers 1–19 and 39–57, which encode global scene context, employ the SAA to preserve overall coherence.\n\n>W2: In the introduction section, the authors claim that the \n downsampling ratio amplified the loss of spatial information. My question is, how do authors mitigate this problem? In my opinion, the attention mask and condition tokens encoded by VAE are still sparse and lack spatial detail.\n\nIn fact, we address the loss of spatial information by introducing the Entity Contour Map as condition tokens.\nThe attention mask is produced by a 16× nearest-neighbour downsample of the input segmentation mask—an irreversible, lossy compression that discards spatial detail.\nOur remedy is to introduce the Entity Contour Map as an additional condition token. Leveraging the VAE’s high-fidelity, near-lossless compression, the Entity Contour Map preserves fine spatial details. \n**As evidenced in Table 4 (row 1 vs. 2) and Table 3 (row 1 vs. 2), the injection of the Entity Contour Map yields a clear MIoU improvement.**\n\n>W3: The paper lacks qualitative ablation results for CITF. Because automatic metrics may not fully reflect human perception, I recommend adding visual comparisons to illustrate CITF’s impact.\n\nThank you for highlighting this point. Owing to the NeurIPS 2025 rebuttal policy, we are unable to provide new visuals during the rebuttal phase. We will include qualitative ablations comparing results with and without Condition Image Token Filtering (CITF) in the final version to illustrate its perceptual impact.\n\n>W4: Could authors further clarify the practical value of the mask-to-image generation task? In real-world applications, semantic masks can be difficult for users to obtain.\n\nThank you for raising this important practical concern. \nFirst, precise spatial control makes Seg2Any well-suited for design workflows such as interior or furniture layout, where users can sketch and instantly visualize the result. Second, our pipeline can automatically create large, labelled datasets for downstream segmentation tasks, drastically reducing manual annotation costs when adapting detectors to new domains or rare object categories.\n\n[1] MultiBooth: Towards Generating All Your Concepts in an Image from Text\n\n[2] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance\n\n[3] MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis"}, {"review_id": "1dMspeYLGS", "reviewer": "Reviewer_Q4gs", "summary": "This paper proposes a segmentation mask-to-image framework that enables precise control over shape and semantics while preventing attribute leakage in open-set S2I generation, including the injection of semantic-shape decoupled layout conditions and attribute isolation via an image self-attention mask. A large-scale open-set dataset, along with a corresponding benchmark, is proposed.", "questions": "1. I am curious about how the model would perform for entities of different sizes. As the entity contour map is down-sampled, how can we ensure the distinction between dense entities?\n\n2. How does performance change if the model is initialized using Stable Diffusion[1]?\n\n[1] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.", "limitations": "yes", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n\n1. The method is reasonable overall.\n2. The proposed large-scale open-set dataset is beneficial to the community.\n\nWeaknesses:\n\n1. In Tab. 1, Seg2Any outperforms previous methods significantly, but in Tab. 2, its performance on generation metrics (FID) is poor, and it also does not perform well on COCO MIoU. This raises concerns about the effectiveness of the method as well as the validity of the benchmark itself. Authors are encouraged to compare under fair settings. For example, how will the results of Tab.1 and Tab.2 change when other baseline methods, especially Freestylenet and PLACE, are initialized by flux? \n\n2. The results of the ablation experiment are confusing. Comparing the results of the first row in Tab. 4 with those in other rows, it appears that some designs negatively impact the final performance.", "was_revised": true, "final_justification": "My main concern is that the authors propose a generative approach but perform worse than baseline methods on nearly all generative metrics. I believe that the weaknesses in multiple metrics cannot be simply explained by the misalignment between FID and human preferences. Therefore, i insist on my init rating.", "rebuttal": "We sincerely thank the reviewer for the insightful and constructive feedback. Our detailed responses are provided below.\n\n# Response\n## Response to weaknesses\n>W1: In Tab. 1, Seg2Any outperforms previous methods significantly, but in Tab. 2, its performance on generation metrics (FID) is poor, and it also does not perform well on COCO MIoU. This raises concerns about the effectiveness of the method as well as the validity of the benchmark itself. Authors are encouraged to compare under fair settings. For example, how will the results of Tab.1 and Tab.2 change when other baseline methods, especially FreestyleNet and PLACE, are initialized by flux?\n\nThanks for raising this insightful question.\n\n**On the relevance of FID**. Recent work[2] has shown that FID correlates weakly with human perceptual preference. FID provides a reference for quality evaluation to a certain extent, but it is not necessarily the best evaluation indicator.\n\n**Fair comparison under FLUX**. As the official PLACE training recipe has not been released, we cannot reproduce its auxiliary losses and its additional training data. To ensure a fair comparison, we reimplement only the essential PLACE attention mask under the FLUX.1 dev architecture while keeping all other training details identical to ours: learning rate 1e-4, batch size 16, 20k steps, 4 NVIDIA H100 GPUs, etc.\nRegarding FreestyleNet under FLUX initialization, our Semantic Alignment Attention Mask (SAA) is the direct implementation of FreestyleNet’s attention mask within the MM-DiT framework. The corresponding metrics are already reported in Table 3 row 1 and Table 4 row 1.\n\n**Table A: Supplementary experiments of FreestyleNet and PLACE with flux initialization on COCO Stuff and ADE20K. Bold represent the\nbest.**\n\n|                               | COCO Stuff MIoU↑ | COCO Stuff FID↓ | ADE20K MIoU↑ | ADE20K FID↓ |\n| ----------------------------- | ---------------- | --------------- | ------------ | ----------- |\n| SAA(=FreestyleNet attention)  | 43.48            | 20.57           | 44.85        | 33.14       |\n| PLACE attention               | 44.56            | 20.10           | 51.56        | **32.22**   |\n| ours (equal to Table 3 row 3) | **45.54**        | **19.90**       | **54.46**    | 32.89       |\n\n**Table B: Supplementary experiments of FreestyleNet and PLACE with flux initialization on the SACap-Eval benchmark. Bold represents the\nbest.**\n\n|                               | class-agnostic MIoU | Spatial Accuracy | Colo Accuracy | Shape Accuracy | Texture Accuracy | IR↑      | Pick↑     | CLIP↑     | FID↓      |\n| ----------------------------- | ------------------- | ---------------- | ------------- | -------------- | ---------------- | -------- | --------- | --------- | --------- |\n| SAA(=FreestyleNet attention)  | 89.39               | 93.36            | 89.26         | 85.84          | 87.86            | 0.47     | 21.83     | **28.09** | **15.30** |\n| PLACE Attention               | 90.80                | 93.03            | 89.12         | 86.36          | 88.08            | **0.51** | **21.90** | 27.93     | 15.81     |\n| ours (equal to Table 4 row 5) | **94.90**           | **93.89**        | **91.52**     | **88.15**      | **90.12**        | 0.44     | 21.66     | 27.87     | 15.53     |\n\n**As the above Tables A and B show, under identical FLUX initialization and training settings, our model outperforms PLACE attention by 2.9% MIoU on ADE20K and by 4.1% MIoU on SACap-Eval, confirming the superiority of our design in a strictly fair comparison.**\n\n\n>W2: The results of the ablation experiment are confusing. Comparing the results of the first row in Tab. 4 with those in other rows, it appears that some designs negatively impact the final performance.\n\nThank you for highlighting the sub-optimal metrics in the final configuration. In the final version, we will incorporate the following explicit justifications for each component:\n- SSFA is essential because it delivers a remarkable +4.7% boost in class-agnostic mIoU in Table 4, directly addressing our primary objective of precise shape preservation.\n- AIA improves the average of four region-wise metrics by 1.4% in Table 4 and effectively prevents attribute leakage. We will add new visual results in the final version to make this benefit concrete.\n- Training-based AIA is essential for recovering global coherence, as evidenced by realistic reflections and shadows.  \n- CITF is retained solely for efficiency. Standard metrics (FID, CLIP, IR, and Pick) imperfectly capture perceptual quality. We will provide extra visual results in the final version confirming CITF leaves image fidelity almost unchanged.\n\n## Response to questions\n\n>Q1: I am curious about how the model would perform for entities of different sizes. As the entity contour map is down-sampled, how can we ensure the distinction between dense entities?\n\nThank you for this valuable question. As illustrated in Figure 1 row 2, Seg2Any preserves the fine vertical bars on the railing, demonstrating robustness to small-scale objects. Additionally, Appendix Figure 6 (last row column 4) shows that our method performs equally well in dense scenes.\nDue to the NeurIPS 2025 rebuttal policy, we cannot include new visuals here. In the final version, we will include additional examples such as around 20 ceiling lights smaller than 64×64 pixels and densely overlapping scenes (e.g., 19 fruits across nine categories, 12 cars of distinct colors, and five heavily occluded football players), where our method continues to yield accurate boundaries. We acknowledge that extreme density can still induce object merging. Therefore, we will also present failure cases to transparently report these limitations.\n\n>Q2: How does performance change if the model is initialized using Stable Diffusion[1]?\n\nThank you for this question.\nHowever, Seg2Any is specially designed for MM-DiT:\n- Seg2Any assigns distinct attention masks to different layers: Attribute Isolation Attention Mask (AIA) on FLUX layers 20–38 for instance rendering,  Semantic Alignment Attention Mask (SAA) elsewhere for global coherence. Such layer-wise scheduling is absent in Stable Diffusion (SD).\n- Condition Image Token Filtering (CITF) discards negligible condition tokens via token-level pruning within DiT’s unified sequence. However, SD’s separate encoder blocks preclude this filtering.\n\nOwing to the tight rebuttal schedule, adapting Seg2Any to Stable Diffusion would demand a complete redesign of the attention mask schedules and Entity Contour Map injection pipeline, we will complete this ablation during the discussion period. \nThe planned setup employs SD 1.5, injects the entity contour map via a ControlNet branch, and applies attention mask strategy to the U-Net cross-attention layers. We will benchmark this setup on COCO-Stuff and ADE20K.\nPlease let us know whether this setup meets your intent during the discussion period, and we will run the experiments immediately upon confirmation.\n\n[1] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.\n\n[2] Rethinking FID: Towards a Better Evaluation Metric for Image Generation. CVPR, 2024"}]

Author rebuttal:

Meta-review: The paper proposes Seg2Any, a segmentation-mask-to-image (S2I) generation framework that decouples semantic and shape consistency using a Semantic Alignment Attention Mask (SAA), an Entity Contour Map (ECM), and an Attribute Isolation Attention Mask (AIA). The authors also introduce a new dataset (SACap-1M) and evaluation benchmark (SACap-Eval). Reviewers acknowledge the novelty of the technical contributions, the value of the dataset, and strong results on both open- and closed-set benchmarks.

Strengths highlighted across reviews:

+ Novel methodological contributions: decoupling shape and semantic control via ECM and AIA to prevent attribute leakage (tBGK, q31Y, T7Yi).
+ Creation of SACap-1M and SACap-Eval, a large-scale dataset and benchmark that will benefit the community (q31Y, T7Yi, Q4gs).
+ Strong experimental performance and compelling qualitative results, showing fine-grained spatial and attribute control (q31Y, T7Yi).
+ Clear motivation and well-written paper with abundant technical detail (tBGK, q31Y).

Weaknesses and concerns:

- Ablation studies viewed as inconclusive or unconvincing, with missing visualizations to clearly support claims about AIA and CITF (tBGK).
- Complexity of the framework with multiple custom components, raising questions about necessity and interpretability (q31Y).
- Performance gaps on certain benchmarks, e.g. ADE20K MIoU lower than PLACE; weaker generative metrics such as FID, CLIP, PICK, raised doubts about general quality (q31Y, Q4gs).
- Concerns about dataset noise due to VLM-generated captions (q31Y).
- Comparisons to related techniques (MIGC, MultiBooth, MS-Diffusion) not fully elaborated, leaving unclear what is fundamentally new (T7Yi).

Rebuttal and Discussion:
The authors clarified that Seg2Any is modular, with each component addressing a distinct failure mode, and provided quantitative evidence for each. They reimplemented PLACE fairly under identical FLUX settings, showing Seg2Any’s superiority (+2.9% MIoU ADE20K, +4.1% SACap-Eval), which resolved concerns about ADE20K performance. They explained that annotation noise in SACap-1M has limited effect, supported by qualitative evidence. They also promised to include missing attention visualizations and additional ablations in the camera-ready. Reviewers q31Y and T7Yi explicitly raised their scores after rebuttal, finding concerns addressed. Reviewer tBGK acknowledged that responses were thorough but lacking visualizations due to policy restrictions, maintained a borderline score. Reviewer Q4gs remained critical, emphasizing weaker global metrics (FID, IR, CLIP, PICK) and expressing skepticism despite clarifications.

Recommendation:
Overall, three reviewers (q31Y, T7Yi, tBGK) are inclined to accept, noting novelty, strong results, and valuable dataset contributions. One reviewer (Q4gs) remains unconvinced due to concerns about generative quality metrics. Given the technical innovations, extensive evaluation, clear community contribution of a dataset and benchmark, and convincing rebuttal addressing most substantive issues, I recommend accept.
Decision: Accept (poster)