Title: Seg2Any: Open-set Segmentation-Mask-to-Image Generation with Precise Shape and Semantic Control
Venue: NeurIPS 2025
Accepted: yes
Keywords: controllable image generation

text to image generation
Abstract: Despite recent advances in diffusion models, top-tier text-to-image (T2I) models still struggle to achieve precise spatial layout control, *i.e.* accurately generating entities with specified attributes and locations. Segmentation-mask-to-image (S2I) generation has emerged as a promising solution by incorporating pixel-level spatial guidance and regional text prompts. However, existing S2I methods fail to simultaneously ensure semantic consistency and shape consistency.
To address these challenges, we propose Seg2Any, a novel S2I framework built upon advanced multimodal diffusion transformers (*e.g.* FLUX). First, to achieve both semantic and shape consistency, we decouple segmentation mask conditions into regional semantic and high-frequency shape components. The regional semantic condition is introduced by a Semantic Alignment Attention Mask, ensuring that generated entities adhere to their assigned text prompts. The high-frequency shape condition, representing entity boundaries, is encoded as an Entity Contour Map and then introduced as an additional modality via multi-modal attention to guide image spatial structure. Second, to prevent attribute leakage across entities in multi-entity scenarios, we introduce an Attribute Isolation Attention Mask mechanism, which constrains each entity’s image tokens to attend exclusively to themselves during image self-attention.
To support open-set S2I generation, we construct SACap-1M, a large-scale dataset containing 1 million images with 5.9 million segmented entities and detailed regional captions, along with a SACap-Eval benchmark for comprehensive S2I evaluation.
Extensive experiments demonstrate that Seg2Any achieves state-of-the-art performance on both open-set and closed-set S2I benchmarks, particularly in fine-grained spatial and attribute control of entities.


Peer reviews:
comments: SUMMARY:
This paper proposes a segmentation mask-to-image framework that enables precise control over shape and semantics while preventing attribute leakage in open-set S2I generation, including the injection of semantic-shape decoupled layout conditions and attribute isolation via an image self-attention mask. A large-scale open-set dataset, along with a corresponding benchmark, is proposed.

STRENGTHS AND WEAKNESSES:
Strengths:

1. The method is reasonable overall.
2. The proposed large-scale open-set dataset is beneficial to the community.

Weaknesses:

1. In Tab. 1, Seg2Any outperforms previous methods significantly, but in Tab. 2, its performance on generation metrics (FID) is poor, and it also does not perform well on COCO MIoU. This raises concerns about the effectiveness of the method as well as the validity of the benchmark itself. Authors are encouraged to compare under fair settings. For example, how will the results of Tab.1 and Tab.2 change when other baseline methods, especially Freestylenet and PLACE, are initialized by flux? 

2. The results of the ablation experiment are confusing. Comparing the results of the first row in Tab. 4 with those in other rows, it appears that some designs negatively impact the final performance.

QUESTIONS:
1. I am curious about how the model would perform for entities of different sizes. As the entity contour map is down-sampled, how can we ensure the distinction between dense entities?

2. How does performance change if the model is initialized using Stable Diffusion[1]?

[1] High-Resolution Image Synthesis with Latent Diffusion Models. CVPR, 2022.

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-16

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSeg2Any tackles open-set mask-to-image generation by disentangling shape and semantics: a Semantic Alignment Attention Mask precisely links each text prompt to its corresponding mask region, while an Entity Contour Map provides high-fidelity shape guidance via sparse condition tokens. An Attribute Isolation Attention Mask further prevents attribute leakage among multiple instances, and a lightweight Condition Image Token Filtering step drops zero-information patches to cut compute without harming quality. Trained on the newly released SACap-1M dataset (1 M images with 5.9 M mask-level captions) and evaluated on the SACap-Eval benchmark plus COCO-Stuff/ADE20K, Seg2Any sets or matches state-of-the-art results in both open- and closed-set settings, achieving superior shape consistency and attribute control while remaining efficient.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- The qualitative results are impressive, and Seg2Any performs on par with other strong baselines.\n\n- The proposed SACap-1M dataset and SACap-Eval benchmark will be valuable resources for the community.\n\n- The open-set mask-to-image generation setting is interesting.\n\n### Weaknesses\n\n- Both the Semantic Alignment Attention Mask and the Attribute Isolation Attention Mask resemble existing techniques for layout-controllable generation [1, 2]. How does your Attribute Isolation Attention Mask differ from MIGC’s [3] attention-mask design?\n\n- In the introduction section, the authors claim that the $16\\times$ downsampling ratio amplified the loss of spatial information. My question is, how do authors mitigate this problem? In my opinion, the attention mask and condition tokens encoded by VAE are still sparse and lack spatial detail.\n\n- The paper lacks qualitative ablation results for CITF. Because automatic metrics may not fully reflect human perception, I recommend adding visual comparisons to illustrate CITF’s impact.\n\n- Could authors further clarify the practical value of the mask-to-image generation task? In real-world applications, semantic masks can be difficult for users to obtain.\n\n[1] MultiBooth: Towards Generating All Your  Concepts in an Image from Text\n[2] MS-Diffusion: Multi-subject Zero-shot Image Personalization with Layout Guidance\n[3] MIGC: Multi-Instance Generation Controller for Text-to-Image Synthesis\n\nQUESTIONS:\nPlease see the weaknesses.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Seg2Any, a new framework for generating images from segmentation masks that gives users precise control. It tackles a key problem where existing methods either get the object's appearance wrong (semantic inconsistency) or fail to match the provided shape accurately (shape inconsistency). Seg2Any's main idea is to decouple these two problems. It uses the outlines of the masks to guide the shape and special attention mechanisms to ensure each object matches its text description and that attributes like color don't leak between objects. A major contribution is also the creation of a new, large-scale dataset called SACap-1M to support this kind of detailed, open-set generation.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe core idea of decoupling shape and semantic control is elegant and addresses a clear failure point in prior work.\n\nCreating and sharing a large, high-quality dataset (SACap-1M) and benchmark is a significant contribution to the community.\n\nThe technical solutions are novel, especially the use of an Entity Contour Map for shape and an Attribute Isolation Attention Mask to prevent visual attribute bleeding.\n\nThe experimental results are very strong and convincing, showing clear improvements over many recent state-of-the-art models in both quantitative metrics and visual examples.\n\n**Weaknesses:**\n\nThe framework is quite complex, with multiple custom components (two attention masks, a new condition type, etc.) that need to work in concert.\n\nThe method's performance on one of the closed-set benchmarks (ADE20K) is strong but still beaten by a prior method (PLACE) on the MIoU metric.\n\nThe new dataset relies on a VLM for automatic captioning, which, as the authors acknowledge, can introduce label noise.\n\nQUESTIONS:\nThe Attribute Isolation Attention Mask is applied only to the middle layers (20-38) of the FLUX model. What is the reasoning behind this specific layer range? How does performance change if you apply it to fewer or more layers?\n\nYou mention that the shape guidance strength γ can be modulated for flexible, scribble-style control. The examples in the paper showcase excellent results with precise masks. Could you provide a visual example of the model's performance with a looser, scribble-style input?\n\nThe paper acknowledges that using a VLM to create the SACap-1M dataset inevitably introduces some annotation noise. Have you done any analysis on the nature or rate of this noise, and how might it be affecting the model's training and final performance?\n\nThe results shown are very impressive. Could you discuss or show some common failure cases for Seg2Any? For example, does it struggle with an extremely high number of objects, very complex overlapping shapes, or abstract text prompts that are hard to visualize?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper Seg2Any framework to achieve open-set semantic-to-image generation with improved semantic consistency and shape consistency. It proposes Semantic-Shape Decoupled Layout Conditions Injection and Attribute Isolation Attention Mask to improve model performance. In addition, it builds a large-scale datasets SACap-1M with images, open-set segmentation mask annotations, image-level captions and regional captions.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. Good writing;\n2. High novelty\n3. Abundant content\n\nWeakness:\n1. Relatively poor persuasion of ablation study\n2. Lacking visualization of ablation study\n\nQUESTIONS:\n1. According to the Ablation study in Table 3 and Table 4, the Condition Image Token Filtering (CITF) seems to only sightly improve the class-agnostic MIoU while negatively affecting other metrics. Statistically, the overall contribution of CITF is negative, but authors claimed \"Table 4 ... Condition Image Token Filtering (CITF) leaves performance unchanged\" and \"CITF does not affect performance (see Table 3)\". More explainations are needed here.\n2. Figure 3 column 4 does not support the statement in caption: \"direct application of the mask without training leads to visual inconsistencies, manifesting as unnatural shadows and reflections\". Shadows of the plane in Training-free AIA and Training based AIA results are both unatural, even worse than W/O AIA.\n3. What is the definition of 'attribute' in the term 'attribute leakage'? Authors claims that failures in Figure 3 Row 1 are caused by 'attribute leakage', but it seems that the biggest disadvantage of W/O AIA w.r.t Training-based AIA is the text generation.  One of the  most impressive advanatages of Seg2Any in Figure 4 is also good text generation. However, all these experiments can not prove the contribution of AIA, because it is entirely possible that this improvement comes from other modules. Stronger evidence is needed.\n4.  The AIA prevents one entity token attending to other entity tokens, background tokens and global text token. Wouldn't this strategy undermine the global perception and long-term dependences among entities?\n5. Authors should provide attentive map or visualize some results in ablation study section. In particular, it is better to show how the results of attentive map changes with respect to different configurations of key components.\n6. Different key components improve performance in different aspects and the final determined version did not achieve the best results in most of the indicators. Authors should clarify their reasons and dicuss the balance and interply of different components.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}