FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation
Authors: Fan Yang
Yousong Zhu
Xin Li
Yufei Zhan
Hongyin Zhao
Shurong Zheng
Yaowei Wang
Ming Tang
Jinqiao Wang
Venue: NeurIPS 2025 poster ()
Keywords: Large Vision Language Model; Genenration Edit;
Primary area: deep_learning

Abstract: Recent Large Vision Language Models (LVLMs) demonstrate promising capabilities in unifying visual understanding and generative modeling, enabling both accurate content understanding and flexible editing. However, current approaches treat \textbf{\textit{"what to see"}} and \textbf{\textit{"how to edit"}} separately: they either perform isolated object segmentation or utilize segmentation masks merely as conditional prompts for local edit generation tasks, often relying on multiple disjointed models. To bridge these gaps, we introduce FOCUS, a unified LVLM that integrates segmentation-aware perception and controllable object-centric generation within an end-to-end framework. FOCUS employs a dual-branch visual encoder to simultaneously capture global semantic context and fine-grained spatial details. In addition, we leverage a MoVQGAN-based visual tokenizer to produce discrete visual tokens that enhance generation quality. To enable accurate and controllable image editing, we propose a progressive multi-stage training pipeline, where segmentation masks are jointly optimized and used as spatial condition prompts to guide the diffusion decoder. This strategy aligns visual encoding, segmentation, and generation modules, effectively bridging segmentation-aware perception with fine-grained visual synthesis.
Extensive experiments across three core tasks, including multimodal understanding, referring segmentation accuracy, and controllable image generation, demonstrate that FOCUS achieves strong performance by jointly optimizing visual perception and generative capabilities.

FOCUS: Unified Vision-Language Modeling for Interactive Editing Driven by Referential Segmentation
Fan Yangl.23 Yousong Zhu"t, Xin Li,, Yufei Zhan Hongyin Zhao Shurong Zheng 1.2.3 Yaowei Wang", Ming Tang! Jinqiao 12355
Foundation Model Research Center, Institute Automation, Chinese Academy of Sciences, Haidian District, Beijing, China
Peng Cheng Laboratory, Shenzhen, China
'School Artificial Intelligence, University of Chinese Academy of Science, Beijing, China
"School of Artificial Intelligence, China University of Mining and Technology-Bejingg Beijing, China WWuhan AI Research, Wuhan, China
(yangfan 2022, zhanyufei2021, zhengshurongg2023, zhachengyin2020188 aaa.al222o0aaaa....ooo.ommm meiia.accom ac.cn
(tangm, jqwang) @nlpr. ia. en, zhuyousong@cuntb edu. en, [lix07, wangyw @pcl
Abstract
Recent Large Vision Language Models (LVLMs) demonstrate promising capa- bilities in unifying visual understanding and generative modeling, enabling both accurate content understanding and flexible editing. However, current approaches treat "what see" and "how edit" separately: they either perform isolated ob- ject segmentation utilize segmentation masks merely conditional prompts for local edit generation tasks, often relying on multiple disjointed models. To bridge these gaps, we introduce FOCUS, unified LVLM that integrates segmentation- aware perception and controllable object-centric generation within an end-to-end framework. FOCUS employs a dual-branch visual encoder simultancously capture global semantic context and fine -eeeggrinnd spatial details. In addition, we leverage Mo JQGAA--base visual tokenizer to produce discrete visual tokens that enhance generation quality. To enable accurate and controllable image editing, propose a progressive multi-stage training pipeline, where segmentation masks are jointly optimized and used as spatial condition prompts to guide the diffusion decoder. This strategy aligns visual encoding, segmentation, and generation mod- ules, effectively bridging segmentation-aware perception with fine-grained visual synthesis. Extensive experiments across three core tasks, including multimodal understanding, referring segmentation accuracy, and controllable image generation, demonstrate that FOCUS achieves strong performance by jointly optimizing visual perception and generative capabilities.
Introduction
I Large Vision-I LanguaLa Models (LVLMs) are becoming a transformative paradigm in artificial intelligence 45| 38 Through large-scale pretraining, they unify visual and textual modalities
and have achieved remarkable progress across a variety of tasks. These models can jointly process
the corresponding authors
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Image Editing
Mesh
Editing
Original Image / : perforn
(al Segment the two skeas from the 661
(b) Change the snowy noustain descrt.
(c Remove the person on the lett.
(d) Replace the pasam on the left with berrnee.
Questien:
FOCUS
Video Editing
Mask
Editing
Original Video
Change the gollery
Question: Segrient the stan white ral black sut and pollere
Remave
Replace with
FOCES
Figure I: FOCUS enables fine-grained segmentation and editing for both images and videos through multimodal user instructions. It supports various region specification formats including clicks, scribbles, buxes, and masks, and for videos, annotations on any single frame suffice to guide full- clip editing. FOCUS can perform detailed region segmentation (e.g., identifying individual people), and supports diverse editing operations such as removal, replacement, and scene transformation across spatial and temporal domains.
images, videos, and natural language within a single architecture, demonstrating strong performance in visual question answering. image captioning. referring segmentation, and conditional generation 38 (1). The effectiveness of LVLMs largely stems from their ability to align multimodal semantics and generalize across diverse tasks with minimal task-specific adaptation.
33 75
50
At the same time, image and video generation technologics [47 7 46 85] have advanced rapidly, enabling the synthesis of high-quality visual content and reshaping how approach artistic creation and visual communication. These developments have driven increasing demands for controllability and fidelity across domains such art, industry, and education [19]. To meet these demands, some approaches [60]
[0] as is shown in fig. 2 attempt to combine image generation models with segmentation decoders and text processing modules through modular design. However, such methods often rely on manually engineered pipelines and lack unified modeling and deep feature- level interaction. To address these limitations, others method (30 in fig. 2 leverage the instruction understanding capabilities of LVLMs to dynamically dispatch pretrained expert models through task routing, enabling flexible multi-task adaptation. While these approaches improve tool orchestration, they lack deep rross-modal fusion and joint feature optimization, making it difficult to unify perception and generation effectively. More recent methods (55 (56) as is show in fig. ( have begun to construct unified and generation, using the generalization capability of LVLMs to bridge the semantic gap between high-level understanding and low-level synthesis. However, most of these methods still operate coarse level of text-driven control and struggle to support fine-graine tne-grained editing or object-level manipulation.
[
frameworks that combine image understanding
To overcome these challenges, propose FOCUS, end-to-end LVLM framework that unifies segmentation-aware perception and region-controllable generation under natural language guidance, as is shown in fig. []] The core of FOCUS features a dual-branch visual encoder, where a CLiP-like or QwenViT-style ViT-style encoder extracts global semantic representations, while a hierarchical encoder based on ConvNeXt-L focuses on fine-grained local perception. This structure provides stable multi-scale segmentation support and improves adaptability to varying image resolutions. To improve visual generation, FOCUS adopts VQGAN- based visual tokenizer, inspired by 62 57 21), which separately models semantic concepts and texture information. To mitigate the inevitable information loss during quantization, we retain the continuous pre-quantization features from the tokenizer as visual inputs to the language model, enabling fine-grained multimodal understanding. FOCUS trained through a progressive multi-stage strategy, gradually increasing the input and output resolutions from low to high to ensure stable convergence and final performance. Segmentation

---

Image Segmentor
Language Processor
Large Vision Language Model
Large Vision Language Mudel
Large Vision Language Modall
Image Editor
Image Seymentor
Image
Unitied etokenizzrr Diffusion: Decodee
Diftusion Docuder
Editor
Decoder
Image
image
Edited Image
Mask
Edittsd Image
(4) ug. InstructEdit
(b) UnffiedMLLLM
EmuZ, ILLUME+
(d) eg. FOCUS(OUR
Figure Comparison of controllable image editing paradigms. (a) Modular methods rely on
separately trained components. (b) Task-routing LLMs orchestrate existing tools without joint modeling. (c) Unified models combine perception and generation but lack fine-grained control. (d) Our FOCUS jointly models segmentation and generation for precise, region-level editing.
masks are jointly optimized and used spatial condition prompts guide diffusion-based generator for pixel-level editing. During both the multimodal pretraining and instruction tuning stages, we introduce diverse and increasingly complex task distributions, with carefully designed instruction formats, to fully enhance the model's perception understanding and generation capabilities.
Experimental results show that FOCUS achieves significant improvements across three core tasks: multimodal understanding, controllable image generation and editing, and referring segmentation. By jointly modeling pixel-level perception and generation, FOCUS demonstrates higher semantic precision and stronger spatial controllability. Further analysis reveals that pixel-level perception plays crucial role in bridging the gap between high-level understanding and low-level synthesis. These findings confirm the feasibility and effectiveness of unifying segmentation-awan perception and controllable generation within single LVLM framework, paving the way for interactive, precise, and generalizable multimodal editing systems.
We propose FOCUS, unified large vision language model that integrates pixel-level perception with region-controllabbe image generation and editing within an end-to-end framework.
We develop progressive multi-stage training pipeline that gradually increases image resolution and task complexity. This pipeline enables effective alignment and interaction between the segmentation decoder and the generation module across different scales and modalities.
Extensive experiments on multimodal understanding, referential segmentation, and control- lable image editing demonstrate that FOCUS consistently outperforms existing state-of-the- art models both visual understanding and generation quality.
2 Related work
The landscape of large vision-language models (LVMs) has rapidly evolved, aiming to unify multi- modal understanding and generation across images, videos, and texts. representative example the Emu series, which introduces autoregressive modeling to predict the next visual or textual token, thus enabling a generalist interface for diverse multimodal tasks such as image captioning, video event understanding, and cross-modal generation. This contrasts with earlier LVM designs (c.g., BLIP 271, Flamingo (T] that bridge frozen vision and language models using separate modality connectors, often focusing solely text prediction and neglecting direct supervision over visual signals. Emu2 (56) further extends this paradigm by positioning itself as an in-context learner, exploiting interleaved video-text data deliver fine-grained temporal and causal reasoning multimodal inputs.
While many of these works achieve remarkable integration of multimodal comprehension and generation, they often fall short interactive and controllable editing. For instance, the LLaVA- Interactive [|] system explores multi-turn, multimodal interactions combining visual chat, seg-

---

FOCUS Frameworks Visual Input
Visual Encoder
Xw geNANia
Hierarchical Encoder
Vanilla Encoder
Pixel Encoder
Key Value
Query
Guted Cross
A Ateettin
Lesrnable Quany
Frewet Sine
Projector
Projector Comtinuus Toxenss
Image
Video
Promgt
Leamable Query Token
Vision Toleens
Visual Prompt
Point, Scribble, Box.
Mask
n
Large Language Model (Qwen2.5-3B/
Text Prompt
Person, know, mountain, ski pole, suil. snow/boad, door, countec, Selrcase
Referring Preunpt
Prompl Embeilting
Semanbic Recagnition
Mask
Quaniizer
pants estering gloes coor.
back
Rensoning Prompt
Who night formal attim be dotng
Segmentation Predictor
Diffusion Decoder
Be office buikiing?
Replaze Promptt:
background
koosing people page unchanged
Editing Prompt
Replace the man black, Jacket white pazls with Worznn same p060 and sotting
Image Mask
Vidro Mask
Image Editing
Video Editing
Figure 3: Overview of the FOCUS framework and training pipeline. The left part shows the unified architecture, which integrates vanilla encoder, hierarchical encoder, pixel encoder, a large language model, segmentation predictor, and a diffusion decoder for fine-grained perception and controllable image generation.
mentation, and editing but largely depends the synergy of pretrained components without true end-to-end learning. More critically, recent editing frameworks such as InstructEdit emphasize the importance of high-quality segmentation masks by leveraging external Grounded-SAA -AAM masks to guide diffusion- based image editing. However, these approaches are not fully end-to-end; they separate mask generation from the editing pipeline, highlighting major gap in unifying segmentation awareness with controllable generation.
In parallel, dataset-centric efforts like Localized Narratives 48 and Video Localized Narratives 59 have emerged, providing rich multimodal annotations by aligning every word or phrase with specific image or video regions. These resources enable granular referential grounding and enhance the evaluation and training of models that need capture both spatial and temporal semantics across modalities. Additionally, Describe Anything (31 offers detailed localized captioning, helping bridge the gap between referential understanding and region-specific generation in both image and video domains.
Collectively, these prior works lay the foundation for advancing multimodal understanding, generation, and editing. Yet, there remains an unmet need for unified architecture that couples segmentation- aware decoding with mask-driven diffusion editing in fully end-to-end manner. This motivates development of the proposed framework, which seeks to integrate referential localization, structured mask generation, and controllable editing into a seamless multimodal pipeline.
3 Methodology
As illustrated in Fig FOCUS is unified large vision-languse model that integrates pixel-level perception and controllable image generation an end-to-end framework. The model comprises four core components, covering the entire process from visual encoding to image synthesis. Section 3.] introduce dual-branch visual encoder and a generation extract global semantic features and multi-scale fine-grained representations from inputs at different resolutions, discretizing the visual content into high-level tokens suitable for downstream editing large language model based Qwen2.5 is introduced to unify
visual tokenizer are employed to
tasks. Next, in the Section 3.2

---

multimodal input formats through task prompts, enabling support for wide range of vision-language tasks. Building on this, in the Section 3.3 segmentation decoder is designed to perform high- precision fine- fine-grained object segmentation. A diffusion- based image generator then synthesizes high-fidelity images guided by spatial segmentation masks and conditioned on discrete visual tokens in the Section 3.4 In the Section 3.5 further details the progressive training strategy and the construction multi-source datasets.
3.1 Dual Visual Encoders and Generative Tokenizer
A fundamental challenge in building unified large vision-language models for both image understand- ing and generation lies in the significant discrepancy between high-level semantic understanding and low-level fine-grained image synthesis. This discrepancy often leads to mutual interference and optimization conflicts during training. To address this issue, we propose novel architecture that combines a dual-branch visual encoder with generative visual tokenizer. This design preserves the ability model global high-level semantic features while enhancing the representation of fine-grained, multi-scale low-level visual details. Meanwhile, the visual tokenizer discretizes continuous visual information, effectively extending the representational capacity of the image generation module.
Specifically, we process each image at two resolutions. A low-resolution image X2 E R 3xxx666 x 256 fed into a Vanilla Encoder (based on Qwen ViT (3) capture global semantic information. Simultaneously, high-resolution image X E IR3 768 is encoded by Hierarchical Encoder (Con 41) to obtain high-resolution, multi-scale visual features. These two branches fused using a cross-attention mechanism to enhance low-resolution features with local detail:
X = (onNNNct(XH)), X = QwenViT
(1)
ing CrossAut(XX Xzz), Fang = MLP (F) + Fi
To further inject fine-grained visual information into the language model, we introduce gated cross-attention adapter that enhances learnable queries with high-resolution visual features at multiple scales. For the I-th layer query h and j-th scale feature fing. the fusion is formulated as:
h0' = ( + ")) CrossAttn(h)) G
(3)
(1) 00 + tanh(3()) FFN(
Adapter
where G-(-) is a projection function aligning the visual feature space, and 70,) are learnable scaling parameters initialized to zero. This mechanism injects spatial detail while maintaining the efficiency of learnable queries, enhancing both segmentation and generative capabilities.
To enhance the upper bound of image generation quality, we introduce a generative visual tokenizer based on the MoVQGAN 84 architecture to discretize visual information. The visual tokenizer serves as key component for unified autoregressive image generation, but its quantization process inevitably introduces information loss. To address this, we use the continuous features before quantization as the visual input to the large language model (LLM), which provides more informative representation for fine-grained multimodal understanding. The L.MM then outputs discrete visual tokens that guide the diffusion model for high-quality image synthesis.
Specifically, we use the fused dual-branch features Linnng as the global semantic representation for generation, which are quantized into discrete semantie tokens and decoded through lightweight decoder. During the visual tokenizer's pretraining stage, this semantic branch is supervised with a cosine similarity loss against the original encoder features. A downsampling rate of 28 is adopted to aligo with mainstream languane models and focus on high-level semantic concepts.
We further extend the generative visual tokenizer with pixel branch, following the standard MoVQGAN design. This branch uses a 16x downsampling rate to preserve fine textures. After quantization, the semantic and pixel tokens are concatenated along the channel dimension and passed to the decoder for image reconstruction. The pixel branch is trained with a combination of L1 loss, perceptual loss, and adversarial loss. We use large codebooks for both branches, with size of 32,768 for the semantic branch and 98,304 for the pixel branch.
Together, the generative visual tokenizer and dual-branch visual encoder construct a unified and expressive visual representation that enables FOCUS to support both fine-grained perception and controllable image generation within single framework.
5

---

3.2 Large Vision-Language Model and Input Schema
In FOCUS, we adopt Qwen2.5-3B as the large language model (LLM) model unified multimodal inputs from the visual encoder and task instructions. The model takes as input four- tuple (Bimg:
XT, XT, Addpper) where Ling denotes the fused visual features from the dual-branch encoder, X5 represents the structured textual prompts, Xr refers the continuous pixel-level visual features from the pixel encoder, and 4 Adapter is the learnable query feature obtained through multi-scale cross-modal adaptation.
These inputs are fed into the LLM to produce the output representation:
Fg FLLM Fiigg Xs; XT, ; (1())
"Adapter
From the LLM output Fo. we extract two key components. First, we introduce a decoding constraint that enforces the model to predict the names of all present objects before generating the mask tokens. This prior step encourages the generation of semantically enriched mask tokens, which incorporate global semantic context from the image and are subsequently used by the segmentation module to produce masks.
In addition, the LLM supports autoregressive prediction of discrete visual tokens. Following a semantic-fire first strategy, the model first generates semantic tokens to define global content structure, then generates pixel tokens to recover fine visual textures. This two-stage decoding improves the alignment between textual instructions and generated visual content.
Prompt Design. The input prompt schema consists of two components: the task instruction prompt S, and the condition prompi Sc. The task instruction prompt specifies the objective of the model in natural language, guiding the LLM perform segmentation, generation, or editing accordingly. For example, in class-based segmentation tasks such as panoptic, open-vocabulary. or video instance segmentation, the instruction can be "Please segment all the positive objects by the following candidate categories." In referring or reasoning segmentation tasks (e.g.. RES [23 76 34]. R-VOS 28), Reason VOS (70]), the instruction becomes "Please segment the target referred to by the language description." " For visual-guided tasks such as interactive or video object segmentation, the instruction is phrased as "Please segment according to the given visual reference regions."
The condition prompt provides additional information specific to each task. In blass-based seg- mentation, lists the candidate category labels; in referring segmentation, provides natural language expression; in visual-guided settings, it includes pooled region features sampled from CLIP embeddings specified coordinates. For image generation tasks, the condition prompt consists of a text description such as "a cat sitting on a couch," while for image editing tasks, it contains language-basee or spatial referring expressions like "replace the person the right with dog." " The condition prompt not only supplies contextual information but also serves as an implicit classifier for category-ane aware mask prediction.
3.3 Segmentation Module
The segmentation predictor F, takes three types of inputs: task-specific prompt prompt embeddings {E*
semantic-enhanced mask token embeddings { ) from the LLM output, and multi-scale visual features fing = X и from the visual encoder. Here, K denotes the number of candidate categories, and N represents the number of mask proposals. These inputs are fused to predict segmentation outputs in the following form:
( F, ((E)) (F (
(5)
where mj E TR HR denotes the j-th predicted binary mask, 2j E IR is the associated category score vector, and ej E RD is an optional instance- embedding produced by an auxiliary embedding head enable temporal association in video segmentation tasks.
For video tasks, we adopt a Crame-by-frame processing strategy to generate frame-level segmentation results, ensuring efficient training and inference while maintaining temporal consistency.
To support multi-scale supervision the subsequent image generation stage, the segmentation module consistently produces fixed-resolution masks based on X providing stable spatial guidance. These

---

masks are then rescaled to match the resolution used by the diffusion model, enabling effective alignment between segmentation outputs and the image synthesis pipeline.
3.4 Diffusion Decoder for Controllable Generation
To enable high-quality and regton-controllable image generation, FOCUS incorporates a latent diffusion decoder initialized from SDXL. The generation process is formulated as denoising over latent variables, conditioned on structured visual prompts. We utilize semantic tokens 2em and pixel- level tokens Zpix generated by the large language model. These tokens mapped into continuous embeddings via a learned codebook and concatenated with noisy latents before being passed into a UNet-based denoising backbone.
To achieve spatial control, we leverage predicted segmentation masks m, E H from the segmentation module. These masks are first downsampled to match the latent spatial resolution, yielding m, E IR=''x"" where H' x W" denotes the latent feature resolution. The downsampled mask then flattened and passed through a linear projection layer to obtain a spatial guidance sequence fm E IR RxxCC where L = H' x W and C' is the channel dimension. This sequence is injected into the UNet through cross-attentin at intermediate layers to guide regional generation. The interaction is formalized as:
2 = CrnossAt(( fm), E++1 = UNet(2))
where ((²²) denotes the projected latent features at denoising timestep 4, and fm provides the spatial condition derived from m..
We also experimented with using mask token embeddings {Ea as alternative conditioning inputs, but found that direct spatial control through explicit masks yields better localization and more faithful region editing in our experiments.
3.5 Training Procedure and Data Composition
To support unified pixel-level understanding, high-level multimodal reasoning, and spatially control- lable image generation, FOCUS adopts a progressive four-stage training paradigm that incrementally builds visual tokenization, vision-language alignment, and conditional generation capabilities.
Stage 1: Pretraining of Dual-Branch Visual Tokenizer and Diffusion Decoder. We first train semantic branch and pixel branch to discretize input images into semantic and texture tokens using Si quantizers. A progressive resolution strategy is adopted, starting from 256 x 256 and gradually increasing to 512 x 512. large-scale image-text dataset of approximately 58M pairs is constructed based on resolution constraints and visual diversity. On top of this, we pretrain a latent diffusion decoder using 10M image subset, enabling high-fidelity reconstruction from discrete tokens. This stage focuses purely image modeling and does not involve segmentation or controllable generation.
Visual- Liagua-L Adapter Warmup. To align visual features with the input space of the large language model (LLM), we train projection heads for both the vanilla encoder and the pixel encoder, along with learnable queries and gated cross-attention modules the hierarchical encoder branch. All visual backbone encoders are kept frozen. Training is conducted at 256 x 256 resolution with standard language modeling loss.
Stage 2:
Stage 3: Multimodal Pretraining with Segmentation-Aware Alignment. In this stage, we jointly train the LLM, visual adapters, and the mask decoder to improve the model's perception of segmenta- tion structures and enhance multimodal generation capability. Training performed in two phases with input resolutions of 256 256 and 512 512. Supervision includes token-level cross-entropy loss, segmentation mask supervision (Dice BCE), and image reconstruction loss (L.) from diffusion outputs. The training corpus includes multimodal data spanning image-text pairs, segmentation tasks, and vision- -anguage reasoning. Full dataset details are presented in appendix.
Stage 4: Instruction Tuning for Region-Controlled Editing. This stage introduces region-level controllability for editing and generation tasks. We jointly train the LLM, the mask decoder, and
7

---

cross coss-attention layers within the diffusion model, while freezing the visual encoders, visual tokenizer, and VAE components. Input images are resized using bucketed aspect ratio cropping strategy with total pixel counts ranging from 5122 to 1024 Supervision includes token prediction (cross-entropy), segmentation mask prediction (Dice BCE), and image reconstruction (1.2 loss). Detailed task and dataset configurations are summarized in appendix.
4
Experiments
We conduct comprehensive experiments to evaluate the performance of FOCUS across three repre- sentative tasks: multimodal understanding, referential segmentation, and generation and controllable editing. All experiments are designed to validate the model's 's ability to unify perception and genera- tion in an end-to-end framework, with strong alignment to natural language instructions, fine-grained localization, and object-aware editing fidelity.
4.1 Implementation Details
In our experiments, we adopt Qwen2.5 as the backbone large language model (LLM). The visual encoder consists of two branches: the semantic decoder is composed of four attention blocks with 2D relative positional encoding (2D-RoPE), while the pixel encoder and decoder follow MoVQGAN- based architecture with base channel dimensions of 128 and 384, respectively. The codebook size 728 for the semantic branch and 98,304 for the pixel branch, with both using code dimension of 32. We employ the AdamW optimizer without weight decay and use constant learning rate across the visual encoder, diffusion decoder, and the large vision language model. Detailed training hyperparameters for each component are summarized in the supplementary materials. The training of the Dual-Branch Visual Tokenizer and the diffusion decoder each took approximately days on a computing cluster, while the 3B 3B-parameter MLLM required around 13 days to complete the shree-stage training process.
Table 1: FOCUS performance on general and document-orienter benchmarks.
Mithod
MrdBerch
Uesero: MME-P
MM--Vs
MMMU
VQ4-cai
ChanQA
DscVOA
Is6eVA
OCRBench
Undesszaddigg. Unl
Quee-711
60.6
1487.5
35.9
45.9
61.5
62.6
453
1L. 15
Visus-78 7B
85.9
64.3
59.6
1510.7
31.1
35.4
54.5
59.2
18.2
28.1
25.8
318
Mleet M
Viaiaa7BB
68.8
89-7
1967.4
12.6
12.2
54
M0-
11.5
171
NoXV Es
Vxuss-7B
85.5
67.4
64.7
1519
43.9
35.1
56.6
64.1
56.
14.4
37.1
552
scrucch
Unbestarding
Tmmins-10
Chamelcor
scrach
23.4
193 X1
LLaVA-2-7B
75.2
9.6
18.8
Show-e [4]
PSi-1.8
73.5
$48.4
25.1
VIL.A-1/(P5 (156)
11aMA-3--5
16.3
1116
32.7
48.3
VILA-UU
LLaMA,2-77
Jamo-Thu-11
BeepSork-11 M--1 MB
86.1
75.5
3
1484.0
19.8
16.5
Janus Pro
13
DeepSeek-11.. LLM. 7B
87.4
79.2
72.1
1567.1
50.0
41.0
ILLIMB+
Quee2 S-16
87A
20.8
23.3
1416.0
40.3
46.5
74.2
60.9
69.9
10.8
46.1
TOCUS
Quee2 16
35.0
WL3
153
1510L
003
119
74.
70.1
70.
$1.1
11.
4.2 Multimodal understanding
To evaluate the multimodal understanding capabilities of our model, we conduct systematic evalua- tions on two categories of widely-used benchmarks, is show table [ (1) General benchmarks, including POPE, MMBench, SEED, MME-P, MM-Vet, MMMU, and AI2D; and (2) Document- oriented benchmarks, including VQA-text, ChartQA, DOc. IQA. and OCRBench. Experi- mental results show that, despite using only 3--parameter model, FOCUS achieves performance comparable to state-of-the-at unified models such as Janus-Pro-7B and I...UMF-7B, and notably outperforms ILLUME+ with the same parameter scale. This performance gain is largely attributed to the incorporation of multi-scale high-resolution features and segmentation masks, which significantly enhance pixel-level perception.
4.3 Controllable Generation and Editing
Multimodal image generation. To evaluate the multimodal visual generation capability, we use the MJHQ-30K, GenAl-bench and GenEval benchmarks in the table For MJHQ-30K, we adopt the
8

---

Fréchet Inception Distance (FID) metric on 30K generated images compared to 30K high-quality images, measuring the generation quality and diversity. GenAl-bence belbhehch and GenEval are challenging text-to-image generation benchmarks designed to reflect the consistency between text descriptions and generated images. We compare FOCUS with previous state-ofthe-arr multimodal generation-only and unified models. This highlights the superior generation quality and diversity enabled by our diffusion-based approach. Additionally, FOCUS achieves competitive results on the GenAl-bench and Gval benchmarks and attains the highest accuracy in advanced categories on GenAl-bench, demonstrating its ability to understand and generate images from complex text descriptions. Figure 7 shows more results of FOCUS on generating flexible resolution images.
Table 2: Evaluation results on multimodal image generation benchmarks.
Parums,
Type
FID
JowcTJooch
Bask
Overall Gooeration Cnly
GrnEval Counting
Colors
Position
Color
0.98
Dillasim Dillusion
0:97
0.86
PixAn--
0:48
0.98
0.45
0.07
SDXL
2
9.55
0.33
0.98
0.41
0.15
Gen
Ausores.
0.58
0.51
0.21
Unjy Undersionday and Commrnoos
Clamnloo
1
Autorey
UWM 35
7BI
Aunone::
17:77
0.61
0.53
0.47
0.93
0.41
0.09
0.09
0.15
Show-n 65
Aunong
15.10
0.70
0.60
0.45
0.95
0.52
0:49
0.83
0.78
0.30
12.81
Aunrnyy
7.60
Autory-
10.1
Jamss-ho-11
1.38
Autore.
0.99
0.86
0.26
hamss-ho-77t
7B
Aueeng
0.80
0.69
0.59
0.90
0.59
0.79
0.66
0.25
1.1UME
6.00
0.71
0.23
1.00
0.99
0.58
0.62
0.84
0.11
FOCUS
38
Ausoreg
6.05
0.83
0.72
0.75
1.20
0.38
0.57
0.56
0.1
0.57
Multimodal image editing. To assess the multimodal image editing capability of our method, we evaluate it on the Emu Edit benchmark and report the CLIP-I, CLIP-T, CLIP-DIR and DINO scores. The CLIP-I and DINO scores measure the model's ability to preserve elements from the source image, while the CLIP-T and CLIP-DIR score measures the consistency between the output image and the target caption. As illustrated in the table 4. our model demonstrates strong performance in image editing tasks, surpassing specialized models, particularly in the CLIP-T metric. This indicates that the unified model's superior understanding enhances its ability to interpret editing instructions, resulting in more precise modifications.
Table 4: Quantitative results on image editing benchmarks. The performance with top-1 I and top-2 value are denoted in bold and underline.
Table Comparisons with other referring segmentation.
Metual
Retcoco
Relonsi
pRCCCOCO
Type
Tasis
DINO CLIP-E CLIP-T
[X]
CLIP-DIR
CRIS 100
LWT MX
10.5
72.7
75.8
73.2
688
05
613
62.1
61
61.
537
35
012 02.
315
00.4
553
57.6
63.
633
$1.
55.0
IrstrudPPsPPPs
Diflucin
0.762
0.534
0.219
0.075
MagicBrash
Dilluiion
eniy
0.222
151M
Omniien 55
Diffuson
lddt
0.804
0.836
0.235
135A-78 P4
11.
7511
723
651
709
58
67.9
706
52.6 44.1
Pinu lift
Diffucon
fatn
0.819
0.859
0.231
0,109
Reel Mna K0
73.0
682
663
717
583
09
705
PUMA [+ ILLUME
Alk
0:785
0.545
0,270
Hoperher
55.7
834
780
753
764
47.5
20.1
Alk
Und. Gem, Fdtt
0.991
0,879
0.260
TOCUS
311
153
323
733
901
183
223
701
401
203
350
IEUMP+
AIR
Und. Cum, Rdi
0.676
0.872
0.228
0.101
FOCUS
AR
Uni, Clan, Rax
0,831
03.076
0.278
0.105
4.4 Referring Segmentation Accuracy
We evaluate the referential segmentation performance of FOCUS four standard benchmarks: RefCOCO, RefCOCO+, RefCOCOg, and gRefCOCO, using mean Intersection-ovee over-Union (mloU) as the evaluation metric. As shown in table FOCUS achieves competitive or superior performance compared to both segmentation-sseecift and LVLM methods. These results demonstrate the effectiveness of FOCUS in pixel-level target localization and its strong capacity to align complex referring expressions with visual semantics in an end-to-end framework. More empirical evidence is presented in the appendix.
4.5
Ablation Studies
4.5.1
The effect of multi-stage training
The following table shows the results of using different image resolutions (256, 512, and 1024) for all-stage task training. We observe a clear trend: higher image resolutions improve the model's performance in image generation and segmentation, but lead to decline in language understanding.

---

Table The effect of different image resolutions and multi-stage training strategy on model perfor-
mance across various tasks.
indicates higher is better, +. indicates lower is better.
MJIIQ30K
Image Understanding
RefCOCO
Image Size
FID(J)
Busic(")
Adv((1)
POPE(T)
MMB(1)
SEED(1)
testA(")
TestB(")
Valid(")
256
11.85
0.63
0.56
85.3
73.1
70.1
78.5
79.2
75.4
7.39
0.70
0.61
87.3
80.6
71.3
81.3
82.4
79.8
1024
6.03
0.76
0.69
77,6(L)
68.2(L)
68.8(L)
83.4
84.9
81.5
256->1024(Our)
6,05
0.83
0.72
88.0
81.5
73.9
84.1
86.3
82.7
Our analysis suggests that this decline is mainly due to an increase in hallucination issues at higher resolutions-whee processing high-resolution images, the model is more likely generate text that is inconsistent with the visual input. This increased hallucination negatively impacts the model's s language comprehension. highlighting the challenge of balancing visual precision and robust language understanding in multimodal models.
To address this, multi-stage training strategy progressively increases image resolution and task complexity, starting from simpler tasks at lower resolutions. This approach enables strong and balanced performance across all tasks.
4.5.2 The effect of multi-scale features in Gated Cruss-Attention
Table 6: Ablation study on the effect of multi-scale feature fusion in Gated Cross- Atteenioon
Model Variant
Scales
RefCOCO mloU(f)
MMBench Acc.(f)
CLIP-T(Editt()
FOCUS- SingleScale (Fine)
79.5
72.8
0.265
FOCUS-SingleSect SingleScalc (Coarse)
1
78.2
73.9
0.260
2
80.5
73.5
0.270
FOCUS (Ours)
3
81.4
73.9
0.275
To effectively fuse multi-scale features, our model employs a Gated Cross-Attention module to
process outputs from the ConvNeXt-L backbone, achieving a balance between fine tnne-graiedd detail and global contextual understanding. Specifically, we design a 3-layer Gated
adapter that hierarchically integrates three distinct feature scales: the first layer focuses fine features (f2), the second on mid-level features (fa), and the third on coarse semantic features (f4).
As illustrated in the table, compare shree-scale model (f2, f3. fa) with variants that utilize only a single scale (f4), dual scales (fa. fa), and all four scales (f2-f4). The results (mloU on ReiCOCO) demonstrate that our multi-scale fusion strategy is essential for accurately segmenting fine details, providing the optimal balance between performance and computational efficiency.
4.5.3 The effect of continuous visual tokens
Table 7: Ablation study on effect of continuous visual token input on model performance.
Continuous MJHQ30K
GenAl-Bench
Image Understanding
Input
FID4
Basict
Adv.
POPET
MMB1
SEEDT
6.85
0.68
0.65
82.1
50.4
56.0
6.05
0.83
0.72
85.3
70.9
66.6
Our ablation study demonstrates that feeding continuous visual features directly into the LLM is essential for optimal performance. By bypassing discrete tokenization, this approach effectively reduces information loss and retains fine-grained visual details. As result, the LLM receives richer and more precise visual representation, which significantly enhances its ability to comprehend complex multimodal scenarios and accurately interpret referential instructions.
Conclusion
FOCUS demonstrates robust generalization and strong task performance across multimodal dialogue, referential segmentation, and controlled visual editing. The consistent improvements over baseline models affirm the effectiveness of our unified architecture, which tightly integrates segmentation- aware perception with instruction-guii -uided generation. These results further support the practical value of FOCUS in real-world multimodal interaction scenarios.
10

---

6 Acknowledgement
This work was supported by the National Key Research and Development Program of China (No. 2024YFE0210600) the National Key R&D Program of China under Grant No. 20222D0160601, the National Natural Science Foundation of China under Grant Nos. 62176254 and 62276260, and the Beijing Natural Science Foundation (No. L247028).
References
[1] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Micch, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: visual language model for few-shot learning. Advances in Neural Information Processing Systems, 25:3371 2022.
Anthropic. Claude 3:
Next-generation models.
https://www.nthhrt
news/
elaude-3)
2024. Accessed: 2025-05-12.
[3] Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang. Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A versatile lanounl model for understanding, localization, text reading, and beyond. arXiv preprint arXiv:2308, 12966, 2023.
[4] Tim Brooks, Aleksander Holynski, and Alexei A. Efros. Instructpix2pix: Learning to follow image editing instructions, 2023. URL battts:://aastttt http://axxvvvv...aaaa.mm http://wwww....//////.1.....
[5] Minwoo Byeon, Beomhee Park, Haccheon Kim, Sungjun Lee, Woonhyuk Back, Sachoon Kim. Coyo- 700m: Image-text pair dataset. htttps://ggitii
com/ www.///////oe//eeee www.kkaaaaaaaaaaa wtw.kaawwrraaaoooooooooooooo coyo poroyo--atttteo dat aset 2022.
Sergi Caclles, Alberto Montes, Kevis-Kokits Maninis, Yuhua Chen, Luc Van Gool, Federico Perazzi, and Jordi Tuset. The 2018 davis challenge on video object segmentation, 2018. URL https
//axxiv.org wtt.v/.orrgabb/1103.3.. 10t...//183..00 00557|
[7] Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdan Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-o: Fast training of di diffusson transformer for photorealistic text-to-image synthesis, 2023. URL https:tt///// /ettps://axivv.. org/ abs / /2310 00426
[8] Kai Chen, Yunhao Gou, Runhui Huang, Zhili Liu, Daxin Tan, Jing Xu, Chunwei Wang. Yi Zhu, Yihan Zeng, Kuo Yang, Dingdong Wang. Kun Xiang, Haoyuan Li, Haoli Bai, Jianhua Han, Xiaohui Li, Weike Jin, Nian Xie, Yu Zhang, James T. Kwok, Hengshuang Zhao, Xiaodan Liang, Dit- Yan Yeung, Xiao Chen, Zhenguo Li, Wei Zhang. Qun Liu, Jun Yao, Langing Hong. Lu Hou. and Hang Emova: Empowering language models to see, hear and speak with vivid emotions, 2025. URLbttps http://axxxvv..ooo /bs/
[9] Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. Sharegpt4v: Improving large multi-mon modal models with better captions, 2023. URL http::://a/aat
org/ abs/ /toos/231111 12193
[10] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng Gao, and Chunyuan Li. Llava-interactive: An all-in-one demo for image chat, segmentation, generation and editing, 2023. (Rtppsss:///rx
org/ aba//b///// 00bo/2311... 3111.00771
[11] Wei-Ge Chen, Irina Spiridonova, Jianwei Yang, Jianfeng Gao, and Chunyuan Li. Ilava-interactive: An all-in-one demo for image chat, segmentation, generation and editing. arXiv preprint arXiv:2311.00577 2023.
[12] Xianjie Chen, Roozbeh Mottaghi, Xiaobai Liu, Sanja Fidler, Raquel Urtasun, and Alan Yuille. Detect what you can: Detecting and representing objects using holistic models and body parts. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1971-1978, 2014.
[13] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling, 2025.
wtg/abs///eaee.orr..ooo.ee 2501. 17811
[14] Bowen Cheng. Ishan Misra, Alexander G. Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. 2022. https:/ /ttp://axxvvv./.v org/ abs/
[15] Guillaume Couairon, Jakob Verbeek, Holger Schwenk, and Matthieu Cord. Diffedit: Diffusion -aased semantic image editing with mask guidance, 2022. (tLlatts:: /ttp://////iv.v/ee www.:/iioooroooeooooo.eeo abs/abbsl000 11427

---

[16] Wenliang Dai, Junnan Li, Dongxu Li. Anthony Meng Huat Tiong, Jungi Zhao, Weisheng Wang, Boyang Li, Pascale N Fung, and Steven Hoi. Instructblip: Towards general-purpuse vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 36, 2024.
[17] Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jac Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo. YenSung Chen, Ajay Patel, Mark Yatskar. Chris Callison-Burch Andrew Head, Rose Hendrix. Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michacl Schmitz, Aaron Saraat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittli,, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah Smith, Hannanch Hajishirzi, Ross Girshick, Ali Farhadi, and Aniruddha Kembhavi. Molmo and pixmo: Open weights and open data for state-of-the-art vision-language models, 2024. 17146
http:://axxiv.. org/ /
[18] Rongyao Fang, Chengqi Duan, Kun Wang, Hao Li, Hao Tian, Xingyu Zeng. Rui Zhao, Jifeng Dai, Hongsheng Li, and Xihui Liu. Puma: Empowering unified mllm with multi-granular visual generation, 2024. URL
btttps://arxiiii http://awwwww..../ab//////s 2410. 13861
[19] Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Minlic Huang, Nan Duan, and Weizhu Chen. Tora: A tool-integrated reasoning agent for mathematical problem solving, 2024. URL
https://arxiv. wwtp://///aaaae....oo..ooe.e http:/ab/2233..1//5 17452
[20] Ju He, Shuo Yang, Shaokang Yang, Adam Kortylewski, Xianding Yuan, Jie-Neng Chen, Shuai Liu, Cheng Yang, Qihang Yu, and Alan Yuille. Partimagenet: A large, high-quality dataset of parts. In European Conference on Computer Vision, pages 128- 145. Springer, 2022.
[21] Runhui Huang. Chunwei Wang. Junwei Yang. Guansong Lu, Yunlong Yuan, Jianhua Han, Lu Hou, Wei Zhang, Langing Hong. Hengshuang Zhao, and Hang Xu. Illume+: Illuminating unitied milm with dual visual tokenization and diffusion refinement, 2025. URI. / org/ abs/ 2504. 01 934
[22] Siming Huang. Tianhao Cheng, J. K. Liu, Jiaran Hao, Liuyihan Song, Yang Xu, J. Yang. Jiaheng Liu, Chenchen Zhang, Linzheng Chai, Ruifeng Yuan, Zhaoxiang Zhang, Jie Fu, Qian Liu, Ge Zhang, Zili Wang. Yuan Qi. Yinghui Xu, and Wei Chu. Opencoder: The open cookbook for top-tier code large language models, 2025. httpthttt////attt www.aiiww.o....o abs /211/21111 2211.04900 04905
[23] Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), Jan 2014. doi: 10.3115/v1/d14-1100 1086. URL/http://cxx dei. org/
10.31 10.3115/51/a14-110
[24] Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model. arXiv preprint arXiv:2308 00692, 2023.
[25] Bo Li, Yuanhan Zhang, Dong Guo. Renrui Zhang, Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li. Ziwei Liu, and Chunyuan Li. Llava-onevision: Easy visual task transfer, 2024. URL
https://arxe www.xxi..oo///b.... wtt/abs///aa.. 2200/4433..0 03326
[26] Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. Seed-bench: Benchmarking multimodal Ilms with generative comprehension, 2023. URL
1612
[27] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models.
[28] Xiang Li, Jinglu Wang, Xiaohao Xu, Xiao Li, Bhiksha Raj, and Yan Lu. Towards robust referring vidco object segmentation with cyclic relational consensus, 2023. htRps:/t http:t/s//aaai..ooo//////// 01203
[29] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen, Evaluating object hal- lucination in large vasion-language models, 2023. URL htttpp::/aaaai..oo//////////////
10355
[30] Zhaowei Li, Wei Wang, YiQing Cai, Xu Qi. Pengyu Wang, Dong Zhang, Hang Song. Botian Jiang, Zhida Huang, and Tao Wang. Unifiedmllm: Enahling umified representation for multi-modal multi-tasks with large language model, 2024. URL
www.vv.rrgg..... 2408 02503
[31] Long Lian, Yifan Ding, Yunhao Ge, Sifei Liu, Hanzi Mao, Boyi Li. Marco Pavone, Ming-Yt Yu Liu, Trevor Darrell, Adam Yala, and Yin Cui. Describe anything: Detailed localized image and video captioning, 2025.
/ arrivv org/ abs /21//254..166
12

---

[32] Wing Lian, Bleys Goodson, Eugene Pentland. Austin Cook. Chanvichet Vong, and "Teknium". Openorca: An open dataset of gpt augmented flan reasoning traces. huttaa//hattee://uugg..iggt 2023.
[33] Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment hefore projection. arXiv preprint arXiv:2311. 2023.
[34] Chang Liu, Henghui Ding, and Xudong Jiang. Gres: Generalized referring expression segmentation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23592- 23601, 2023.
[35] Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million length video and language with blockwise ringattention, 2025.
08268
[36] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning, 2023.
[37] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, 2023.
[38] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning. Advances in neural information proessing systems, 36, 2024.
[39] Jiang Liu, Hui Ding, Zhaowei Cai, Yuting Zhang, Ravi Kumar Satzoda, Vijay Mahadevan, and Manmatha. Polyformer: Referring image segmentation as sequential polygon generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 18653-18663, 2023.
[40] Yuan Liu, Haodong Duan, Yuanhan Zhang. Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang. Conghui He, Ziwei Liu, Kai Chen, and Dahua Lin. Mmbench: Is your multi-modal model an all-around player?, 2024. URL . httpp://aaxxiiii
org
wtwg/ba///aaa /23aaab//332..C 06281
[41] Zhuang Liu, Hanzi Mao, Chao- Wu, Christoph Feichtenhofer, Trevor Darrell, and Saining Xic. convnet for the 2020s. In Proceedings of the IEEE/CVF conference on computer vision and patter recognition, pages 11976-1196,, 2022.
[42] Dakuan Lu. Xiaoyu Tan, Rui Xu, Tianchu Yao, Chao Qu Wei Chu, Yinghui Xu, and Yuan Qi. Sep- 1 16k: A hugh-quality prolemmssluuton dataset and generalized pipeline for automated extraction in the higher education science domain, 2025. URI URL http://ttt///xii..//aeaa
abs /250 15587
[43] Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: unified model for vision, language, and multi-modal tasks, 2022.
abs/
2206. 08916
[44] Junhua Mao, Jonathan Huang, Alexander Toshev, Oana Camburu, Alan Yuille, and Kevin Murphy. Generation and comprehension of unambiguous object descriptions. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016. doi: 10.1109/cvpr.201699.
//xp:////x//// doi
org/10. 1T09/ c10./ 6.9
[45] OpenAL. Chatgpt: Language model (gpt-4). https://chas: cttps:///caat...rai.... com 2023. Accessed: 2025-
05-12. 12.
[46] William Peebles and Saining Xie, Scalable diffusion models with transformers, 2023. URL. httpa
/arxiv.oro abs abs/2212. 0ttp/2222.07749
[47] Dustin Podell, Zion English, Kyle Laccy, Andreas Blattaann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis, 2023.
BtRLhttps::2 //aavvv./a//iva/aa http:/awwwvvvr..reaaaa/ abs///b///// 2200/333.0011. 01952
[48] Jordi Pont-Tuset, Jasper Uijlings, Soravit Changpinyo, Radu Soricut, and Vittorio Ferrari, Connecting vision and language with localized narratives, 2020. URI
992.. 03098
[49] Vignesh Ramanathan, Anmol Kalia, Vladan Petrovie, Yi Wen, Baixue Zheng, Baishan Guo, Rui Wang, Aaron Marquez, Rama Kovvuri, Abhishek Kadian, et al. Paco: Parts and attributes of common objects. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7141-7151, 2023.
[50] Zhongwei Ren, Zhicheng Huang. Yunchao Wei, Yao Zhao, Dongmei Fu, Jiashi Feng, and Xiaojie Jin. Pixellm: Pixel reasoning with large multimodal model. arXiv preprint arXiv:2312.02226 2023,
13

---

[51]
Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, June 2022.
[52] Sconguk Seo, Joon- Young Lec, and Bohyung Han. Urvos: Unified referring video object segmentation network with large-scale benchmark. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors. Computer Vision ECCV 2020, pages 208 223, Cham, 2020. Springer International Publishing. ISBN 978-3-030-58555-6.
[53] Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks, 2023, URL
attps://arxiv: http://wxxx....gaaaaaaa abs///b///// aata/abl.l..... 2211 1.411.100 10089
[54] Yichun Shi, Peng Wang, and Weilin Huang. Seededit: Align image re-generation to image editing, 2024. httptpp:///axxi.orr///.e/
[55] Quan Sun, Qiying Yu. Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang. Hongcheng Gao, Jingjing Liu. Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality. arXiv preprint arXiv:2307.05522, 2023.
[56] Quan Sun, Yufeng Cui, Xiaosong Zhang, Fan Zhang, Qiying Yu, Yueze Wang, Yongming Rao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Generative multimodal models are in-context learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14398-14409) 2024.
[57] Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2025. URL https
//arxiv. abs/2405. attpxv/v.rr///a//255..... 09818
[58] Teknium. Openhermes 2.5: An open dataset of synthetic data for generalist IIm assistants, 2023, URL. http://aaaaaaaaaaaaeaaoooee/eaee OpenHe 2.5
https:
[59] Paul Voigtlaender, Soravit Changpinyo, Jordi Pont-Tuset, Radu Soricut, and Vittorio Femarr. Connecting vision and language with video localized narratives, 2023. http:s:tt:////ttt httpttps:/aaxxx..rrtee org/ abs/2b00000
11217
[60] Qian Wang. Biao Zhang, Michael Birsak, and Peter Wonka. Instructedit: Improving automatic masks for diffusion-basse based image editing with user instructions, 2023. htttttt///axxiiiii org/ abs hatp://bb///////
18047
[61] Weihan Wang, Qingsong Lv, Wemmeng Yu, Wenyi Hong. Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Xixuan Song, Jiazheng Xu, Bin Xu, Juanzi Li, Yuxiao Dong, Ming Ding, and Jie Tang. Cogvim: Visual expert for pretrained language models. Nov 2023.
[62] Xinlong Wang. Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang. Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, and Zhongyuan Wang. Emu3: Next-token prediction is all you need, 2024. URL aatps
//axxiv.org/ 2009.//249...888
[63] Ryan Webster, Julien Rabin, Loic Simon, and Frederic Jurie. On the de-duplication of laion-2b, 2023. MtLhhttpss//aaxiv http:///wwxxx.oooaa///////
12733
[64] Cong Wei, Yujic Zhong, Haoxian Tan, Yong Liu, Zheng Zhao, Jie Hu, and Yujiu Yang. Hyperseg: Towards universal visual segmentation with large language model, 2024. URL http::tttt://att org/ /bs/
2411 2411.17606 17606
[65] Cong Wei, Zheyang Xiong, Weiming Ren, Xinrun Du, Ge Zhang, and Wenhu Chen. Omniedit: Building image editing generalist models through specialist supervision, 2025. URI. htttp:://arxiv.org
abs abs/2411.. .71199
[66] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma. Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, and Ping Luo. Janus: Decoupling visual encoding for unified multimodal understanding and generation, 2024.
[67] Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang. Ligeng Zhu, Enze Xie. Hongxu Yin, Li Yi, Song Han, and Yao Lu. Vila-u: unified foundation model integrating visual understanding and generation, 2025. URL arxiv. org/ abs/2409.04429
14

---

[68] Jinheng Xie, Weijia Mao. Zechen Bai, David Junhao Zhang. Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation, 2024. URL /ta.ps//aaxx...ooggg
2408 248.
[69] Zhangehen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, and Bill Yuchen Lin. Magpie: Alignment data synthesis from scratch by prompting aligned Iims with nothing, 2024. URL.
https:://axxyy: org/ abs 2206. 08464
[70] Cilin Yan, Haochen Wang. Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang. Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models, 2024. URL ptt
//arxiv.org/ab//200..
[71] An Yang. Baosong Yang. Beichen Zhang, Binyuan Hui, Bo Zheng. Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang. Haoran Wei, et al. Qwen2. technical report. arXiv preprint arXiv:24/2.15115, 2024.
[72] Linjic Yang. Yuchen Fan, and Ning Xu. Video instance segmentation, 2019. URL URLpbttss://axxt org/ abs/1905. 04804
[73] Senqiao Yang. Tianyuan Qu, Xin Lai. Zhuotao Tian, Bohao Peng, Shu Liu, and Jiaya Jia. An improved baseline for reasoning segmentation with large language model. arXiv preprint arXiv:2312.17240 2023.
[74] Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language- aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18155-18165, 2022.
[75] Нaoxuan You, Haotian Zhang, Zhe Gan, Xianzhi Du, Bowen Zhang, Zirui Wang, Liangliang Cao, Shih-Fu Chang, and Yinfei Yang, Ferret: Refer and ground anything anywhere at any granularity. Oct 2023.
[76] Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara I. Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14. pages 69-85. Springer, 2016.
[77] Qifan Yu. Wei Chow, Zhongqi Yue, Kaibang Pan, Yang Wu, Xiaoyang Wan, Juncheng Li, Siliang Tang. Hanwang Zhang, and Yueting Zhuang. Anyedit: Mastering unitied high-quality image editing for any idea. arXiv preprint arXiv:2411. 15738, 2024.
[78] Weihao Yu, Zhengyuan Yang. Linjie Li, Jianfeng Wang. Kevin Lin, Zicheng Liu, Xinchao Wang. and Lijuan Wang. Mm-vet: Evaluating large multimodal models for integrated capabilities, 2024. URL
attps://arxivy: org/abs/////oooooo.. abs/ 2308 220./22290000
[79] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng, Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, Cong Wei, Botao Yu, Ruibin Yuan, Renliang Sun, Ming Yin, Boyuan Zheng, Zhenzhu Yang, Yibo Liu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mmmu: massive multi-discipline multimodal understanding and reasoning benchmark for expert agi, 2024. URL
https: /ttp://aaaivvvvvva org/ abs/ / 2311. 16002
[80] Rowan Zellers, Yonatan Bisk, Ali Farhadi, Yojin Choi, and Paul Allen. From recognition to cognition:
Visual commonsense reasoning.
[81] Kai Zhang, Lingbo Mo. Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guide guided image editing, 2024. URL httltttts///xxiii/ea wttp://www....r//a//.......0.
[82] Zheng Zhang, Yeyao Ma, Enming Zhang, and Xiang Bai. Psalm: Pixelwise segmentation with large multi-modal model, 2024. htt.:tttts://axxi..coo
[83] Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-base basod ane-grained image editing at scale, 2024. URL
https:
org/ abs/ 2207 05282
[84] Chuanxia Zheng, Long Tung Vuong, Jianfei Cai, and Dinh Phung. Movq: Modulating quantized vectors for high-fidelity image generation. 2022. URL httpss//t///t wwwxiv.oog/aaa ab///209./222//../////// 2209. 09002
[85] Zangwei Zheng, Xiangyu Peng. Tianji Yang. Chenhui Shen, Shenggui Li, Hongxin Liu, Yukun Zhou, Tianyi Li, and Yang You. Open- sora: Democratizing efficient video production for all, 2024. URL
https://arxiv wwt./bs//////a////eoeee 20404
[86]
Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: Enhancing vision- language understanding with advanced large language models, 2023. URL httt:ttt///////i....oo
abs/ / 2200 105922
15

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The abstract and introduction clearly outline the primary contributions of the paper, including the proposal of a unified end-to-end large vision language model for pixel-level perception and controllable generation. These claims are consistently supported by the theoretical design and experimental results presented in the main body of the paper, ensuring alignment between stated contributions and actual findings.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: The paper includes a dedicated "Limitations" section that reflects several key constraints the proposed approach, such as limited generalization to out-of-distribution visual domains and the computational cost associated with joint training of segmentation and generation modules. Additionally, the authors discuss the assumptions made during pretraining and inference, and acknowledge potential issues in real-world deployment scenarios involving ambiguous or complex instructions.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best
16

---

judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Answer: [NA]
Justification: The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper provides detailed information on the experimental setup, including model architecture, training hyperparameterss, dataset splits, evaluation protocols, and hardware configurations. All critical implementation details that affect the main claims are included in the main text appendix. This ensures that researchers reproduce the results even without access to the code or pretrained models.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (c.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
17

---

(c) If the contribution is new model (e.g.. a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: We provide partial code in the supplemental material to demonstrate the openness and transparency of our method. Full code, pretrained models, and detailed instructions for reproducing the main results will be released after paper acceptance to ensure compliance with anonymity requirements during the review process.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurIPS code and data submission guidelines (https://nipp) cc/
pub public/Sn guides CodeSubmi ssi asionnollii icy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https)
hetooiiic/gidds...... //adeee///oSSeee CodeeodSuuuig Submiss wrriiisssooPoioggg for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (c.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification: The paper provides comprehensive experimental details, including dataset splits, model architecture configurations, training hyperparameters, optimizer types, learning rate schedules, and input resolutions. These are specified in the main and appendix to ensure clarity and reproducibility. The rationale behind hyperparameter selection is also explained based on validation performance.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
18

---

7. Experiment statistical significance
Question: Does the paper report enror bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [NA]
Justification: While the paper includes extensive quantitative results and comparisons, do not report statistical significance metrics such as error bars or confidence intervals, as the primary experiments are deterministic and evaluated on standardized benchmarks. Variance analysis is left for future work.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the enror bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (c.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We report the GPU type (e.g.. A100), memory size, number of training hours, and cluster environment used for each major experiment in the appendix. This ensures that others can estimate the required compute for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
ness?
Answer: [Yes]
19

---

Justification: The research fully adheres to the NeurIPS Code of Ethics. All data used are publicly available or appropriately licensed. no personally identifiable information is involved, and no human subjects were used. We have taken care to ensure fairesss, transparency, and reproducibility throughout the study, and all content has been anonymized in accordance with double-bling review requirements.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: The work focuses developing foundational methods for multimodal vi- sion languag modeling and does not directly target downstream applications with im- mediate socictal implications. As such, potential societal impacts-wwin -whethr positive or negative-ars are not discussed in the paper. We consider this research as building block for future systems, and a more thorough societal impact assessment would be appropriate the deployment stage.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (c.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer:[NA]
Justification:
The paper poses no such risks
Guidelines:
The answer NA means that the paper poses such risks.
20

---

Released models that have high risk for misuse or dual use should be released with necessary safeguards to allow for controlled use of the model. for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: we cite the original paper that produced the code package dataset
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperrs thcode com / da t. sets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re- re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
21

---

Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/teview based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if' applicable), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, or originality the research, declaration is not required.
Answer: [Yes]
Justification: We use an existing large language model (LLM) part of our framework support downstream vision tasks such as segmentation and inage-conditioned generation. However, do not modify the architecture pretraining of the LLM itself. All fine-tuning is conducted on publicly available datasets, and the LLM is treated as a fixed backbone to support multimodal learning.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non non-standard components.
Please refer to our LLM policy (https: / oc cct.c//oeeeeeeece/22555555
LLM) for what should should not described.
A Dual Quantizer and Diffusion Decoder Training
To enable high-quality image generation, FOCUS pretrains dual quantizers with three-branch visual encoder and subsequently trains a diffusion decoder conditioned on the tokens. This section details the training process for both the quantizers and the diffusion decoder.
Dual Quantizer Training. As illustrated in Fig. Ta)) a), the semantic quantizer is constructed from two visual branches: vanilla encoder that captures global semantics from low-resolution inputs,
22

---

Vanilla Encoder
Semantic Decader
Vanilla
Encoder
Quanilarr
Becossraccedd
Hiemrchica
Hierachical
Recenstructed
Reconstructed
Pixel E Features
Pixel Features
Pixel
Pixel
Pixel
Diffasion
Decoder
Quantizer
Quantizer
@ Dual Vision Quantizer
(b) Diffusion Decoder
Figure 4: (a) Dual Vision Quantizer: Two separate branches encode semantic and pixel information via quantization and reconstruction. (b) Diffusion Decoder: Reuses pretrained tokens to condition UNet-based denoising for image reconstruction.
Homnrchiett Enzxder
to
Poot Encadar
Vartilla Ercodor
Hlenrhoooa Encoder
Phod Enaoder
Venilla + Enooder
Hicrancicaal + Emooder
Pacd Einonder
Enodeer
Lavriilo X Query ScGated
Leanatic Quey &Caind
Laumnillo Quety &Catad
liat
Ptojestor
Projectox
Texi
Propector
QuasAlletion
Ptojectol
Ptojertrr
Projectir
LLM(QWen2.533B)
LLM(Qwen2.5-33B
LLMIQwes2.5:30
Projecoo
Misk Decodlr
Difflasion Docoder
Projector
Mazk Demader
Liiffunon Decoder
Prijectol
Decodir
Decodrr
Text
Test
Stage I. Pretraining
Stare 2 Supervised
Stage3 Supervised Finetuning of Geveration
Finetuing of Unbestaadling
Figure 5: Three-stage progressive training strategy of FOCUS, Frozen modules are shown in blue, trainable modules in orange. Each stage incrementally activates relevant components to align semantic perception and generative editing.
and a hierarchical encoder that extracts high-resolution spatial features. These streams cross attention to provide rich semantic representations, which are quantized via a Sim VQ [86] module and reconstructed using lightweight transformer as the semantic decoder.
In parallel, a dedicated pixel encoder processes high-resolution images to extract fine-grained, low- level textures. Following the Mo MoVQGAN design (84) its features quantized and decoded via pixel-level decoder. This branch is supervised using LI, perceptual, and adversarial losses.
Each quantizer maintains a separate codebook: 32,768 entries for semantic tokens and 98,304 entries for pixel tokens as (2 The quantizer trained progressively from 256 to 512 x 512 resolution using a bucketed batching strategy, and optimized on corpus of 45M diverse image-text pairs.
Diffusion decoder training. After quantizer training, we train UNet-based diffusion decoder, initialized from SDXL [47, to reconstruct high-resolution images conditioned on the learned dis- crete tokens. As shown in Fig. (b), semantic and pixel tokens are embedded into continuous representations and concatenated with noisy latents guide the denoising process.
To accommodate diverse image shapes, 11 aspect-ratio-specffi canvas sizes predefined: (1:1, 3:4, 4:3, 2:3, 3:2, 1:2, 2:1, 1:3, 3:1, 1:4, 4:1| [21]. and samples with more than 20% content loss after cropping are discarded. The training proceeds in two stages: the first at 512 512 resolution and the second at 1024 x 1024 for super-iesolution refinement. During this stage, all encoders and codebooks are frozen, and only the diffusion decoder updated.
Progressive Large Vision Language Model Training
A core challenge in building a unified large vision language model lies in the optimization conflict between high-level semantic understanding language and low-level visual synthesis in image generation. FOCUS addresses this by adopting progressive three-stage training paradigm (Fig.
23

---

that incrementally activates model components to align pixel-level perception and controllable image generation.
Languae Projector Warmup. This stage establishes initial alignment between visual features and the large language model (LLM). We train only the projection heads of the vanilla and pixel encoders, along with the learnable queries and gated cross-attention modules in the hicrarchical encoder. All backbone encoders, the LLM, the mask decoder Mask2Former [14), and the diffusion decoder are frozen. Training is performed on 256 256 image-text pairs using a standard language modeling objective, without any segmentation or generation supervision.
Multimodal Pretraining with Segmentation-Aware Alignment. We activate and jointly train the LLM, Sauulllanguage adapters, and the mask decoder to enhance multimodal understanding and segmentation capability. The diffusion decoder remains frozen. Supervision includes token- level cross-entropy, segmentation losses (Dice and BCH), and visual reconstruction losses using precomputed features. Training is conducted at both 256 and 512x512 resolutions on datasets covering image-text alignment, referring segmentation, and multimodal reasoning.
Instruction Tuning for Region-Controlledd Editing. This stage focuses on segmentation perception and controllable generation for LVLMs. We instruct tuning the LLM, the mask decoder, and the cross- attention layers in the diffusion decoder, while freezing all visual encoders. Images are processed at resolutions up to 1024x 1024. Segmentation outputs used spatial guidance and injected into the diffusion decoder's 's attention layers. Supervision includes text generation loss, segmentation loss, and 1.2 loss between denoised outputs and targets.
Table 8: Dataset distribution across training stages in FOCUS
Stage
Number
Task
Source
Stage Visual Quantizer and
Diffusion Pretraining
45M
Image to- lo-mmage
COYO EMOVA LAION
Stuge II Visual- Language
Projector Warmup
30M
Image- to- Text
LLaVA-150K COYO, EMOVA
35M
Image-to- Text & Editing
Ulirabdii [3], Anybdit [77] SEED-EOR [5
RcCCCO-SSetis V6 4H) RerCler [6]
Paco-LVIS [49). PartimaeeNet 20
Segmentation
Stage III: Multimodal Pretraining with Generative and Stractural Signals
DAVIS-2017 (6. Pascal-Part [2]
YouTube-V1S2019 [?]]
Magnie [9 OpenOrca k2
SM
Dialog / QA
SCP-116K [2] Openleermes 58
OPC-SPT-Stage
22
EMOVA-SFI. Pixmo [17] M4-Instruct [36]
Image Editing
OmniFdii [55 AnyFddit,
Ultraidit, InstructPsx2PPx 4 Maglie
Stage IV: Instruction Tuning For Controllable Editing
ReasonSeg Lasu+ Inst. Sog. Col [73
2M
Segmentation
ReVOS 70 Ref- Ref-Youmbo--OS 52
0.5M
Interactive Editing
COCO-Interactive 82
C Progressive Dataset Structuring
Each stage in FOCUS adopts dedicated datasets aligned with its training objectives, rather than relying on a uniform corpus across all phases. The dataset distribution and task assignments are summarized in Table
Stage 1: Visual Quantizer and Diffusion Pretraining. This stage focuses on learning discrete representations for both semantic abstraction and fine-grained reconstruction. We utilize 45M image- text pairs from COYO, EMOVA, and LAION-2B. COYO contributes OCR-rich, language-aligned samples; EMOVA provides aesthetic-oriented captions; and LAION-2B enhances domain diversity. These datasets collectively support the training of dual visual quantizers and the diffusion decoder.
Stage 2: Visual-Language Adapter Warmup. To establish foundational alignment between visual embeddings and the language model, we train on 30M image-text pairs from COYO, EMOVA- Pretrain, and LLLVA-150K. These samples span natural scenes, documents, and instruction-follwwee captions. This stage optimizes only the projection heads and adapter modules, using language modeling loss while keeping all backbone components frozen.
Stage 3: Multimodal Pretraining with Generative and Structural Signals. This stage enhances the model's s ability to generate, localize, and reason across modalities. We leverage 35M samples from UltraEdit, SEED-Edit and AdyEdit for text-guided editing. An additional 3M segmentation
24

---

2.0% 5.0%
11:38
18.7%
40.4%
8.4%
12.1%
17.8%
16.0%
100.0%
10.7%
7.1%
21.7%
(sal-Bolennner3335 Liffesivn(L00X)
Stopell Stagelli (73ND
MILM SET (1.MD
Acstheea
MotlySktencee
(033, Charc) (Math Sceerc)
Nurti-inape
Figure Summary of the data mixture in each stage. Our training data gradually covers a wide range of tasks and various image resoluton.
annotations are sourced from RefCOCO-series, RefClef, and video datasets like DAVIS-2017 and YouTube-VIS2019 For dialogue and question answering, include 5M samples from Magpie, OpenOrca, SCP-116K, OpenHermes, and OPC-SFT-Stagel.. This stage jointly supervises multimodal reasoning, structural grounding, and fine-grainee visual tasks.
Stage 4: Instruction Tuning for Controllable Editing. The final stage fine-tunes the model for spatially guided, instruction-based editing. We use 7M samples from EMOVA-SFT, Pixmo, M4- Instruct, OmniEdit, AnyEdit, UltraEdit, InstructPix2Pix, and Magpie for complex editing tasks. We incorporate 2M segmentation-centric samples from ReasonSeg, Lisa++, and video segmentation sets like RVOS and RefYoutubeVOS, Additionally, COCO-Interactive provides 0.5M samples for interactive region-level editing. This stage improves instruction following, multi-object control, and editing consistency across visual contexts.
D Training Configurations Across Different Stages
Table 9: Training hyperparameters across different stages in FOCUS.
Settings
Visual Quantizee (Tokenizer)
DifTasiin Decoder (Image Resonstructio)
Projector Warmup (Projector Wanmu)
Multimodal Pretraining
Instruction Tuning
(Seg. Pretmin)
(Instraction Tuning)
Leaming Rate
1o-4 (sementic) 2e-4 (pisel)
2e-5
le-3
20-5 (Vinnal esender, LLM)
Me-k Decoder)
2--6 (Mask Decoder. Dillusion)
2e-5 (Visual enender: 1. L.
Training Steps
28k (semantic)
136k (pisel)
2208 All Allencoders Codebooks
lepoch
3epoch
lepuch
Image
1024
Frozen Modules
Hierarchical Encoder
Vinilla encoder
Mask Decoder, Diffusion
Visual Fncoder,
Diffisson
Visual Encuder
We adopt spage-specific training configurations to align with the objectives and resolution requirements of each module, as summarized in Table
Dual Visual Tokenizer using fixed learning rate of le-4 and batch size of 256. The training follows three progressive resolution stages: 136k steps 256x256, 28k steps 512x512. The semantic branch is optimized with cosine similarity loss, while the pixel branch is trained with combination of L1, perceptual, and adversarial losses.
Diffusion Decoder using a learning rate of 2e-5 and batch size 128 for 265k steps. Training employs multi-aspect-ratio cropping up 1024x1024 resolution. All visual encoders and tokenizers are frozen, and supervision is provided through 1.2 reconstruction loss.
LVLM Training is conducted in three distinct stages.
25

---

Projector Warmup: Only the projection heads and gated visual adapters are trained using learning rates of 1e-3 (projectors), with batch size of 512 at 256 x 256 resolution for epoch.
Multimodal Pretraining: The visual encoder, large language model and mask decoder optimized jointly. Phase one uses 256 x 256 resolution with a batch size of 256 for epoch, and phase uses 512 512 with batch size of 128 for epochs. A learning rate of 2e-5 is used throughout, and le-3 3 is used for the mask decoder.
Instruction Tuning:
The LLM, segmentation decoder, and diffusion model's attention layers are fine-tuned with learning rates of 2e-5 (LLM) and 2e-6 (mask decoder, diffusion), with batch size of 256 for epoch. Training involves input resolutions randomly sampled from 512 x 512 to 1024 x 1024 using bucketed cropping.
E Metrics.
We evaluate FOCUS across three core tasks using standard benchmarks and metrics. For multimodal understanding, we report accuracy on POPE (29]. MM- Vet [78]. MMBench [40], SEED [26]. and MMMU [79] to assess the model's ability in Asion-language reasoning, classification, and grounding. For referring segmentation, we use mean Intersection over Union (mloU) on RefCOCO, RefCOCO+, RefCOCOg. and gRefCOCO, measuring the overlap between predicted masks and ground truth under language prompts. For controllable image generation and editing, we evaluate fidelity using FID) (Fréchet Inception Distance), and alignment using CLIP-basee scores (CLIP-I, CLIP-T, CLIP-DIR to assess visual consistency and instruction compliance. Additional breakdowns from GenAI-bench and GenEval are used for fine-gun ene-ggained control metrics such as object count, color, and spatial placement.
F
Unified Segmentation Capability of FOCUS
FOCUS is designed as a unified model capable of handling diverse segmentation tasks spanning spatial, temporal, and interactive modalities. We evaluate its generalization and adaptability across five key segmentation scenarios, each highlighting a distinct dimension of its fine-grained visual perception.
F1 Contextual Reasoning and Referential Understanding
We evaluate the ability FOCUS perform segmentation under semantically complex and referential conditions using the ReasonSeg and ReVOS benchmarks. These tasks require the model to accurately localize and segment objects based on contextual or linguistic descriptions with varying temporal spans. As shown in Table I0 FOCUS achieves leading performance across both benchmarks. On ReasonSeg, it obtains a gloU of 62.1 and a cloU 58.6, surpassing prior methods in semantic segmentation precision. On ReVOS, FOCUS ranks first in all reasoning and referring metrics, including J&F of 57.2 reasoning and 58.9 referring, validating its unified capability both spatial understanding and temporal reference tracking.
Table 10: Performance on Referring Video Object Segmentation (Re and ReasonSeg bench-
marks. Bold: best; Underlined: second- best.
ReVOS Reasoning
ReVOS-Re Relerring
ReVOS-Overal
ReasonSeg gloU
Method
Backbone
J&F
I&F
J&F
cloU
LMPM
Swin-T
13.3
24.3
18.8
29.0
39.1
34.1
21.2
31.7
26.4
ReferForner
Video-Swin-B
21.3
25.6
23.4
31.2
34.3
32.7
26.2
29.9
28.1
LISA-7B
VIT-H
33.8
38.4
36.1
44.3
47.1
45.7
39.1
42.7
40.9
52.9
54.0
LaSagnA-7B
VIT-H
48.8
47.2
Efficient
SAM4MLLM-7B
ViT-SAM-XLI
46.7
48.1
TrackGPT-13B
VIT-H
38.1
42.9
40.5
48.3
50.6
49.5
43.2
46.8
45.0
VISA-7B
VIT-H
36.7
41.7
39.2
51.1
54.7
52.9
43.9
48.2
46.1
52.7
57.8
VISA-13B
VIT-III
38.3
43.5
40.9
$2.3
55.8
54.1
45.3
49.7
47.5
HyperSeg-3B
Swin-B
50.2
55.8
53.0
56.0
60.9
58.5
53.1
58.4
55.7
59.2
56.7
FOCUS
ConvNexta
51.6
56.3
57.2
56.8
61.0
58.9
54.3
59.1
56.7
62.1
58.6
26

---

F.2 User-Guided Interactive Perception
To assess FOCUS's adaptability to user-driven segmentation, we evaluate it on the COCO-Interaction benchmark under four input modes: box, point, mask, and scribble. These interactions simulate real-time editing scenarios where users specify partial object regions.
As summarized Table 11 score of 83.4 and point score of 78.6, demonstrating robust generalization across sparse and dense cues. This consistent performance under varied interaction types highlights FOCUS's potential for responsive and controllable editing applications.
FOCUS achieves the highest loU across all modalities, including a box
Table 11: Performance on the Backbone
benchmark. IoU (%) under different imput modalities.
Method
Box
Scribble
Mask
Point
SAM
VIT-B
68.7
33.6
SAM
VIT-L
71.6
37.7
SEEM
DaViT-B
42.1
44.0
65.0
57.8
PSALM
Swin-B
80.9
80.0
82.4
74.0
HyperSeg
Swin-B
77.3
75.2
79.5
63.4
FOCUS
ConvNext-L
83.4
82.7
85.2
78.6
F.3 Temporal Consistency in Dynamic Scenes
To evaluate FOCUS's capability in video-level segmentation, we benchmark on both Video Object Segmentation (VOS) and Video Instance Segmentation (VIS) tasks. These tasks test the model's ability to consistently track and segment objects across time, even in the presence of occlusions, motion blur, viewpoint shifts.
Table 12 shows that FOCUS achieves top performance across all datasets, including a J&F score of 79.1 DAVIS17 and mAP of 65.7 on YouTube-VIS. Its strong performance both generic and referring-basee video benchmarks confirms robustness temporally consistent, instance-aware segmentation.
Table Video segmentation results across DAVIS17 (VOS), Ref-YT, Ref-DAVIS (R-VO VOS), and YouTube- VIS (VIS). Bold: best; Underlined: second-best.
Method
Backbone
DAVISI7 (J&F) Ref-YT (J&F)
Ref-DAVIS (I&F)
YT-VIS (mAP)
SEEM
DaViT-B
62.8
OMG-Seg
ConvNeXt-L
74.3
56.4
ReferFormer
Video-Swin-B
62.9
61.1
OnlineRefer
Swin-L
63.5
64.8
UNINEXT
ConvNeXt-L
77.2
66.2
66.7
64.3
LISA-7B
ViT-H
53.9
64.8
VISA-13B
VIT-H
63.0
70.4
VideoLISA-3.8B
VIT-H
63.7
68.8
HBperSeg-3B
Swin-B
77.6
68.5
71.2
53.8
FOCUS
ConyNeXt-L
79.1
69.3
72.4
65.7
G The role of the segmentation mask
Table 13: Ablation study on the effect of segmentation mask guidance on Emußdit benchmark.
EmuEdit
Method
DINO(T)
CLIP-I(T)
CLIP-T(T)
CLIP-DIR(()
No Mask Guidance
0.812
0.866
0.268
0.113
Using General- Mask
0.819
0.868
0.271
0.108
FOCUS(Ours)
0.826
0.872
0.275
0.101
To highlight the importance segmentation masks our framework, we design three comparative experiments. FOCUS (our full model) utilizes internally generated masks precisely aligned with
27

---

language instructions. Baseline removes mask guidance entirely, relying solely text instructions. Baseline 2 employs masks from an external general purpose segmentation model (Groundng+SAM22 These experiments demonstrate that precise, instruction-allgned masks are essential for optimal editing performance.
H Visualization of Generation Quality
present a set images generated by FOCUS based on natural language prompts (see figure 7). These results span various styles including realistic scenes, conceptual designs, and artistic illustrations. They demonstrate the model's ability to produce high-quality and semantically consistent outputs.
Each image is generated solely from text input without using any visual masks or interactive guidance. The visualizations confirm FOCUS's strength in aligning language instructions with fine-graine visual content, showcasing both fidelity and controllability.
ased designed of cadiee ruite. in decualent focest, blockiti bright buttorflios
Ploase create skenched Equze of Phillip listering music in mall slte photo.
7 want nendere painting of and perthed branch the woods.
Besntiful landscape photogmaphs, sunmer Indoresia
Desutifu survel sytbolism the mesmstiinn vasun a Clespeira Queest of Egypt mesnenzznng beown eyes, blacs helr and itheral fistures rillating celestaal lize phstocraphyy composttngg authentte, sty st000
Besl photo of cup of steaming codioe and hrass vase with lurge banquet of spring clowess an oud oak window it surrise, fine detarls, rich colers taken with ailonn camera and shumer spoed of 1400 lotor. UHD dim HDR dk.
Aschitestrrl persmetric pevilioa made Crumn wood and glass, with organtic canities, sucounded by bocutiul forest. Thaaattc sonne, photowalietie hypereelltii, tracing rellections intracato citail 1Joyde Wright
detailed high resiutiin photogmmia of caplrriing cylergank girl with vilbratt
pank har locking intendly at the camen she sands confidoetly husting
cyberpunk town. The colors include hold blurs, and purples, neon highlights.
Frink
beight
Create Cigurs A full biome glanst enclused insiće bottle, with detalls lighting. gresented ulies coacept
Tiny adumble mosse dresad kiop castie unchoponurpiicc Monga, cinematic lighting. pontait, style
Can generste skutched drewing of
Creste a saxte Af vihnas and nolare- inspired lily flower grows out of the streets of Becoidyn, with the socni Brooklya Bridge serving broatttaking backdrop.
Elren Frincess sumounded malistic watercofour
depected
style
Figure 7: Visual examples of image generation results from FOCUS given only natural language prompts. The model produces diverse and high-fidelity outputs across a range of styles and scenes, demonstrating strong semantic alignment and visual controllability.
28

---

I Controllable Image Editing via Segmentation Guidance
To further demonstrate the controllability of FOCUS localized image editing, we visualize repre- sentative examples where the model edits specific regions based on natural language instructions. As shown in Figure X the model takes an input image and user-provided instruction describing the desired modification. It first predicts segmentation mask that identifies the target region referenced the text, then uses this mask to guide the generation an edited output.
This pipeline illustrates the effectiveness of FOCUS in grounding language to spatial regions and applying precise, content-aware modifications. The examples cover diverse scenarios including object transformation, replacement, and contextual adjustment. The results validate that segmentation-awee diffusion guidance enables fine-grained, localized editing while preserving the global scene structure.
Table 14: Prompt schema design for different vision- saiinnlanguage tasks. Each row shows how FOCUS
handles a specific task by pairing a general instruction (task prompt) with task susk-specific condition. This formulation supports unified handling of segmentation, generation, and editing tasks.
Task Type
Task Instruction Prompt (SI)
Condition Prompt (SC)
Class-based Segmenlation
Please segment all the posilive objects
["person", "do", "car", "tzee", ....
by the following candidate cutegories.
Referring Segmentation
Please segment targel referred by the language description.
"The man wearing a red sLanding beside the yellow ca-."
Reasoning Segmentation
Please segment the targel referred to
"The object that the man is reaching for in the office." " Fooled CLIP region leatures (c.g. clicks, scrib- bles, boxes)
by the reasoning-buse bused description.
Interactive Segmentation
Please segment according the given visual reference regions.
Image Generation
Please generate image accurding to following description.
"A tiny brown dog with white patches, eagerly holding a
blue black Frisbee." black jacket with woman in the same pose."
Image Editing
Please edit the image according to the
"Replace the
following instruction.
J
Prompt Design for Multi-Task Vision-Language Modeling
To support a wide range of vision-language tasks within a unified framework, FOCUS adopts a structured prompt schema consisting of two components: a task instruction prompt and condition prompt in Table. [4] The task instruction prompt defines the model objective in natural language, such segmentation, generation, editing. The condition prompt provides task-specific contextual information, such as category labels, referential deseriptions, or visual cues.
This design enables the model to flexibly adapt to diverse tasks including class-based segmentation, referring and reasoning segmentation, interactive segmentation with visual cues, text-to-imaae generation, and fine-grained image editing. By standardizing task formulation through prompt schema, FOCUS achieves better generalization across modalities and applications.
29

---

Input Image
User instrcution
Mask prediction
Edit generation
"Change the corgi a
"Edit the cupboards to bookshelves"
"Transform the green apple to orange."
"Replace the cat to the fox." "
"Adjust the dog in the jungle"
Figure 8: Visualization of controllable image editing results. Given an input image and a user instruction. FOCUS first predicts spatial mask corresponding the referential target, then performs localized generation to edit the specified region. The examples demonstrate accurate region identifi- cation and high-fidelity edits aligned with the instruction.
30


Reviews:
[{"review_id": "0Ititjh4wI", "reviewer": "Reviewer_WJ6V", "summary": "The paper present FOCUS, a unified vision-language model that tackles the problems of segmentation-aware image understanding (high-level) and image generation (low-level).  The authors propose a novel dual-branch visual encoder composed of two encoders that extract the fine (ConvNext) and coarse-grained (QwenViT) information from the input image. Additionally, FOCUS also utilises a visual tokeniser for image generation, along with continuous tokens, to account for the loss of information that occurs during the quantisation process. The continuous tokens provide fine-grained details to the LLM for image understanding tasks. The LLM outputs the mask embeddings, prompt embeddings, image quantizer embeddings, and the semantic recognition tokens. The mask embeddings and the prompt embeddings are used in the mask decoder module for the referring object segmentation task, while the quantised image embeddings, along with decode masks, are used in the diffusion decoder for image generation and editing tasks. The models are trained in multiple stages (4) to learn referring object segmentation and image generation tasks. FOCUS is evaluated on various tasks (image generation, image understanding, image editing, and referring object segmentation) for multiple datasets, showing performance improvements over existing methods on all these tasks.", "questions": "1. How does the dataset size affect the performance of these models? Unified understanding and generation performance. Are the potential improvements due to using larger training data, or does the design decision remain unclear? \n2. The authors resize the segmentation masks to the latent spatial resolution. This resolution is much smaller than the full image resolution. How does this affect the editing and generation of objects that are very small in size? \n3. Major ablations (see weakness section) are missing. The authors need to quantify the impact of each component of the model architecture and training. For example, what is the impact of the decoding constraints the authors impose when decoding the tokens from the LLM? These would help us understand the impact of the work further", "limitations": "The paper does not have a dedicated limitation section (main paper and supplemental), which the authors say they provide in the checklist. \n1. One limitation I see is in editing small objects, as the masks are downsampled to the latent spatial resolution. \n2. Another limitation I see is that replicating the work is challenging due to the high computational resources required for training the model. \n3. Due to reliance on SDXL for editing the models also inherits the limitation of these models, For example, in Figure 5 is the models are able to make edits which change the spatial relations between the objects For example, in image 5: the Prompt “make the cat sit on top of dogs head”", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "- Strengths:\n    1. The paper is well written. The motivation for the problem the authors are solving and how it differs from other VLM models is explained well.\n    2. The methodology is well explained. The authors clearly explain everything, and the details about the method are well presented.\n    3. The paper presents impressive results and outperforms many baseline methods in image understanding (across multiple datasets), image generation, referring segmentation, and region-level editing tasks.\n    4. The paper introduces a novel dual-encoder module that combines high- and low-level features of images required to solve joint image understanding and generation tasks.\n    5. The paper demonstrates good results for temporal consistency in tracking and segmenting objects across the video, as shown in Table 5 of the appendix.\n\n- Weakness:\n    1. Missing ablations that show the impact of each change in the paper. For example,\n        - What is the effect of multi-stage training, which gradually increases the image sizes and task complexity?\n        - What is the effect of increasing or decreasing the number of multi-scale features in the gated cross-attention module? It would be good to see qualitative and quantitative results here.\n        - The effect of using the continuous tokens in the visual tokeniser.\n        - The effect of putting constraints on the decoding strategy in LLM.\n    2. The paper claims to have a dedicated limitations section, discussing the limitations of the work, which I was unable to find.\n    3. There are several writing issues in the experiment section:\n        - No citation for dataset used in Table 1, eg, POPE, MMBench, etc. All datasets or works that the authors use in their research should be properly cited. The authors have citations for these in the supplementary material, but not in the main paper.\n        - Table 2 has incorrect bolding of numbers in multiple cases; similarly, Table 3 lacks bolding altogether.\n        - The authors do not provide citations to FID, MJHQ metrics, etc. The authors state in the checklist that all datasets and code used are cited, but this is not the case.\n  4. One minor point regarding the writing for me is that the paper places some important details and visualisations (Figs. 1 and 2) in the supplementary section. If these details were in the main paper, it would make understanding the methodology much easier.", "was_revised": true, "final_justification": "I have read the final responses from the author and their rebuttal. I would be happy to keep my original rating of accept for this paper.", "rebuttal": "**Regarding Weakness 1 & Question3 (Lack of ablation studies):**\n\n*   **The effect of multi-stage training:**\n\n    The following table shows the results of using different image resolutions (256, 512, and 1024) for all-stage task training. We observe a clear trend: higher image resolutions improve the model’s performance in image generation and segmentation, but lead to a decline in language understanding. Our analysis suggests that this decline is mainly due to an increase in hallucination issues at higher resolutions—when processing high-resolution images, the model is more likely to generate text that is inconsistent with the visual input. This increased hallucination negatively impacts the model’s language comprehension, highlighting the challenge of balancing visual precision and robust language understanding in multimodal models.\n\n    To address this issue, our multi-stage training strategy starts with simpler tasks at lower image resolutions and gradually increases both the image size and task complexity in subsequent stages. This progressive approach enables our model to achieve strong and balanced performance across all tasks.\n\n    Table 1: The effect of different image resolutions and multi-stage training strategy on model performance across various tasks. ↑ indicates higher is better, ↓ indicates lower is better.\n    | Image Size | MJHQ30K | GenAI-Bench |         | Image Understanding |         |         | RefCOCO  |          |          |\n    | :--------- | :------ | :---------- | :------ | :------------------ | ------- | ------- | -------- | -------- | -------- |\n    |            | FID(↓)  | Basic(↑)    | Adv.(↑) | POPE(↑)             | MMB(↑)  | SEED(↑) | testA(↑) | TestB(↑) | Valid(↑) |\n    | 256        | 11.85   | 0.63        | 0.56    | 85.3                | 73.1    | 70.1    | 78.5     | 79.2     | 75.4     |\n    | 512        | 7.39    | 0.70        | 0.61    | 87.3                | 80.6    | 71.3    | 81.3     | 82.4     | 79.8     |\n    | 1024       | 6.03    | 0.76        | 0.69    | 77.6(↓)             | 68.2(↓) | 68.8(↓) | 83.4     | 84.9     | 81.5     |\n    | 256->1024(Our)        | 6.05    | 0.83        | 0.72    | 88.0                | 81.5    | 73.9    | 84.1     | 86.3     | 82.7     |\n\n*   **The effect of multi-scale features in Gated Cross-Attention.**\n\n    To effectively fuse multi-scale features, our model employs a Gated Cross-Attention module to process outputs from the ConvNeXt-L backbone, achieving a balance between fine-grained detail and global contextual understanding. Specifically, we design a **3-layer** Gated Cross-Attention adapter that hierarchically integrates three distinct feature scales: the first layer focuses on fine features ($f_2$), the second on mid-level features ($f_3$), and the third on coarse semantic features ($f_4$).\n\n    As illustrated in the table, we compare our three-scale model ($f_2$, $f_3$, $f_4$) with variants that utilize only a single scale ($f_4$), dual scales ($f_3$, $f_4$), and all four scales ($f_2$–$f_4$). The results (mIoU on RefCOCO) demonstrate that our multi-scale fusion strategy is essential for accurately segmenting fine details, providing the optimal balance between performance and computational efficiency.\n\n    Table 2: Ablation study on the effect of multi-scale feature fusion in Gated Cross-Attention.\n\n    | Model Variant              | #Scales | RefCOCO mIoU(↑) | MMBench Acc.(↑) | CLIP-T(Edit)(↑) |\n    | :------------------------- | :------ | :-------------- | :-------------- | :-------------- |\n    | FOCUS-SingleScale (Fine)   | 1       | 79.5            | 72.8            | 0.265           |\n    | FOCUS-SingleScale (Coarse) | 1       | 78.2            | 73.9            | 0.260           |\n    | FOCUS-DualScale            | 2       | 80.5            | 73.5            | 0.270           |\n    | FOCUS (Ours)               | 3       | 81.4            | 73.9            | 0.275           |\n\n*   **The effect of continuous visual tokens.** Our ablation study demonstrates that feeding **continuous visual features** directly into the LLM is essential for optimal performance. By bypassing discrete tokenization, this approach effectively reduces **information loss** and retains **fine-grained** visual details. As a result, the LLM receives a richer and more precise visual representation, which significantly enhances its ability to comprehend complex multimodal scenarios and accurately interpret referential instructions.\n\n    Table 3: Ablation study on the effect of continuous visual token input on model performance.\n\n    | Continuous Input | MJHQ30K | GenAI-Bench |         | Image Understanding |        |         |\n    | :--------------- | :------ | :---------- | :------ | ------------------- | ------ | ------- |\n    |                  | FID(↓)  | Basic(↑)    | Adv.(↑) | POPE(↑)             | MMB(↑) | SEED(↑) |\n    | **×**                |  6.85   |  0.68       |  0.65   | 82.1                | 50.4   | 56.0    |\n    | **√**               |  6.05   |  0.83       |  0.72   | 85.3                | 70.9   | 66.6    |\n\n*   **The effect of LLM decoding constraints**: The results in the table demonstrate the effectiveness of our decoding constraint, which requires the model to first identify object categories before generating segmentation masks. This structured decoding process not only enhances semantic consistency and reduces the risk of arbitrary segmentation from ambiguous instructions, but also leads to a significant improvement in final segmentation accuracy.\n\n    Table 4: Ablation study on the effect of LLM decoding constraints on segmentation performance.\n\n    | **constraints** | YT-VIS (mAP) | RefCOCO (cIoU) |\n    | :-------------- | :----------- | :------------- |\n    | **×**               | 50.8         | 83.5           |\n    | **√**               | 54.6         | 84.1           |\n\n\n**Regarding Weakness 2 & Question2 (Lack of Limitations Discussion)**: Thank you for your feedback. We apologize that the 'Limitations' section, though mentioned in our submission checklist, was omitted from the main manuscript. We will add the limitation: **(1)** the limited precision when editing small objects due to latent space downsampling; **(2)** the high computational cost, which challenges replication; and **(3)** inherited weaknesses from SDXL, such as rendering complex spatial relationships, which we frame as a direction for future work.\n\n**Regarding Weakness 3\\&4 (Writing Issues in the Experimental Section)**: Thank you for your suggestions and feedback. We will revise the manuscript to further improve the clarity and completeness of the paper. First, we will add citations in the main paper for all datasets (e.g., POPE, MMBench) and evaluation metrics (e.g., FID, MJHQ) to ensure proper attribution. Second, we have corrected the incorrect bolding in Table 2 and added appropriate bolding in Table 3 to accurately highlight the best-performing results. Finally, we will move Figures 1 and 2 from the supplementary material into the main paper to enhance clarity and readability.\n\n**Regarding Question1**:\n\nThe progressive training paradigm we adopt, by elevating task complexity and resolution in stages, has at its core the optimization of data utilization strategy and effectiveness, rather than a reliance on the expansion of data scale."}, {"review_id": "swriP970II", "reviewer": "Reviewer_9FR4", "summary": "This paper proposes FOCUS, a large MLLM that is capable of both image understanding and image editing. The proposed architecture uses predicted segmentation masks for more precise image editing, and learn these ablities through end-to-end training.", "questions": "1. I would like to suggest the authors to provide careful ablation studies on the key components of their models. For example, would the image editing get worse without the predicted segmentation masks or with inaccurate segmentation masks?\n\n2. Since the model uses mainly human-annotated segmentation data for supervision, I wonder how the segmentation decoder performs on OOD data, and how would it affect the image editing process?", "limitations": "Yes.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 3, "strengths_and_weaknesses": "- Strengths:\n\n1. The motivation and idea of using predicted segmentation masks for better controlling precise image editing is reasonable.  \n2. The dual encoders is carefully designed to facilitate catching both global and fine-grained information.\n3. The overall performance of the proposed method compared to the provided baselines is good.\n\n- Weaknesses:\n\n1. The model uses a large amount of human-annotated segmentation training data, which is hard to scale-up.\n2. Lack of carefully ablation studies of the proposed key components. \n3. Lack of discussions and comparisons on existing works which combine LLM and specialized decoders for generating heterogeneous data such as [1-4].\n\n[1] NExT-GPT: Any-to-Any Multimodal LLM, on ICML'24.\n[2] VisionLLM: Large Language Model is also an Open-Ended Decoder for Vision-Centric Tasks, on NeurIPS'23.\n[3] VisionLLM v2: A Generalist Multimodal Large Language Model for Hundeds of Vision-Language Tasks, on NeurIPS'24.\n[4] Emerging Properties in Unified Multimodal Pretraining, technical report (optional, since it is not officially published).", "was_revised": true, "final_justification": "The authors has addressed most of my concerns. As stated in the initial review, the motivation and idea of using predicted segmentation masks for better controlling precise image editing is reasonable, and the proposed method demonstrates its effectiveness on several benchmarks. However, the proposed method in the manuscript only uses human-annotated data, although the semi- or un-supervised segmentation labels should be able to be used to scale up the training. Therefore, I recommend \"borderline accept\".", "rebuttal": "Dear Reviewer, Thank you for your review and the insightful questions you've raised about our work. Your concerns have helped us to more clearly position the value of our contributions.\n\n**Regarding Weakness 1 & Question 2 (Dependence on manually annotated segmentation data and OOD performance)**\n\nOur method should be extensible and employs a highly scalable strategy that eliminates the need for manual annotation by utilizing an automated data generation pipeline. This pipeline combines GroundDINO for object localization, SAM2 for precise segmentation, and a Large Vision-Language Model (LVLM) for further refinement, enabling the efficient production of high-quality training data.\n\nMeanwhile, our model (FOCUS) achieves state-of-the-art (SOTA) performance on out-of-distribution (OOD) benchmarks, including ADE-OV, Citys-OV, PC59-OV, and PAS20-OV. This outstanding performance is largely due to the integrated LLM’s advanced language understanding capabilities for visual grounding, which enable robust results even when processing unfamiliar instructions across a wide range of scenarios.\n\nTable 1: Comparison of segmentation performance on out-of-distribution (OOD) benchmarks.\n\n| Method   | ADE-OV |      | Citys-OVs | PC59-OV | PAS20-OV |\n| :------- | :----- | :--- | :-------- | :------ | -------- |\n|          | PQ     | mIoU | PQ        | mIoU    | mIoU     |\n| PSALM    | 13.7   | 18.2 | 28.8      | 48.5    | 81.3     |\n| HyperSeg | 16.1   | 22.3 | 31.1      | 64.6    | 92.1     |\n| FOCUS(Ours)    | 18.5   | 25.7 | 33.4      |  67.3   |  93.8    |\n\n**Regarding Weakness 2 Lack of ablation studies**\n\n*   **The effect of multi-stage training:**\n\n    The following table shows the results of using different image resolutions (256, 512, and 1024) for all-stage task training. We observe a clear trend: higher image resolutions improve the model’s performance in image generation and segmentation, but lead to a decline in language understanding. Our analysis suggests that this decline is mainly due to an increase in hallucination issues at higher resolutions—when processing high-resolution images, the model is more likely to generate text that is inconsistent with the visual input. This increased hallucination negatively impacts the model’s language comprehension, highlighting the challenge of balancing visual precision and robust language understanding in multimodal models.\n\n    To address this issue, our multi-stage training strategy starts with simpler tasks at lower image resolutions and gradually increases both the image size and task complexity in subsequent stages. This progressive approach enables our model to achieve strong and balanced performance across all tasks.\n\n    Table 2: The effect of different image resolutions and multi-stage training strategy on model performance across various tasks. ↑ indicates higher is better, ↓ indicates lower is better.\n    | Image Size | MJHQ30K | GenAI-Bench |         | Image Understanding |         |         | RefCOCO  |          |          |\n    | :--------- | :------ | :---------- | :------ | :------------------ | ------- | ------- | -------- | -------- | -------- |\n    |            | FID(↓)  | Basic(↑)    | Adv.(↑) | POPE(↑)             | MMB(↑)  | SEED(↑) | testA(↑) | TestB(↑) | Valid(↑) |\n    | 256        | 11.85   | 0.63        | 0.56    | 85.3                | 73.1    | 70.1    | 78.5     | 79.2     | 75.4     |\n    | 512        | 7.39    | 0.70        | 0.61    | 87.3                | 80.6    | 71.3    | 81.3     | 82.4     | 79.8     |\n    | 1024       | 6.03    | 0.76        | 0.69    | 77.6(↓)             | 68.2(↓) | 68.8(↓) | 83.4     | 84.9     | 81.5     |\n    | 256->1024(Our)        | 6.05    | 0.83        | 0.72    | 88.0                | 81.5    | 73.9    | 84.1     | 86.3     | 82.7     |\n\n*   **The effect of multi-scale features in Gated Cross-Attention.**\n\n    To effectively fuse multi-scale features, our model employs a Gated Cross-Attention module to process outputs from the ConvNeXt-L backbone, achieving a balance between fine-grained detail and global contextual understanding. Specifically, we design a **3-layer** Gated Cross-Attention adapter that hierarchically integrates three distinct feature scales: the first layer focuses on fine features ($f_2$), the second on mid-level features ($f_3$), and the third on coarse semantic features ($f_4$).\n\n    As illustrated in the table, we compare our three-scale model ($f_2$, $f_3$, $f_4$) with variants that utilize only a single scale ($f_4$), dual scales ($f_3$, $f_4$), and all four scales ($f_2$–$f_4$). The results (mIoU on RefCOCO) demonstrate that our multi-scale fusion strategy is essential for accurately segmenting fine details, providing the optimal balance between performance and computational efficiency.\n\n    Table 3: Ablation study on the effect of multi-scale feature fusion in Gated Cross-Attention.\n\n    | Model Variant              | #Scales | RefCOCO mIoU(↑) | MMBench Acc.(↑) | CLIP-T(Edit)(↑) |\n    | :------------------------- | :------ | :-------------- | :-------------- | :-------------- |\n    | FOCUS-SingleScale (Fine)   | 1       | 79.5            | 72.8            | 0.265           |\n    | FOCUS-SingleScale (Coarse) | 1       | 78.2            | 73.9            | 0.260           |\n    | FOCUS-DualScale            | 2       | 80.5            | 73.5            | 0.270           |\n    | FOCUS (Ours)               | 3       | 81.4            | 73.9            | 0.275           |\n\n\n**Regarding Weakness 3 (Comparison with works like NExT-GPT, VisionLLM, etc.):**\n\nPrevious frameworks such as NExT-GPT and VisionLLM v1 & v2 have made progress toward unifying vision tasks, but their reliance primarily on text conditioning limits their ability to achieve precise spatial control in image editing. In contrast, our method stands out for its pixel-aware design: by utilizing the edge information of the target region, our approach enables highly accurate and localized modifications.\n\nThis advantage is consistently reflected across multiple benchmarks. **Tables 6 and 7** showcase our superior performance in text-guided video editing (DAVIS) and image editing (COCO-caption), respectively. Furthermore, as illustrated in **Table 8**, our model achieves segmentation accuracy on par with state-of-the-art (SOTA) methods such as VisionLLMv2, while uniquely offering robust editing and generation capabilities.\n\nWe also recognize Bagel, a concurrent work with promising results, as an important reference point. We believe that future progress can be achieved by drawing on the strengths and insights of both approaches.\n\nTable 6: Quantitative comparison of text-guided video editing methods on DAVIS.\n\n| Method    | CLIP-T(↑) | CLIP-I(↑) |\n| :-------- | :-------- | :-------- |\n| CogVideo  | 0.2391    | 0.9064    |\n| TuneVideo | 0.2758    | 0.9240    |\n| SDEdit    | 0.2775    | 0.8731    |\n| Pix2Video | 0.2891    | 0.9767    |\n| NExT-GPT  | 0.2684    | 0.9647    |\n| FOCUS(Ours)      | 0.2943    | 0.9815    |\n\n\nTable 7: Quantitative comparison of text-guided image editing methods on COCO-caption.\n\n| Method   | Object  |        | Background |        |\n| :------- | :------ | :----- | :--------- | :----- |\n|          | CLIP(↑) | FID(↓) | CLIP(↑)    | FID(↓) |\n| PTP      | 30.33   | 9.58   | 31.55      | 13.92  |\n| BLDM     | 29.95   | 6.14   | 30.38      | 20.44  |\n| DiffEdit | 29.30   | 3.78   | 26.92      | 1.74   |\n| PFB-Diff | 30.81   | 5.93   | 32.25      | 13.77  |\n| NExT-GPT | 29.32   | 6.62   | 27.31      | 14.27  |\n| FOCUS(Ours)    |  31.22  |  3.45  |  33.10     |  1.98  |\n\nTable 8: Quantitative comparison of segmentation performance on RefCOCO and ReasonSeg benchmarks.\n\n| Method      | RefCOCO |          |          | ReasonSeg |\n| :---------- | :------ | :------- | -------- | --------- |\n|             | Val(↑)  | testA(↑) | testB(↑) | gIoU(↑)   |\n| VisionLLMv2 | 76.6    | 79.3     | 74.3     | 51.0      |\n| PSALM       | 83.6    | 84.7     | 81.6     | -         |\n| HyperSeg    | 84.8    | 85.7     | 83.4     | 59.2      |\n| FOCUS(Ours)       |  84.1   | 86.3     |  82.7    |  61.6     |\n\n**Regarding Question 1 the role of the segmentation mask**\n\nTo clearly illustrate the fundamental importance of the segmentation mask in our framework, we have designed a series of comparative experiments:\n\n1.  **FOCUS (Full Model):** Utilizes internally generated masks that are precisely aligned with the given language instructions.\n2.  **Baseline 1 (No Mask Guidance):** Removes mask guidance entirely, relying solely on text instructions to guide the diffusion model during editing.\n3.  **Baseline 2 (General-Purpose Mask):** Employs masks produced by an external, pre-trained, general-purpose segmentation model (Grounding+SAM2) for guidance.\n\nThese results demonstrate that precise and instruction-aligned segmentation masks are essential for optimal editing performance in our framework.\n\nTable 9: Ablation study on the effect of segmentation mask guidance on EmuEdit benchmark.\n\n| Method                           | EmuEdit |           |           |             |\n| :------------------------------- | :------ | --------- | --------- | ----------- |\n|                                  | DINO(↑) | CLIP-I(↑) | CLIP-T(↑) | CLIP-DIR(↓) |\n| **No Mask Guidance**             | 0.812   | 0.866     | 0.268     | 0.113       |\n| **Using a General-Purpose Mask** | 0.819   | 0.868     | 0.271     | 0.108       |\n| **FOCUS(Ours)**                  | 0.826   | 0.872     | 0.275     | 0.101       |"}, {"review_id": "7qm1QoTYUo", "reviewer": "Reviewer_tU7V", "summary": "FOCUS introduces a single large vision–language model that unifies three capabilities in one network: (i) segmentation-aware perception, (ii) discrete-token visual representation via MoVQGAN, and (iii) controllable, object-centric diffusion editing. A dual-branch encoder captures global semantics and fine spatial detail; predicted masks are jointly treated as supervision targets and spatial prompts that steer the decoder. A progressive multi-stage schedule aligns the encoder, the referential-segmentation head and the diffusion generator. Experiments on multimodal understanding, referring-instance segmentation and controllable image generation show consistent improvements over cascaded pipelines.", "questions": "1. The heart of FOCUS is a dual-branch visual encoder: one branch—built on a CLIP- or Qwen-ViT backbone—captures global, language-aligned semantics, while the other— a hierarchical ConvNeXt-style encoder—provides fine-grained local detail. Why is this two-branch design needed, and can the authors cite prior work that motivates or validates it?\n2. Will a larger model size necessarily yield better performance?", "limitations": "see weakness", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths:\nEnd-to-end model – one network handles both object finding and image editing; no multi-stage pipeline.\nThe model predicts the mask and then reuses it as the spatial condition, giving sharp, well-localized edits.\nA global CLIP-style branch + a fine-detail ConvNeXt branch, with MoVQGAN tokenization, keeps both generation quality and recognition accuracy high.\nResolution and task difficulty are raised stage-by-stage, yielding stable convergence and high-res results.\nOutperforms two-stage baselines on multimodal understanding, referring segmentation, and controllable generation benchmarks.\n\n\n\n\nWeaknesses:\n1. The generation module relies on an LLM + diffusion stack, yet the paper does not justify why a large language model is needed at all. For mask-guided editing, conditioning a diffusion model directly on the user-provided mask is both simpler and already effective. How does the end-to-end LLM pipeline outperform—or even differ from—this baseline?\n2. The “dual-branch” backbone simply pairs a CLIP-style (or Qwen-ViT-style) encoder for global semantics with a ConvNeXt-L hierarchy for local details. This combination has become fairly standard and the paper does not explain what new capability it unlocks for referential editing.\n3. Several tables omit whether a higher or lower score is better. Please add up- or down-arrow symbols or a brief caption so readers can interpret the numbers at a glance.\n4. The appendix notes that replacing binary masks with mask-token embeddings degrades localisation and fidelity, but offers no analysis. Understanding why token conditioning fails would clarify the model’s limitations and guide future improvements.", "was_revised": true, "final_justification": "The authors have addressed all my concerns during the rebuttal and discussion period.", "rebuttal": "Thank you for your insightful and challenging questions. They have prompted us to reflect more deeply on the core design of our work.\n\n**Regarding Weakness 1 & Question 2 (Why an LLM is necessary, instead of directly using a mask to guide the diffusion model):**\n\nA basic model that does not incorporate a large language model (LLM)—for example, one that simply uses a user-provided mask to guide a diffusion model—can only handle straightforward image editing tasks. However, such a model is fundamentally unable to process complex natural language instructions that require semantic comprehension and reasoning.\n\nConsider the following examples:\n\n*   \"Please change the hair of the smiling lady on the left to blonde.\"\n*   \"Remove the cup in front of the cat on the table.\"\n\nThese instructions involve intricate spatial and attribute-based descriptions such as “on the left,” “smiling,” and “in front of,” which cannot be accurately interpreted or localized using manual masks or conventional vision modules. In our framework, the LLM acts as a crucial bridge, connecting high-level language understanding with precise, pixel-level localization.\n\nFOCUS harnesses the reasoning power of the LLM to accurately identify target regions and execute fine-grained edits in response to complex user instructions. This capability is essential for enabling truly interactive image editing. We will emphasize this point more clearly in the introduction and method sections to further distinguish FOCUS from traditional mask-guided editing approaches.\n\nAs demonstrated in **Table 1**, FOCUS surpasses all baselines in the Reason-Edit benchmark (PSNR, SSIM, CLIP Score), highlighting its ability to accurately localize edits while preserving the background under complex instructions.\\\nAdditionally, **Tables 2 and 3** present strong results on text-guided video editing (DAVIS) and image editing (COCO-caption), respectively.\n\nImportantly, FOCUS is built upon the relatively lightweight Qwen2.5-3B model. Despite having significantly fewer parameters than mainstream 7B-scale LLMs, it achieves comparable or even superior performance to larger models such as Janus-Pro-7B across multiple multimodal understanding tasks. This design also endows the model with excellent scalability.\n\n \n\nTable 1: Comparison of different methods on understanding and reasoning scenarios. Higher is better for PSNR, SSIM, and CLIP Score.\n| Methods           | Understanding Scenarios |            |            | Reasoning Scenarios |        |            |\n| :---------------- | :---------------------- | :--------- | :--------- | :------------------ | :----- | :--------- |\n|                   | PSNR(↑)                    | SSIM(↑)       | CLIP Score(↑) | PSNR(↑)                | SSIM(↑)   | CLIP Score(↑) |\n| InstructPix2Pix   | 21.576                  | 0.721      | 22.762     | 24.234              | 0.707  | 19.413     |\n| MagicBrush        | 18.120                  | 0.68       | 22.620     | 22.101              | 0.694  | 19.755     |\n| InstructDiffusion | 23.258                  | 0.743      | 23.080     | 21.453              | 0.666  | 19.523     |\n| SmartEdit         | 22.049                  | 0.731      | 23.611     | 25.258              | 0.742  | 20.950     |\n| UnifiedMLLM       | 20.670                  | 0.776      | 23.633     | 26.667              | 0.808  | 21.104     |\n|  FOCUS(Ours)            |  **24.102**             |  **0.804** |  24.204    |  **28.236**         |  0.842 |  22.317    |\n\n\nTable 2: Comparison of different methods on text-guided video editing (DAVIS). Higher is better.\n| Method    | CLIP-T(↑) | CLIP-I(↑) |\n| :-------- | :------ | :------ |\n| CogVideo  | 0.2391  | 0.9064  |\n| TuneVideo | 0.2758  | 0.9240  |\n| SDEdit    | 0.2775  | 0.8731  |\n| Pix2Video | 0.2891  | 0.9767  |\n| NExT-GPT  | 0.2684  | 0.9647  |\n| FOCUS(Ours)     | 0.2943  | 0.9815  |\n\nTable 3: Comparison of different methods on text-guided image editing (COCO-caption). Higher is better for CLIP, lower is better for FID.\n| Method   | Object  |        | Background |        |\n| :------- | :------ | :----- | :--------- | :----- |\n|          | CLIP(↑) | FID(↓) | CLIP(↑)    | FID(↓) |\n| PTP      | 30.33   | 9.58   | 31.55      | 13.92  |\n| BLDM     | 29.95   | 6.14   | 30.38      | 20.44  |\n| DiffEdit | 29.30   | 3.78   | 26.92      | 1.74   |\n| PFB-Diff | 30.81   | 5.93   | 32.25      | 13.77  |\n| NExT-GPT | 29.32   | 6.62   | 27.31      | 14.27  |\n| FOCUS(Ours)    |  31.22  |  3.45  |  33.10     |  1.98  |\n\n \n\n**Regarding Weakness 2 & Question 1 (The necessity and novelty of the dual-branch encoder):** Although the dual-branch architecture itself is not novel, its application in our work is essential for addressing the dual requirements of referential editing: LLM-based semantic reasoning and pixel-level accurate segmentation. Our approach enables **semantically-aware and pixel-precise editing**. Specifically, the global branch is responsible for understanding \"what\" to edit (i.e., comprehending the language prompt), while the local branch determines \"where\" to edit (i.e., generating precise masks). Our main contribution lies in the effective integration of these two branches, resulting in a robust end-to-end solution for this challenging task.\n\nThe performance improvements of our model are primarily due to its **unified framework design**, rather than simply stacking individual modules. To support this, we conducted a series of ablation studies to clarify the unique contributions of each encoder module. The results (as shown in the table) are summarized as follows:\n\n*   The **Vanilla Encoder** is essential for vision-language understanding. Removing this module (i.e., using only the Hierarchical + Pixel Encoder) results in a **significant decrease in language comprehension**.\n*   The **Hierarchical Encoder** provides strong multi-scale visual perception. Without this module (i.e., Vanilla + Pixel Encoder), the model exhibits a **notable drop in segmentation accuracy** on high-resolution images.\n*   The **Pixel Encoder** enhances pixel-level perception for generation tasks. Excluding it (i.e., Vanilla + Hierarchical Encoder) maintains robust segmentation, but the **detail and quality of generated images are diminished**.\n\nThese comparisons demonstrate that any single or dual-encoder combination is insufficient to address all aspects of the task, each showing clear limitations. Only by **jointly training** these three complementary modules within our **unified framework** can we fully leverage their **synergistic effects**, ultimately achieving comprehensive and superior performance.\n\nTable 4: Ablation study of different encoder combinations on Refcoco, Emu Edit, and Image Understanding benchmarks. Higher is better.\n\n| Method                                 | Refcoco  |          |          | Emu Edit  | Image Understanding |        |         |\n| :------------------------------------- | :------- | :------- | :------- | :-------- | :------------------ | :----- | :------ |\n|                                        | testA(↑) | testB(↑) | Valid(↑) | CLIP-T(↑) | POPE(↑)             | MMB(↑) | SEED(↑) |\n| Vanilla Encoder + Pixel Encoder        | 77.8     | 79.3     | 74.4     | 0.215     | 86.4                | 76.3   | 74.1    |\n| Vanilla Encoder + Hierarchical Encoder | 83.8     | 84.6     | 82.3     | -         | 88.1                | 79.9   | 73.8    |\n| Hierarchical Encoder+Pixel Encoder     | 83.1     | 83.9     | 81.8     | 0.277     | 85.6                | 70.1   | 67.5    |\n| **FOCUS(Our)**                        | 84.1     | 86.3     | 82.7     | 0.283     | 88.0                | 81.5   | 73.9    |\n\n**Regarding Weakness 3 (Table readability)**: We will add arrow symbols (↑/↓) to all performance tables to clearly indicate which direction represents better performance. In addition, we will provide a detailed analysis in the appendix explaining why using a mask is more effective than using mask token embeddings.\n\n**Regarding Weakness 4 (Mask token analysis):**\n\n Our hypothesis is that mask tokens lose precise spatial topological structure after quantization. In contrast, a downsampled binary mask, despite its lower resolution, retains an explicit and continuous spatial guidance signal, which is more friendly to the attention mechanism of the diffusion model.\n\nTable 5: Comparison of mask token and mask usage effects on the diffusion model in EmuEdit. For DINO, CLIP-I, and CLIP-T, higher values indicate better performance; for CLIP-DIR, lower values are better.\n| Method     | EmuEdit |           |           |             |\n| :--------- | :------ | --------- | --------- | ----------- |\n|            | DINO(↑) | CLIP-I(↑) | CLIP-T(↑) | CLIP-DIR(↓) |\n| mask token | 0.817   | 0.831     | 0.253     | 0.127       |\n| mask       | 0.826   | 0.872     | 0.275     | 0.101       |"}, {"review_id": "Aw1Xq4gTiD", "reviewer": "Reviewer_up1b", "summary": "This paper proposes a unified framework for segmentation and controllable object-centric generation, trainable in an end-to-end manner. Unlike previous works which separate segmentation mask generation from the editing pipeline (using separate models for each), this paper introduces as unified architecture that couples segmentation with mask-driven diffusion editing, supporting fine-grained editing or object-level manipulation, in a fully end-to-end manner, integrating referential localization, structured mask generation, and controllable editing. Experiments are conducted on general and document-oriented benchmarks, multimodal image generation benchmarks, referring segmentation and image editing benchmarks.", "questions": "I think the authors have to be more clear about the \"unified\" terminology, and justify with some ablation studies the true advantage of an end-to-end framework, as the improvements could be coming from additional encoders which provide stronger visual features.", "limitations": "The authors mention that \"The paper includes a dedicated \"Limitations\" section that reflects on several key constraints of the proposed approach\" but I was not able to find it neither in the main paper neither in the supplementary.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\n- Unifying segmentation and image editing is an important aspect as the two tasks are inherently fused in the same model.\n- Impressive quantitative and qualitative results\n\nWeaknesses:\n\n- The method in principle is only different than EMU by just adding a segmentation decoder. Furthermore, I would expect the word \"unified\" to translate to a single VLM that is capable of performing all the tasks mentioned. For example, see [R1] where they combine text generation and image generation into a single unified model. The authors on the other hand use 6 task-specific models, trained end-to-end.  \n- The method adds a Pixel Encoder and Hierachical Encoder, providing more strong visual features. Therefore, I am not sure if the improvement comes from these extra models rather than the unified training. The baseline methods could also add these extra encoders and achieve strong results? \n- The method is highly-engineered, it is not a simple and elegant framework. Many components, additional models, bells and whistles and engineering tricks involved. While this itself is not a weakness (and it will not affect my decision), i wonder why don't the authors consider a fully autoregressive approach (for image generation also) to perform the tasks, since they already use discrete tokens from VQGAN? I believe the autoregressive image generation pipeline is becoming stronger than continous diffusion-based generation. There are many papers for that, but some papers that I remember well are [R2, R3, R4]. Some of them also unify text and image generation, as in [R1] but autoregressively. \n- Figure 1 is very confusing. What exactly are the text instructions? Is it the Question: \"segment the two skiers and perform editing\" or is it the text in (a), (b), (c) and (d) ? Or is it the scribbles/boxes/masks. It is not clear what the prompt is, the clarity of this figure is very bad. The same in Figure 3, it in unclear which prompts are used. Is it the instruction prompt or the visual prompt? For example, when the instruction prompt is \"replace the snowy mountain background...etc\", is there also a visual prompt (e.g., scribbles)? From my understanding, it seems that either, but not both. \n\n[R1] Transfusion: Predict the Next Token and Diffuse Images with One Multi-Modal Model\\\n[R2] GigaTok: Scaling Visual Tokenizers to 3 Billion Parameters for Autoregressive Image Generation\\\n[R3] Fluid: Scaling Autoregressive Text-to-image Generative Models with Continuous Tokens\\\n[R4] Liquid: Language Models are Scalable and Unified Multi-modal Generators", "was_revised": true, "final_justification": "The authors have addressed all my concerns during the rebuttal and discussion period. I am generally happy about the paper as a whole. I am still not satisfied with the complex, highly-engineered, computationally-expensive system proposed (mainly because of the extra encoders and unified pipeline training which needs large computation for gradient updates). But this is not a reason to reject the paper by any means. Therefore, I recommend Acceptance.", "rebuttal": "Thank you for your review and your positive feedback on our work. Your questions regarding the definition of \"unified\" and the attribution of performance improvement are very insightful.\n\n**Regard weekness1\\&queston** **about \"unified\"**\n\nPrevious approaches typically employ cascaded pipelines (e.g., LLM+SAM→segmentation→independent editing), where each module is pre-trained independently and lacks joint optimization. This separation often results in suboptimal editing, as the segmentation output may not be well-suited for the requirements of the generation module. In contrast, our approach is **end-to-end trainable**, enabling joint learning of instruction comprehension, object segmentation, and diffusion-based editing. All components share visual representations and operate within a unified, language-aligned space, which facilitates fine-grained, instruction-driven editing. Compared to Transfusion，which also aims to unify perception and generation，our model achieves **joint optimization across all modules**, further improving pixel-level accuracy and consistency.\n\nAs demonstrated in **Table 1**, our method surpasses all baselines on the Reason-Edit task (PSNR, SSIM, CLIP Score), showing both precise region localization and strong background preservation under complex instructions. **Table 2** and **Table 3** further highlight our model’s superior performance in text-guided video (DAVIS) and image (COCO-caption) editing.\n\nTable 1: Quantitative comparison of different methods on understanding and reasoning scenarios. Higher is better for all metrics.\n\n| Methods           | Understanding Scenarios |            |               | Reasoning Scenarios |         |               |\n| :---------------- | :---------------------- | :--------- | :------------ | :------------------ | :------ | :------------ |\n|                   | PSNR(↑)                 | SSIM(↑)    | CLIP Score(↑) | PSNR(↑)             | SSIM(↑) | CLIP Score(↑) |\n| InstructPix2Pix   | 21.576                  | 0.721      | 22.762        | 24.234              | 0.707   | 19.413        |\n| MagicBrush        | 18.120                  | 0.68       | 22.620        | 22.101              | 0.694   | 19.755        |\n| InstructDiffusion | 23.258                  | 0.743      | 23.080        | 21.453              | 0.666   | 19.523        |\n| SmartEdit         | 22.049                  | 0.731      | 23.611        | 25.258              | 0.742   | 20.950        |\n| UnifiedMLLM       | 20.670                  | 0.776      | 23.633        | 26.667              | 0.808   | 21.104        |\n|  FOCUS(Ours)            |  24.102             |  0.804 |  24.204       |  28.236         |  0.842  |  22.317       |\n\nTable 2: Quantitative comparison of different methods on video editing benchmarks. Higher is better for all metrics.\n\n| Method    | CLIP-T(↑) | CLIP-I(↑) |\n| :-------- | :-------- | :-------- |\n| CogVideo  | 0.2391    | 0.9064    |\n| TuneVideo | 0.2758    | 0.9240    |\n| SDEdit    | 0.2775    | 0.8731    |\n| Pix2Video | 0.2891    | 0.9767    |\n| NExT-GPT  | 0.2684    | 0.9647    |\n| FOCUS(Ours)     | 0.2943    | 0.9815    |\n\nTable 3: Quantitative comparison of different methods on object and background editing. Higher CLIP and lower FID indicate better performance.\n\n| Method   | Object  |        | Background |        |\n| :------- | :------ | :----- | ---------- | ------ |\n|          | CLIP(↑) | FID(↓) | CLIP(↑)    | FID(↓) |\n| PTP      | 30.33   | 9.58   | 31.55      | 13.92  |\n| BLDM     | 29.95   | 6.14   | 30.38      | 20.44  |\n| DiffEdit | 29.30   | 3.78   | 26.92      | 1.74   |\n| PFB-Diff | 30.81   | 5.93   | 32.25      | 13.77  |\n| NExT-GPT | 29.32   | 6.62   | 27.31      | 14.27  |\n| FOCUS(Ours)    |  31.22  |  3.45  |  33.10     |  1.98  |\n\n**Regard weekness2 (Source of performance improvement)**\n\nThe performance improvements of our model are mainly due to its **unified framework design**, rather than simply stacking individual modules. To substantiate this, we performed a series of ablation studies to clarify the unique contributions of each encoder module. The results (as shown in the table) are summarized as follows:\n\n*   The **Vanilla Encoder** is essential for vision-language understanding. Removing this module (i.e., using only the Hierarchical + Pixel Encoder) results in a **substantial decrease in language comprehension**.\n*   The **Hierarchical Encoder** offers robust multi-scale visual perception. Excluding this module (i.e., Vanilla + Pixel Encoder) leads to a **notable reduction in segmentation accuracy** on high-resolution images.\n*   The **Pixel Encoder** significantly enhances pixel-level perception for generation tasks. Without it (i.e., Vanilla + Hierarchical Encoder), while segmentation remains strong, the **fidelity and detail of generated images are diminished**.\n\nThese comparisons indicate that any single or dual-encoder combination is inadequate for handling all tasks and presents clear limitations. This demonstrates that only by **jointly training** these three complementary modules within our **unified framework** can we fully leverage their **synergistic effects**, ultimately achieving comprehensive and superior performance.\n\nTable 4: Ablation study results for different encoder combinations on Refcoco, Emu Edit, and Image Understanding benchmarks. Higher is better for all metrics.\n| Method                                 | Refcoco  |          |          | Emu Edit  | Image Understanding |        |         |\n| :------------------------------------- | :------- | :------- | :------- | :-------- | :------------------ | :----- | :------ |\n|                                        | testA(↑) | testB(↑) | Valid(↑) | CLIP-T(↑) | POPE(↑)             | MMB(↑) | SEED(↑) |\n| Vanilla Encoder + Pixel Encoder        | 77.8     | 79.3     | 74.4     | 0.215     | 86.4                | 76.3   | 74.1    |\n| Vanilla Encoder + Hierarchical Encoder | 83.8     | 84.6     | 82.3     | -         | 88.1                | 79.9   | 73.8    |\n| Hierarchical Encoder+Pixel Encoder     | 83.1     | 83.9     | 81.8     | 0.277     | 85.6                | 70.1   | 67.5    |\n| **Our**                                | 84.1     | 86.3     | 82.7     | 0.283     | 88.0                | 81.5   | 73.9    |\n\n**Regarding Weakness 3 (Architectural design and autoregressive solutions)**\n\nFOCUS is a unified framework that seamlessly integrates multiple components to enable high-quality understanding, segmentation, and image editing in a unified manner—an achievement that remains challenging for previous methods. At its core, our design employs a multimodal autoregressive large language model (MLLM) for instruction reasoning and discrete visual token generation. The diffusion module is incorporated not as a replacement for the autoregressive process, but as an advanced image decoder to enhance output fidelity. Leveraging diffusion allows our model to recover fine-grained details, reduce visual artifacts, and achieve greater robustness compared to direct decoding from discrete tokens. Additionally, the diffusion decoder supports resolution upscaling during generation (e.g., from 256×256 to 512×512), which is crucial for producing high-quality results. Given current technical constraints, the combination of an autoregressive MLLM with a diffusion-based decoder strikes an effective balance among semantic understanding, editing accuracy, and visual quality. We also acknowledge the promise of fully autoregressive generation models and view them as a valuable direction for future research.\n\n**Regarding Weakness 4 (Clarity of Prompt and figures)**\n\nWe have provided detailed clarification in the supplementary material. FOCUS adopts a **structured prompt schema** (as shown in Table 10) consisting of two components: a **task instruction prompt** and a **condition prompt**.\n\nThe task instruction prompt defines the model's objective in natural language. For example, “segment the two skiers and perform editing” falls under this category. In contrast, the texts and visual inputs in Figures (a), (b), (c), and (d)—such as scribbles, bounding boxes, and masks—serve as condition prompts, providing necessary contextual information for the task.\n\nIn our framework, textual commands like “replace,” “segment,” “remove,” and “modify” are treated as **text prompts**, while bounding boxes, points, masks, and scribbles are treated as **visual prompts**, primarily used in interactive and video segmentation tasks.\n\nTechnically, we sample CLIP visual features $f_v$​ from the VLLM according to the region coordinates and apply adaptive average pooling to generate final reference features for each visual prompt. Since these visual prompts are also encoded into feature vectors, **text and visual prompts can co-exist** and jointly guide the segmentation and generation process.\n\nWe will revise the figures and descriptions in the final version to ensure that the distinctions between instruction prompts, condition inputs, and model-internal representations are clearly conveyed and easy to understand.\n\n**Regarding the Limitations Section:**\n\n Thank you for your feedback. We apologize that the 'Limitations' section, though mentioned in our submission checklist, was omitted from the main manuscript. We will add the limitation: **(1)** the limited precision when editing small objects due to latent space downsampling; **(2)** the high computational cost, which challenges replication; and **(3)** inherited weaknesses from SDXL, such as rendering complex spatial relationships, which we frame as a direction for future work."}]

Author rebuttal:

Meta-review: This paper introduces a new model that jointly performs image / video understanding and generation. Specifically, the authors propose an architecture capable of both segmentation and object-based editing, built on top of a VLM. The reviewers acknowledge that the model design is well-motivated and that it demonstrates impressive results both qualitatively and quantitatively, outperforming baseline methods on a wide range of tasks. On the other hand, the reviewers raise concerns regarding the clarity of presentation, the lack of ablation studies, and the highly complex model design, which lacks strong justification. Most of these concerns were adequately addressed in the rebuttal, though one reviewer remains skeptical about the necessity of the model’s complexity. The Area Chair agrees with the reviewers’ assessment and considers this paper acceptable, conditioned on the information introduced and promised in the rebuttal being properly incorporated into the final version.
Decision: Accept (poster)