PixWizard: Versatile Image-to-Image Visual Assistant with Open-Language Instructions
Authors: Weifeng Lin
Xinyu Wei
Renrui Zhang
Le Zhuo
Shitian Zhao
Siyuan Huang
Junlin Xie
Peng Gao
Hongsheng Li
Venue: ICLR 2025 Poster (2025)
Keywords: Diffusion Model
Image Generation
Image-to-Image
Primary area: generative models

Abstract: This paper presents a versatile image-to-image visual assistant, PixWizard, designed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle a variety of vision tasks into a unified image-text-to-image generation framework and curate an Omni Pixel-to-Pixel Instruction-Tuning Dataset. By constructing detailed instruction templates in natural language, we comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image prediction, image editing, controllable generation, inpainting/outpainting, and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with a flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of information from the input image. Our experiments demonstrate that PixWizard not only shows impressive generative and understanding  abilities for images with diverse resolutions but also exhibits generalization capabilities with unseen tasks and human instructions.

Published as a conference paper at ICLR 2025
PIXWIZARD: VERSATILE IMAGE-TO-IMAGE VISUAL ASSISTANT WITH OPEN-LANGUAGE INSTRUCTIONS
Weifeng Lin'*
Xinyu -ei
Renrui Zhang"
Le Zhuo
Shitian Zhao
Siyuan Huang
Junlin Xie
Peng Gao
Hongsheng Lif
CUHK MMLab
Peeking University
Shanghai Al Laboratory
ABSTRACT
This paper presents versatile image-to-image visual assistant, PixWizard, de- signed for image generation, manipulation, and translation based on free-from language instructions. To this end, we tackle variety of vision tasks into uni- fied image-text-to- tomae generation framework and curate an Omni Pixel-to-Pive Instruction-Tunie Tuning Dataset. By constructing detailed instruction templates in natu- ral language, comprehensively include a large set of diverse vision tasks such as text-to-image generation, image restoration, image grounding, dense image pre- diction, image editing, controllable generation, impainting/ouppinninng and more. Furthermore, we adopt Diffusion Transformers (DiT) as our foundation model and extend its capabilities with flexible any resolution mechanism, enabling the model to dynamically process images based on the aspect ratio of the input, closely aligning with human perceptual processes. The model also incorporates structure-aware and semantic-aware guidance to facilitate effective fusion of in- formation from the input image. Our experiments demonstrate that Pix Wizar not only shows impressive generative and understanding abilities for images with diverse resolutions but also exhibits generalization capabilities with unseen tasks and human instructions.
INTRODUCTION
Large Language Models (LLMs) (Brown al., 2020; Touvron al., 2023) and Large Vision Models (LVMs) (Radford et al., 2021; Caron et al., 2021; Bai et al., 2024) have gained global popularity by successfully unifying multiple tasks within a single, coherent framework. Nowadays, LLMs have become efficient language assistants, demonstrating strong capabilities in open-world language understanding and reasoning. However, versatile visual assistant capable of following diverse multimodal instructions that align with human intentions to effectively perform various visual tasks in real-world scenarios is still under exploration.
Recently, there are two research lines aiming to achieve general visual assistants: ieftusiion-baeed and in-context learning approaches. The first focuses on developing text-to-image models (Rombach et al., 2022b) as unified foundation model for various visual perception tasks. For example, InstructPix2Pix (Brooks et al., 2023), InstructDiffusion (Geng et al., 2024), and InstructCV (Gan et al., 2024) repurpose generative models universal language interface for image editing and other visual tasks. However, their capable tasks are limited, and their performance lags behind that of task-specific models. The second research direction focuses on visual prompting, where pixel-based prompts are used to tackle various vision tasks within a single learning framework. This approach employs prompting techniques to generate the desired visual outputs an in-context manner. Examples include Painter (Wang et al., 2023b), PromptDiffusion (Wang et al., 2023c), and LVM (Bai et al., 2024), which have successfully handled variety of visual scenarios within one framework. However, these methods inherently lack the ability to follow human language instructions, limiting their controllability and interactivity. (More related works are discussed in Sec. A.)
"Equal Contribution
Corresonniing Authors I Projecct Lead

---

Published as a conference paper at ICLR 2025
HedZimage
ConnyAimage
Depchalmage
Sketch2lmaze
Poso2image
NormaOtmage
Controllable
Generation
Droie hood thr days
Image2Normal
Image25eg
Imoge2Hied
Image2Canny
Image2Skesch
Image2Pose
Dande Image Prediction
Make Aigh salig Camtio soman the thagge dhance
Light
Restoration
Opear Lagg
Volee Ary Pickee
Image Editing
Roraer the don fow Bo pppehaapoon
Sogmentation
Box
Mask
Image Grounding
qpe was Boanding hos sclad) (apdoa)
Iaago Edosig
19%
Inpainting
Outpstmiing
Facee impainting
Omni
Impainting Outpainting
toxel-to-Piiel
Instruction-tuning
ail oily neher
Donce Perducion 1st
Dataset
Text in--nnn
28%
Text-to-Imoge Generation
Figure 1: Task Overview of the Omni Pixel-to-Pixel Instruction-uniing Dataset for Pix Wizard.
Taking into account the strengths and limitations of previous approaches, we introduce Pix Pix Wizard, a versatile interactive image-to-image visual assistant designed for image generation, manipulation. and image-to-iman translation. Pix Wizard is DiT-based framework that can handle wide range of visual tasks when provided with sufficient training data, along with the interface for human language instructions. Specifically, Pix Wizard exhibits three main features as follows:
Given the diverse nature of vision tasks and data formats, ranging from pixels and coordinates binary masks and categories, it is challenging to find a unified representation. We observe that most vision tasks can be framed as image-to-image translation problems. For tasks not naturally suited to image output, we first learn to generate their visualizations and then apply post-processing to convert them into the desired formats. This approach is key step toward developing versatile visual assistant.
I. Task Unification:
2. Data Construction: We aim to leverage and integrate the remarkable diversity of tasks and data in the visual domain. To achieve this, we have built comprehensive training set with total of 30 million data points, enabling our model to support five main capabilities: (i) Image generation, which includes text-to-image generation, controllable generation, inpainting, and outpainting: (ii) Image editing: (iii) Image restoration, covering tasks such as deraining, desnowing, deblurring, super-resolution, and more; (iv) Image grounding. which involves locating objects based on user prompts; and (v) Dense image prediction, which includes depth estimation, surface normal estimation, pose estimation, semantic segmentation, and image-to-canny/HHH wiw.aaaanynaDiiieeeeeeeiiiiiiee.
conversions.
For robust visual assistant, the architecture and scalability of the foundation model are crucial. this work, we use the flow-based Diffusion Transformer (DíT) (Ma et al., 2024), which offer great versatility and stability in modeling data distributions by learning conditional velocity fields. The DIT architecture further enhances scalability and well-suited for incorporating conditional information. Building this foundation, introduce several key components. We extend the dynamic partitioning and padding scheme (Zhuo et al., 2024) to handle input images of any resolution, aligning closely with human perception. Additionally, we implement structure-awad and semantic-awan aware guidance, enabling the model to follow multimodal instructions (images and user prompts) for wide range of manipulations.
3. Architecture Design:
Experimental results show that Pix Wizard achieves competitive performance on certain tasks com- pared to task-specific vision models, and outperforms current ofth--of-theart general visual models
2

---

Published as a conference paper at ICLR 2025
in most tasks, delivering its strong overall multi-task performance. More importantly, our model handles tasks and instruction prompts it has not encountered during training, demonstrating promising generalization capabilities. This further highlights Pix Wizard's strength powerful interactive image-to-image visual assistant.
2 OMNI PIXEL-TO-PIXEL INSTRUCTION-TUNIGG DATASET
To equip our image-to-image visual assistant with comprehensive capabilities in image generation, manipulation, and translation, we compiled a multi-task training dataset for visual instruction tuning, consisting of 30 million instances across seven primary domains, as illustrated in Fig. 1. This is the triplet dataset, built from both open-source and in-house data, filtered with the help of MILMs and manual review.
user-friendly image-instruction-imneg
Image Restoration. We incorporate low-level data to restore images degraded by various envi- ronmental or technical factors. This section utilizes a wide range of open-source datasets covering key restoration tasks, including (1) Denoising, (2) Deraining, (3) Demoireing, (4) Dehazing, (5) Deblurring, (6) Desnowing, (7) Deshadowing, (8) Low-Light Enhancement, (9) Face Restoration, (10) Watermark Removal, and (II) Super Resolution. Since both the inputs and outputs are inherently defined in the RGB space, these tasks can be scamlessly unified by our Pix FixWizard model without any extra transformations. All open-source datasets we use are provided in Sec. B.I.
Image Grounding. Image grounding involves identifying and highlighting specific areas of objects in images based on provided text prompts. The data for this part is sourced from well-known datasets such gRefCOCO (Liu et al., 2023a), RefCOCO3 (Yu et al., 2016), and Visual Genome (Krishna et al., 2017). We focus on three types of grounding tasks: (1) Segmentation Referring, where the target object specified by the user is highlighted in the output image; (2) Box Detection Reserring, where the target object is highlighted using bounding boxes; and (3) Binary Mask Prediction. where a binary mask is directly predicted. (Details are provided in the Sec. B.2.)
Controllable Generation. Referring to ControlNet (Zhang et al., 2023a), we aim to equip our model with natural image generation capabilities given conditional inputs. We first gather natural images from the LAION Art dataset (Schuhmann et al., 2022) and our own collection of high-quality images from the Internet. We then use MiniCPM-Llama3-X V 2.5 (Yao et al., 2024), a robust edge-side multimodal LLM, along with advanced techniques to generate captions and conditional inputs for the images. (Details on data construction can be found in Sec. B.3.)
Dense Image Prediction. Dense image prediction tasks require the model to interpret input images and produce dense annotations. For depth estimation, surface normal estimation, and semantic segmentation, we represent per-pixel labels as RGB images, which can be generated via the image generation capabilities. For other tasks, such as the prediction of human pose maps, sketches, HED boundaries, canny edge maps, and cartoon line art, treat them as image-to-image translation tasks, as we can easily obtain image pairs using open-source tools. (Details are provided in the Sec. B.4.)
Image Editing. Instruction-basett image editing holds great potential for practical applications, as it allows users to perform edits using natural language commands. To enhance our model's ability to modify images according specific user instructions, we unify several public editing datasets, including UltraEdit (2024), MagicBrush (2024a), GQA-Inpaint (2023), Instruct P2P (2023), SEED-X-Edit (2024), GIER (2020), and HQ-Edit (2024). These datasets encompass wide range of semantic entities, varying levels of detail, and multiple editing tasks including object removal, object replacement, object addition, background replacement, and style transfer.
Inpainting involves filling in missing parts of an image with new or modified content. To create inpainting instances, we apply random black or white masks to different regions of the original images. These masks come in various shapes, including circles, rectangles, and free-form patterns, and are randomly placed on the images, resulting in a wide range of occluded areas for inpainting. Outpainting (Image extrapolation) extends image's content beyond its boundaries. To create outpainting instances, we randomly crop the central part of the image, while masking the surrounding areas with black or white. The cropping is not limited square shapes but includes rectangles of varying proportions. This approach ensures that the outpainting task covers a range of extension scenarios, challenging the model to generate coherent content beyond the original image boundaries.
3

---

Published as a conference paper at ICLR 2025
Input Image
Noise
Floar sctinuts the daphs,
estertu copa
breas
Tranctore info Von legh style Ouange pins
Fatch Erizal
Commu
Task Aware
PoxWizard Block
AugPool
Dysomc Sumpler
Toxt Emielding
ArgFoel
PixWizord Block
Groai
Eanedrigg
Teck Emberdding
PocWizard Block
Time
Vi Decodkr
Embeddng
Output Prediction
Figure 2: Overall framework of Pizard.
Text-to- Image Generation. Image-to-image and text-to-image are distinct tasks: the former requires an additional input image as a condition, while the latter does not. Existing instruction-tune tononn methods adapt pretrained exxt-to-image models for image-to-image tasks, but often lose the original text-to- image capabilities. To unify both tasks within a single framework, we propose a "drawing" strategy that preserves text-to-image functionality. Specifically, we introduce additional input-a fully language instructions. This simulates a blank canvas, allowing the model to "draw" images based the text prompts. This approach differentiates our model from previous text-to-image systems.
white or black
To enhance the usability of PixWiz a practical visual assistant, we aim for the model to understand free-form user prompts. Instead of relying on fixed task-specific prompts, we begin by manually writing 6-10 prompts for each vision task describe the task. We then use GPT-4o to generate a wide range of paraphrased variations. This process ensures that our instruction remains diverse while staying true the core intent of the original prompts. Instruction templates and examples are provided in Sec. B.5.
Open-language Instruction.
3 PIXWIZARD
In this section, we present the details of the Pix Wizar framework from the perspectives of model architecture (as shown Fig. 2) and training strategies.
3.1 FLOW-BASED CONDITIONAL INSTRUCTION-TUME TUNING
Previous studies (Wang et al., 2022b; Brooks al., 2023) show that fine-tuning large diffusion models outperforms training models from scratch for image translation and editing tasks. Therefore, we initialize the weights Pix Wizard with the pretrained Lumina-Next-T2: (Zhuo al., 2024) checkpoint, which is a flow-based diffusion transformer, to leverage its extensive text-to-image generation capabilities. We learn network vo that predicts the velocity field u, given image conditioning Cy and text instruction conditioning cr. We minimize the loss function as follow:
C
I (I,, L cr, CT) - ur(2e, 111
3.2
ARCHITECTURE
Text Encoders. We begin by using Gemma-2B (Team et al., 2024) as the text embedder in PixWizard to encode text prompts. However, in multi-task learning, relying solely on text instructions insufficient to guide the model in accurately executing user commands. To better guide the generation process for the correct task, we incorporate the CLIP text encoder (Radford et al., 2021). Global average pooling is applied to the CLIP text embeddings to obtain a coarse-grained text representation, which is passed through Ml.e-based task embedder generate the task embedding.
4

---

Published as a conference paper at ICLR 2025
Ealvodding
Fruj
tak
Ridk Narm
BANS Notm
QKY
Selt- Alti.
Iinot Tukers
Enstedding
Alla
PixWizard Block
Ensrediingg
Embedding
Sevartic Embaridig
Maat-
Spendiy
Clanel
Guabbo-
Ewoudding
Task Aware Dynamic Sampler
Figure 3: The schematic illustrations of Pix Wizard Block and Task-Aware Dynamic Sampler.
This embedding is then integrated into the PixWizard Block by adding it to the timestep embeddings through a modulation mechanism. In Sec. C.2, we show the t-SNE visualization further illustrate the effectiveness of the task embedding.
Strae Guidance. To effectively capture the overall structural features of the input image condition, we begin by encoding the image using variational autoencoder (VAE) (Kingma Welling, 2013) from SDXL (Podell al., 2023). Next, we concatenate the image latent with the noise latent along the channel dimension (Saharia et al., 2022c;a). Following (Brooks et al., 2023), additional input channels are added to the patch embedder, with the weights for these new channels initially set to zero.
Semantic-Awaad Aware Guidance. Besides recognizing structural features, we use CLIP L/14-336 (Rad- ford et al., 2021) to obtain semantic image embeddings. Within the PixWiz block, we introduce two zero-initialized attention mechanisms, allowing latent target image tokens to query information from the condition keys and values. Specifically, vero-initialized gating mechanism is employed to gradually inject conditional image and text information into the token sequences. Given target image queries (Q,), keys (K,), and values (Vi), along with text instruction keys (K₂) and values (V),, and conditional image keys (Kaz) and values (Va), the final attention output formulated as:
Q.KT Vd
QiKT Vd
Q.KT Vd
A = softmax
Vi + tanh(a:) softmax
Ve + tanh(aa)) softmax
Ves
(2)
where Q; and K, stand for applying RoPE (Su et al., 2024), is the dimension queries and keys, and indicates the zero-initialized learnable parameter in gated cross-attention.
However, inputting all image tokens into the attention layer can lead to significant computational demands, and not all semantic tokens relevant the specific task. To address this, introduce the Task-Aware Dynamic Sampler, designed to select the most relevant semantic tokens for each task. This sampler uses lightweight ranking network with four linear layers and activation functions. Inspired by Dynamic ViT (Rao et al., 2021), we map image tokens to both local and global features. Additionally, task embeddings (Itask) are integrated help the sampler identify the tokens most relevant to the task. The computational process is formulated as:
MLP(x + Ftask) E NxC
local = =//;: ŞIERMKS gloobul = Aug(21: 2 :I) E ERI 2. [oloall yslobal]
C
1iiS N.
M MLM RR1
(5)
where M, denotes the importance of the i-th token. However, implementing token sparsification challenging in practice. Directly sampling tokens based on their importance scores non- differentiable, which hinders end-to-end training. To address this,
the Gumbel-Soffmax
5

---

Published as a conference paper at ICLR 2025
technique (Jang et al., 2016) and adapt it into Multi-Hot Gumbel-Softmax (MHGS) enable simultaneous sampling of the top tokens:
f = x MHGS(M)
(6)
where the output of GumbelSoftmax is multi-hot tensor representing the mask the retained tokens. denotes the Hadamard product, where the top K tokens by importance score weighted by 1 and retained, while the remaining (N - K) tokens are weighted by zero and discarded. Each layer is equipped with independent task-aware dynamic sampler. This approach not only captures the most relevant semantic features needed by each layer for different tasks but also reduces the computational cost in the attention process.
3.3 TWO-STAGE TRAINING AND DATA BALANCING STRATEGIESS
To unlock the model's potential and improve its performance on tasks with smaller datasets, we propose two-stage training and data cala-balanncnn strategy. S1: In the first stage, we initialize the model by combining the weights of pre-trained Lumina-Next-T2) (Zhuo et al., 2024) with randomly initialized weights for the newly added modules. We prioritize tasks with smaller datasets, assigning each a sampling weight to increase its data volume. This weight determines how many times the dataset repeated during an epoch. Using this method, each task achieves approximately 20k data points. We then randomly sample from other tasks to match this scale, creating our first-stage training dataset, with training spanning 4 epochs. S2: In the second stage, we initialize the model using the weights from the first stage and combine all the collected data for further training. To balance tasks, we assign manual sampling weights to each dataset, randomly selecting data when a weight less than 1.0. We also include text-to-image data at 1:1 ratio with other tasks, resulting in second-stage training dataset. At this stage, the total dataset reaches up to 20 million samples.
4 EXPERIMENTS
4.1 FIRST SECTION RESULTS
Settings. For image restoration, follow previous works (Conde al., 2024; Potlapalli et al., 2024) and prepare datasets for various restoration tasks during training. For evaluation, we first select two representative benchmarks: Rain (2017) for deraining and SIDD (2018) for denoising. Ad- ditionally, we further assess performance on other restoration tasks and evaluate zero-shot capabilities in the Sec. D.2.
For image grounding, we evaluate referring segmentation tasks on the gRefCOCO (2023a), RefCOCO, and RefCOCO+ validation and test sets. To assess the performance gap with specialized models, we report results from several expert methods and primarily compare our approach with two unified models: Unified-IO and InstructDiffusion. Following standard practices (Liu et al., 2023a), use cumulative IoU (cloU) as the performance metric.
Table 1: Comparison of Pix Wizard with task-specific and vision generalist baselines across six representative tasks, covering both high-level visual understanding and low-level image processing. indicates that the method is incapable of performing the task.
Depth Est. RMSF_
Semantic Seg, Surface Normal Est.
Denoise
Derain
Image Grounding cloUT (val set)
Methods
mloU!
Mean Angle Emor]
PSNR1 SSTM PSNR SSIM
NYU-2 SUNRGB-D
ADE20K
NYU: NYU--epth V2
SIDD
Rain
RefCOCO RefCOCO+
DepthAnything (2024g)
0.206
Marigold (2024a)
0.224
Mask DINO (2023)
60.80
Fosker (2022)
56.10
Bac al. (2021)
14,90
InvPT (2022)
AirNet (2022)
38.32
0.945
32.98
0.951
PromptIR (2024)
39.52
0.954
36.37
0.972
LAYT (2022)
72.73
56.86
ReLA (2023a)
73.21
$6.10
Unified-1O (20226)
0.387
0.287
25.71
X
X
X
46,42
40.50
Painter (2023h)
0.288
0.285
49.90
38.71
0.954
29.87 87.87
0.882
InstrucICV (2024)
0.297
0.279
47.23
X
InstuseDiilluion (2024)
34.26
0.938
19.82
0.741
41.64"
33.20"
Pix Wizand 0,287
0.291
32.76
19.65
38.67
0.957
31.43
0.917
46.44
36.49

---

Published as a conference paper at ICLR 2025
Dense image prediction tasks evaluated across three vision tasks: ADE20k (2017b) for semantic segmentation, NYUv2 (2012) and SUNRGB-D (2014) for monocular depth estimation, and NYU- Depth v2 (2012) for surface normal estimation. Implementation details can be found in Sec. D.I.
Results. Table presents a comprehensive performance comparison with recent state-of-the-ar (SOTA) task-specific and all-in-one methods. As shown in the results, despite denoising and deraining data making up only a small portion the overall training set, our method outperforms other unified methods and compare some task-specific approaches. In the image grounding task, Pix Wizard significantly outperforms the diffusion-basse generalist model InstructDiffusion by 4.8 cloU on RefCOCO (val). However, there is still room for improvement compared to other highly specialized models. Furthermore, as shown in Fig.5. Pix Wizard supports flexible instructions, allowing it to highlight and visualize the target object directly on the image while also generating the corresponding binary mask. Additional quantitative evaluation results are provided in the Sec. D.3.
Make the photo doaare by flisiig out the ranitoop.
Remrve the watrmork t show the orainal picuue.
Eshane the image by increading
Remson motee and ploelctin movel the natural appcerunce described in: vtar the night sky
Figure 4: Qualitative Evaluation of Instruction- based Image Restoration.
Construct blr homttt-hhite maek Ioseed un the abpec desription: hox
Mark the gisris, reds red vehtcie behind the metal post
Locate the sublect bluc the bus on the center Construct white pillow
ynllor The tay "Yoehi" in middle
Mark white pillow with
Figure 5: Qualitative Results of Instruction-based Image Grounding.
Cruts depth mup from the image
Image
ceppt
Image segmentation map. visualty dvvidng
to
nonnal
conty
pose
sketch
Ganarate die simulated
Figure Visualizations of dense image prediction examples.
For dense prediction tasks, Pix Wizard performs competitively against both generalist and task-specific baselines across all three tasks. In depth estimation on the NYUv2 test set, PixWizard achieves a 10.0% improvement in RMSE compared to Unified-IB and performs similarly to Painter and InstructCV. Additionally, Fig. 6 provides examples of Pix Wizard's outputs. As shown, by supplying the corresponding task-ipec prompt for the same image, we can easily generate the respective condition visualization, underscoring Pix Pizard''''s significant practical value.
7

---

Published as a conference paper at ICLR 2025
4.2 SECOND SECTION RESULTS (IMAGE EDITING)
Settings. We evaluate PixWizard across two benchmarks, the MagicBrush Test (Zhang al., 2024a) and the Emu Edit Test (Sheynin al., 2024), to assess its effectiveness in image editing capabilities. For fair comparison, primarily compare with several instruction instruction-guided guided image editing methods. Consistent with Emu Edit, we use LI distance, CLIP image similarity, DINO similarity, CLIP text-image similarity, and CLIP text-image direction similarity metrics.
Table 2: Comparison with image-editing baselines evaluated on Emu Edit and MagicBrush test set.
Emu Edit Test
MagicBrush Test Set
Method
CLIPat 1 CL. CLIP out
Lt4
DINO
CLIP dor
CLIP
CLIPout f Lt DINO
InstructPix2Pix (2023)
0.078
0.834
0.219
0.121
0.762
0.115
0.837
0.245
0.093
0.767
MagicBrush (2024a)
0.090
0.838
0.222
0.100
0.776
0.123
0.883
0.261
0.058
0.871
Emu Edit (2024)
0.109
0.859
0.231
0.094
0.819
0.135
0.897
0.261
0.052
0.879
UltraEdit (2024)
0.107
0.844
0.283
0.071
0.793
0.868
0.088
0.792
Pix Wizard
0.104
0.845
0.248
0.069
0.798
0.124
0.884
0.265
0.063
0.876
Input
PiXWzordd
UltroEdit
IrstructRixPik
MagicBrush
"Change
"Repiats the plain white Tahirt to stort-slesved iaawatlan shart with vibeant pattern red, yellow, and orange hibiscus flnoers leaves
AAdd a mown to the cat's hot.
Figure 7: Qualitative examples comparing Pix Wizard with other editing approaches.
Results. Table 2 presents our results compared to the baselines. The findings show that our model consistently outperforms InstructPix2Pix, MagicBrush, and UltraEdit automatic metrics and achieves comparable performance to state-of of the-art method Emu Edit. As shown in Fig. 7, our model precisely identifies editing region while preserving the rest of the pixels, and it demonstrates the best understanding of the given instructions.
4.3 THIRD SECTION RESULTS (IMAGE GENERATION)
In this section, we focus on evaluating the effectiveness of Pix Wizard's generation capabilities. Specifically, we assess its performance across four tasks: text-to-image generation, controllable image generation, inpainting, and outpainting. Implementation details can be found in Sec. D.1.
Depth-to-Image
Seg- tm-Ie
NED-to-Imoge
Normol-to-Iene
LineArt-to--mag
Figure 8: Visualization examples under different conditions.
8

---

Published as a conference paper at ICLR 2025
Table Comparison of Pix Wizard with task-specific baselines across five representative tasks.
Canny-to-Image
Depth-to-Image
Inpainting
Outpainting
Text-to-Image
Methods
FI
HIDJ CLIP-ST RMSF, FID CLIP-ST FID] LPIPS. FID. IST
FID
HPSv2T
MultiGen 20M
MultiGen 20M
Places
Places
COCO 30K
Photo
ControlNet-SDI15 (2023a)
34.65
14.73
32.15
17.76
32.45
121-Adgpee-SDD.55 (2024) 23.65
15.96
31.71
48.40
22.52
31.46
LDM-4 (20226)
9.39
0.246
LaMa (2022)
12.0
0.24
DeepFill v2 (2019)
11.51
17.70
MaskGIT (2022h)
7.80
22.95
DALL-E (2021)
10.32
27.24
SD (20226)
9.62
27.46 0.198
PixArt-a (2024b)
7.32
Next (2024)
0.203
PixWizard 35.46 15.76 32.01
33.83
16.94
31.84
9.27
0.25
7.54
22.18
9.56
27.47 0.183
Controllable Generation Results. Without the need for separate training for each model, Pix Wizard an all-in-one solution capable of handling multiple conditions. As shown in Table 3 and Fig. Pix Wizard achieves the highest controllability and best image quality under depth conditions, while also being comparable to current separate models in image-text alignment.
Inpainting Results. As shown in Table 3, Pix Pizard outperforms other inpainting approaches, improving overall image quality based on FID and LPIPS scores. The qualitative examples in Fig. further demonstrate PixWizard effectiveness in generating coherent content. Building its strong inpainting capabilities, Pix Wizard enables users to perform more precise image editing tasks, such as Remove Anything, Replace Anything, and Add Anything. More details can be found in Sec. D.4
Outpainting Results. As shown in the quantitative comparison results in Table 3, PixWizard
outperforms other baselines in the outpainting task, delivering state-of-the-ard image generation quality with a FID score of 7.54 and an IS score of 22.18. The samples in Fig. demonstrate PixWizard's ability synthesize images in various scenes and styles, and flexibly handles image extrapolation in multiple directions and aspect ratios with better marginal coherence.
Text-to-Image Results. As the results shown in Table 3, Pix PixWizard achieves FID score of 9.56 when tested for zero-shot performance on the COCO dataset. Although some models achieve lower FID, they focus solely on text-to-image tasks and rely on significantly more training resources. Additionally, we evaluated the Human Preference Score (HPS v2), robust benchmark for assessing human preferences in text-to-image synthesis. Pix Wizard performed well, delivering image quality comparable to popular text-to-image generators. We provide visual samples in Fig. 9. Pix Wizard supports high-resolution image synthesis, up to 1024 1024, with any resolution and aspect ratio.
leseding bode aaN S FRE
masers diwingg nn
kiroe fe Boon Shot farus Rree eversa conpies
the Opcie bo
3o IMAC sole
POROUT wevocope soat and dary Shanc
FoBte ERom Clann Inpainting
Outpainting
Text-to-Image
Figure 9: Visualization results of Inpainting, Outpainting and Text-to-Image Generation.

---

Published as a conference paper at ICLR 2025
4.4 ABLATION STUDY
We performed ablation studies to assess the impact of each component's design and the training process on learning in PixWizard. Given computational limitations, we conducted the ablation on PixWizard, training for 40k steps.
Table 4: Comparison of the models with two different guidances, dynamic semantic tokens sampling (DSTS), and two-stage training and data balancing strategy for different tasks.
Deraining Rain100L
RefCOCO
Depth Estim. Image Editing
Canny-to-Image MultiGen-20M
Inpainting Places
Methods
val
NYUv2
Emu Edit
PSNR1
cloUt
RMSEL
CLI
F11
FIDI
FIDI
MI
(-.3)
40.78-0.94)
(00000) (0.00))
32.98
17.41
10.91 (00000
M2
14.72
18.43,
0.586 262)
0.071 (-0.11)
11.12 89)
19.34,
13.87 (-2.9)
PixWizard wo DSTS
30.19 (0.00)
41.66 (000))
0.318 000000
0.091
(+0.000)
32.93
(2.00)
10.93 +7.02
Pix wo two-stage 29.17 (1007)
40.18
1.04)
0.322 (10.0000000 002)
0.085
0.000)
32.87 0.144
17.21 (.007)
10.97 0.00)
Pix PixWizard
30.24
41.72
0.324
0,088
33.01
17.14
10.88
Structural-Aware vs. Semantic-Aware Guidance. We demonstrate the importance of integrating both structure-aware and semantic-aware guidance to enhance Pix Wizard's performance across diverse tasks. To validate this, we trained two additional models: M 1 with only the structure-aware module and M2 with only the semantic-aware module. Results show that M1 performs better on tasks requiring preservation of image structure and details, while M2, which relies solely on cross- attention for injecting image features, struggles with most visual tasks and often generates outputs that deviate from the input. However, as noted by other methods (Ye et al., 2023; Hu al., 2024), semantic guidance excels in tasks needing flexibility, such as text-to-image generation, conditional generation, and image editing. In comparison, Pix Wizard, which combines both modules, achieves balanced performance across broader range of tasks.
Influence of Dynamic Semantic Tokens Sampling. As shown in Table 4, the impact of Dynamic Semantic Tokens Sampling (DSTS) on task performance is minimal, but it improves overall perfor- mance on average. This indicates that modeling semantic tokens for the entire image is unnecessary. By dynamically sampling relevant features for each task's 's focus, the model operates more efficiently. Moreover, using fewer tokens reduces computational load during attention calculations, resulting in faster inference.
Influence of Two-Stage Training and Data Balancing. Table 4 presents the results of our proposed two-stage training and data balancing strategy. As shown, our approach is crucial, as the two-stage method significantly improves performance on tasks with smaller datasets while maintaining the same number of training steps and achieving faster convergence. Notably, for tasks with larger datasets, the performance remains comparable.
DISCUSSION AND CONCLUSION
In this work, we explore how to build versatile interactive image-to-image visual assistant from three key aspects: task definition, data construction, and model architecture. Our goal is to create a system that can precisely follow free-form user instructions for image generation, manipulation, and translation. Our Pix Wizard eliminates the need for task-specific design choices and achieves highly competitive performance across a diverse set of tasks, with strong generalization capabilities.
However. this work has some limitations. First, the current model architecture does not yet support multi-image input conditions, which is an increasingly important research area. Second, there is room for improvement in challenging tasks like segmentation and image grounding compared to specialized models. Additionally, the performance of the text encoder and foundation model also plays a crucial role. Better text encoding improves the model's ability to understand and execute instructions, while larger, more robust architectures enhance output quality. Notably, the modules and strategies proposed in Pix Pizard can be easily applied other powerful text-to-imant to-image generators. In the future, we plan to explore more advanced diffusion models, such as SD3 and FLUX, and continue advancing this field toward "GPT-4 moment".
10

---

Published as a conference paper at ICLR 2025
ETHICS IMPACT
Modern text-to-image generation models, such as Stable Diffusion (Rombach et al., 2022b) and Lumina-T21 (Zhuo al., 2024), raise ethical concerns, including social bias. PixWizard, which uses similar large-scale datasets, faces these same challenges, such as creating and spreading manipulated data or misinformation. Future work will focus on developing responsible use framework, balancing the benefits of research transparency with the risks of open access, to ensure safe and beneficial use.
Notably, datasets and benchmarks used PixWizard were legally sourced from public repositories under academic-use licenses. Additionally, any extra image sources used in the paper have been responsibly sourced to ensure they are free, copyright-free and commercially usable, preventing legal violations. We take full responsibility for adhering data licenses and managing any legal risks.
7 ACKNOWLEDGEMENT
We thank Ziheng Wu, Huan Teng, and Tonton Su for their valuable comments and suggestion to this project, including data construction, model discussions, and evaluation work. This project is funded part by National Key R&D Program of China Project 20222D016110M by the Centre for Perceptual and Interactive Intelligence (CPII) Ltd under the Innovation and Technology Commission (TTC)'s 's InnoHK, by NSFC-RGC Project N 498/44 Hongsheng Li is of CPII under the InnoHK.
REFERENCES
Abdelrahman Abdelhamed, Stephen Lin, and Michael S Brown. A high-quality denoising dataset for smartphone cameras. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1692-1700, 2018.
Abdullah Abuolaim and Michael S Brown. Defocus deblurring using dual-pixel data. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part X 16, pp. 111-126. Springer, 2020.
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et Gpt-4 technical report. arXiv preprint arXhv:2303.08774 2023.
Eirikur Agustsson and Radu Timofte. Nure 2017 challenge on single image super-resolution: Dataset and study. In Proceedings of the IEEE conference computer vision and pattern recognition workshops, pp. 126-135, 2017.
Yuang Ai, Huaibo Huang, Xiaogiang Zhou, Jiexiang Wang, and Ran He. Multimodal prompt perceiver: Empower adaptiveness generalizability and fidelity for all-in-one image restoration. In Proceedings of the IEEE/CVE Conference Computer Vision and Puttern Recognition, pp.
25432-25444, 2024.
Michael Samuel Albergo and Eric Vanden-Eijnden. Building normalizing flows with stochastic interpolants. In The Eleventh International Conference on Learning Representations, 2023. URL
pettps://openretee
net fent//foummiddi id=1 BgBBbCR
Codruta O Aneuti, Cosmin Ancuti, Mateu Sbert, and Radu Timofte. Dense-haze: benchmark for image dehazing with dense-haze and haze-free images. In 2019 IEEE international conference on image processing (ICIP), pp- 1014-1018. IEEE, 2019.
Codruta O Ancuti, Cosmin Ancuti, and Radu Timofte. Nh-haze: An image dehazing benchmark with non-homogeneous hazy and haze-free images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops, pp. 444-445, 2020.
Omri Awrahami, Dani Lischinski, and Ohad Fried. Blended diffusion for text-driven editing of natural images. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 88208-18216 2022.
11

---

Published as a conference paper at ICLR 2025
Roman Bachmann, Oğuzhan Fatih Kar, David Mizrahi, Ali Garjani, Mingfei Gao, David Griffiths, Jiaming Hu, Afshin Dehghan, and Amir Zamir, 4m-21: An any-to-any vision model for tens of tasks and modalities. arXiv preprint arXiv:2406.094 2024.
Gwangbin Bae and Andrew J. Davison. Rethinking inductive biases for surface normal estimation. In JEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024.
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncer- tainty in surface normal estimation. In Proceedings of the IEEE/CVF International Conference Computer Vision. pp. 13137-13146, 2021.
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan L Yuille, Trevor Darrell, Jitendra Malik, and Alexei Efros. Sequential modeling enables scalable learning for large vision models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22861-22872, 2024.
Dmitry Baranchuk, Andrey Voynov, Ivan Rubachev, Valentin Khrulkov, and Artem Babenko. Label- efficient semantic segmentation with diffusion models. In International Conference on Learning Representations, 2022. URL ht tps: /ppp/ppeeevieeeeeeeeeeeeeeee www./nee//eeeee fte//foruu forum 1=$1===222x222
James Betker, Gabriel Goh, Li Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang. Joyce Lee, Yufei Guo, al. Improving image generation with better captions. Computer Science. https://cdn. openai. com/papers/ialll--. pdf. 2(3):8, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 13392-18402. 2023.
Tim Brooks, Bill Peebles, Connor Holmes, Will DePue, Yufci Guo, Li Jing, David Schnurr, Joe Taylor, Troy Luhman, Eric Luhman, Clarence Ng. Ricky Wang, and Aditya Ramesh. Video generation models as world simulators. 2024. URL
ctm///ccooo///oooooooo./ooc wwt.://oo./...eee. research/
video- generat iion coom--ooelss as world simulators.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-1901, 2020.
Yohann Cabon, Naila Murray, and Martin Humenberger. Virtual kitti 2. arXiv preprint
arXiv:2001.10773 2020.
Jianrui Cai, Hui Zeng, Hongwei Yong. Zisheng Cao, and Lei Zhang. Toward real-world single image super-resolution: A new benchmark and a new model. In Proceedings of the IEEE International Conference on Computer Vision, 2019.
John Canny. A computational approach to edge detection. JEEE Transactions on pattern analysis and machine intelligence, (6):679-66) 698, 1986.
multi-person Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh. Realtime multi person 2d pose estimation using part affinity fields. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 7291 1229,7 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William Freeman. Maskgit: Masked generative image transformer. In Proceedings the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11 11315-11322 13325, 2022a.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11315-11325 2022b.
12

---

Published as a conference paper at ICLR 2025
Junsong Chen, Chongjian Ge, Enze Xie, Yue Wu, Lewei Yao, Xiaozhe Ren, Zhongdao Wang, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- ssima: Weak-to-strongg training of diffusion transformer for 4k -oxt-to-image generation. arXiv preprint arXiv:2403.04699 2024a.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart- StalphaS: Fast training of diffusion transformer for photorealistic text-to-image synthesis. The Twelfih International Conference on Learning Representations, 2024b. URL
ew. net / /t///oormm/ i d=eAKm@Pe.3ml
Liangyu Chen, Xiaojie Chu, Xiangyu Zhang, and Jian Sun. Simple baselines for image restoration. In European conference on computer vision, pp. 17-33. Springer, 2022a.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J Fleet, and Geoffrey E Hinton. unified sequence interface for vision tasks. Advances in Neural Information Processing Systems, 35: 31333-31346, 2022b.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked- attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1290-1299 1299, 2022.
Marcos V Conde, Gregor Geigle, and Radu Timofte. High-quality image restoration following human instructions. arXiv preprint arXiv:240I. 16468, 2024.
Antonio Criminisi, Patrick Perez, and Kentaro Toyama. Object removal by exemplar- seesela-based inpainting. In 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings. volume 2. pp. II-II. IEEE, 2003.
Antonio Criminisi, Patrick Pérez, and Kentaro Toyama. Region filling and object removal by exemplar-based image inpainting. IEEE Transactions on image processing, 13(0):1200-1212 2004.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 31:8780-87799 0794,, 2021.
Rumpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, and Li Yi. DreamLL.M: Synergistic multimodal comprehension and creation. In The Twelfth International Conference on Learning Representations, 2024. URL https: http///pppppeeeecp////pee net/ forum? 1d-y01kGvd9Bw.
Omar Elharrouss, Noor Almaadeed, Somaya AI- Ml-Meadeed, and Younes Akbari. Image inpainting: review. Neural Processing Letters, 51:2007-2022 2020.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEF/CVF conference on computer vision and pultern recognition, pp. 12873-12883, 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning. 2024.
Xueyang Fu, Jiabin Huang, Delu Zeng, Yue Huang, Xinghao Ding, and John Paisley. Removing rain from single images via a deep detail network. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3855-3863. 2017.
Yulu Gan, Sungwoo Park, Alexander Marcel Schubert, Anthony Philippakis, and Ahmed Alaa. InstructCV: Instruction-tunel text-to-image -moi diffusion models vision generalists. In The Twelfth International Conference on Learning Representations, 2024, URL https: httpts/::///peeeee /pppereeview. net/ forum? i d=Nu 9mH
13

---

Published as a conference paper at ICLR 2025
Peng Gao, Renrui Zhang, Chris Liu, Longtian Qiu, Siyuan Huang, Weifeng Lin, Shitian Zhao, Shijie Geng, Ziyi Lin, Peng Jin, et al. Sphinx-x: Scaling data and parameters for a family of multi-modal large language models. ICML 2024, 2024.
Yuying Ge. Sijie Zhao, Chen Li, Yixiao Ge, and Ying Shan. leed-data-edie technical report: A hybrid dataset for instructional image editing. arXiv preprint arXiv:2405.04000 04000, 2024.
Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang. Houqiang Li, Han Hu, et al. Instructdiffusion: generalist modeling interface for vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12709-12720, 2024.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang, Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for 3d understanding, generation, and instruction following. arXiv preprint arXiv:2309.006/ 0061, 2023.
Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Chengzhuo Tong, Peng Gao, Chunyuan Li, and Pheng- Ann Heng. Sam2point: Segment any 3d videos in zero-shot and promptable manners. arXiv preprint arXiv:2408. 16768, 2024.
Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng- Ann Heng. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprini 2xXX::2501113220 2025.
Jiaming Han, Renrui Zhang, Wenqi Shao, Peng Gao, Peng Xu, Han Xiao, Kaipeng Zhang, Chris Liu, Song Wen, Ziyu Guo, et al. Imagebind-llm: Multi-modality instruction tuning. arXiv preprint SXXi::2309.09000 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv preprint arXiv:2207.12596. 2022.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Jonathan Ho, Tim Salimans, Alexey Gritsenko, William Chan, Mohammad Norouzi, and David J Fleet. Video diffusion models. Advances in Neural Information Processing Systems, 35:8633-8666 2022.
Hexiang Hu, Kelvin CK Chan, Yu-Chuan Su, Wenhu Chen, Yandong Li, Kihyuk Sohn, Yang Zhao, Xue Ben, Boqing Gong, William Cohen, et al. Instruct-imagen: Image generation with multi- modal instruction. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4754-4763, 2024.
Huaibo Huang, Mandi Luo, and Ran He. Memory uncertainty learning for real-worke single image deraining. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(3):3446-3466 2022.
Rongjie Huang, Jiawei Huang, Dongchao Yang. Yi Ren, Luping Liu, Mingze Li, Zhenhui Ye, Jinglin Liu, Xiang Yin, and Zhou Zhao. Mlake-an-audioo Text-to-audio generation with prompt-enhanced diffusion models. In International Conference on Machine Learning, pp. 13916-13932. PMLR, 2023.
Mude Hui, Siwei Yang, Bingchen Zhao, Yichun Shi, Heng Wang, Peng Wang, Yuyin Zhou, and Cihang Xie. Hq-edit: A high-quality dataset for instruction- image editing. arXiv preprins 0:Xi::2404.099900 2024.
Jitesh Jain, Jiachen Li, Mang Tik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi. Oneformer: One transformer to rule universal image segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 2989-2998, 2023.
Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv: 16/ 1.01 144, 2016.
14

---

Published as a conference paper at ICLR 2025
Dongzhi Jiang, Guanglu Song, Xiaoshi Wu, Renrui Zhang, Dazhong Shen, Zhuofan Zong, Yu Liu, and Hongsheng Li. Comat: Aligning text-to-image diffusion model with image-to-text concept matching. arXiv preprint (XXv::244403653, 2024a.
Dongzhi Jiang, Renrui Zhang, Ziyu Guo, Yanmin Wu, Jiayi Lei, Pengshuo Qiu, Pan Lu, Zehui Chen, Guanglu Song, Peng Gao, et al. Mmsearch: Benchmarking the potential of large models as multi-modal search engines. arXiv preprint arXiv:2409. 12959, 2024b.
Dongzii Jiang, Renrui Zhang, Ziyu Guo, Yanwei Li, Yu Qi, Xinyan Chen, Liuhui Wang. Jianhan Jin, Claire Guo, Shen Yan, et al. Mme-cot: Benchmarking chain-of-thought in large multimodal models for reasoning quality, robustness, and efficiency. arXiv preprint arXiv:2502.090 2502209211 2025.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Bahjat Kawar, Shiran Zada, Oran Lang, Omer Tov, Huiwen Chang, Tali Dekel, Inbar Mosseri, and Michal Irani. Imagic: Text-based real image editing with diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6007-6017, 2023.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion- image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
9492-9502, 2024a.
Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing iiffuusionbbssee image generators for monocular depth estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9492-9502, 2024b.
Diederik P Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes. arXiv preprint
Jing Yu Koh, Daniel Fried, and Russ R Salakhutdinov. Generating images with multimodal language models. Advances Neural Information Processing Systems, 36, 2024.
Zhifeng Kong, Wei Ping, Jiaji Huang, Kexin Zhao, and Bryan Catanzaro. Diffwave: A versatile diffusion model for audio synthesis. In International Conference on Learning Representations, 2021. URL hwwps:ttt.........t....tt openrevieeeeee net. Corrmmiideee d=a-- x7K8
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123:32-73, 2017.
Hsin-Ying Ying Lee, Hung- Yu Tseng, and Ming-Hsuan Yang. Exploiting diffusion prior for generalizable dense prediction. Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 7861-7871, 2024.
Jiayi Lei, Renrui Zhang, Xiangfei Hu, Weifeng Lin, Zhen Li, Wenjian Sun, Ruoyi Du, Le Zhuo, Zhongyu Li, Xinyue Li, et al. Imagine-e: Image generation intelligence evaluation of state-of-the- art text-to-image models. arXiv preprint 25011 2025.
Boyi Li, Wenqi Ren, Dengpan Fu, Dacheng Tao, Dan Feng. Wenjun Zeng, and Zhangyang Wang. Benchmarking single-image dehazing and beyond. IEEE Transactions on Image Processing, 28 (1):492-505, 2018.
Boyun Li, Xiao Liu, Peng Hu, Zhongqin Wu, Jiancheng Lv, and Xi Peng. All-in-one image restoration for unknown corruption. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 17452-17462 2022.
Chongyi Li, Chunle Guo, Wenqi Ren, Rummin Cong, Junhui Hou, Sam Kwong, and Dacheng Tao. An underwater image enhancement benchmark dataset and beyond. IEEE transactions on image processing, 29:4376-433 4389, 2019a.
15

---

Published as a conference paper at ICLR 2025
Feng Li, Hao Zhang, Huaizhe Xu, Shilong Liu, Lei Zhang. Lionel M Ni, and Heung- Yeung Shum. Mask dino: Towards a unified transformer- based framework for object detection and segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 3041-3050, 2023.
Ming Li, Taojiannan Yang, Huafeng Kuang. Jie Wu, Zhaoning Wang, Xuefeng Xiao, and Chen Chen. Controlnet+4: Improving conditional controls with efficient consistency feedback. arXiv preprint arXiv:2404.07987. 2024.
Ruoteng Li, Loong- Cheong, and Robby T Tan. Heavy rain image restoration: Integrating physics model and conditional adversarial learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 1633-1642, 2019b.
Jingyun Liang, Jiezhang Cao, Guolei Sun, Kai Zhang. Luc Van Gool, and Radu Timofte. Swinir: Im- age restoration using swin transformer. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 1833-1844, 2021.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision- ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13. pp. 740-755. Springer, 2014.
Yaron Lipman, Ricky Q. Chen, Heli Ben-Hamu, Maximilian Nickel, and Matthew Le. Flow matching for generative modeling. In The Eleventh International Conference Learning Repre- sentations, 2023. URL https: sepenneview.nee forum? id=P gVMRDDJTSt..
Chang Liu, Henghui Ding, and Xudong Jiang, Gres: Generalized referring expression segmentation. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 23592 226011, 2023a.
Xingchao Liu, Chengyue Gong, and qiang liu. Flow straight and fast: Learning generate and transfer data with rectified flow. In The Eleventh International Conference on Learning Representations. 2023b. URL https: wpw.reveeeeeeeeeeeeeeeeeeeeeeeee net/ wwwpeeewwwweeeeeooueeeeteeeeeeee wow.mmmdddXV//Tlianaa
Yang Liu, Zhen Zhu, and Xiang Bai. Wdnet: Watermark-deccoppsitiie network for visible watermark removal. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 3685-6693, 2021.
Yun-Fu Liu, Da-Wet Jaw, Shib-Chia Huang, and Jenq-Neng Hwang. Desnownet: Context-aware deep network for snow removal. IEEE Transactions Image Processing, 27(3)30064 3073, 2018.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes the wild. In Proceedings of International Conference on Computer Vision (ICCV), December 2015.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-mode modal tasks. In The Eleventh International Conference on Learning Representations, 2022a.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified- io: A unified model for vision, language, and multi-modas tasks. In The Eleventh International Conference on Learning Representations, 2022b.
Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, and Aniruddha Kembhavi. Unified-io 2: Scaling autoregressive multimodal models with vision language audio and action. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26439-26444 26455, 2024.
Ziwei Luo, Fredrik K Gustafsson, Zheng Zhao Jens Sjölund, and Thomas B Schön. Controlling vision-language models for universal image restoration. arXiv preprint arXiv:2310.0101 2023.
Nanye Ma, Mark Goldstein, Michael S Albergo. Nicholas M Boffi, Eric Vanden-Eijnde and Saining Xie. Sit: Exploring flow and transformers. arXiv preprint 240.00000000 2024.
generative models with scalable interpolant
16

---

Published as a conference paper at ICLR 2025
David Martin, Charless Fowlkes, Doron Tal, and Jitendra Malik. A database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. In Proceedings eighth IEEE international conference computer vision. ICCV 2001, volume 2, pp. 416-423. IEEE, 2001.
David Mizrahi, Roman Bachmann, Oguzhan Kar, Teresa Yeo, Mingfei Gao, Afshin Dehghan, and Amir Zamir. 4m: Massively multimodal masked modeling. Advances Neural Information Processing Systems, 36, 2024.
Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 4296-4304, 2024.
Seungjun Nah, Tae Hyun Kim, and Kyoung Mu Lee. Deep multi-scale convolutional neural network for dynamic scene deblurring. In Proceedings of the IEEE conference computer vision and paltern recognition, pp. 3883-3891, 2017.
Seungjun Nah, Sungyong Baik, Seokil Hong. Gyeongsik Moon, Sanghyun Son, Radu Timofte, and Kyoung Mu Lee. Ntire 2019 challenge on video deblurring and super-resolution: Dataset and study. In CVPR Workshops, June 2019.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
Bowen Peng and Jeffrey Quesnelle. Ntk-aware scaled rope allows Ilama models to have extended (8k+) context size without any fine-tuning and minimal perplexity degradation, 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952 2023.
Vaishnav Potlapalli, Syed Waqas Zamir, Salman Khan, and Fahad Shahbaz Khan. Promptir: Prompting for all-in-one image restoration. Advances in Neural Information Processing Systems, 36, 2024.
Rui Qian, Robby T Tan, Wenhan Yang, Jiajun Su, and Jiaying Liu. Attentive generative adversarial network for raindrop removal from a single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2482-2491, 2018.
Liangqiong Qu, Jiandong Tian, Shengfeng He, Yandong Tang, and Rynson Lau Deshadownet: A multi-context embedding deep network for shadow removal. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4067-4075, 2017.
Ruijie Quan, Xin Yu, Yuanzhi Liang, and Yang. Removing raindrops and rain streaks in one go. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 9147-9156, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 7748-8763. PMLR, 2021.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot tox-tto-image generation. In International conference on machine learning, pp. 8821-8831. Pmlr, 2021.
Yongming Rao. Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsich. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:1 13937-13949, 2021.
Jaesung Rim, Haeyun Lee, Jucheol Won, and Sunghyun Cho. Real-world blur dataset for learning and benchmarking deblurring algorithms. In Computer Eiiio-ECCVV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 184-201. Springer, 2020.
17

---

Published as a conference paper at ICLR 2025
Mike Roberts, Jason Ramapuram, Anurag Ranjan, Atulit Kumar, Miguel Angel Bautista, Nathan Paczan, Russ Webb, and Joshua M. Susskind. Hypersim: A photorealistic synthetic dataset for holistic indoor scene understanding. In International Conference on Computer Vision (ICCV) 2021, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz. Patrick Esser, and Björ Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022a.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- computer vision and pattern recognition, pp. 10684-10695, 2022b.
Chitwan Saharia, William Chan, Huiwen Chang, Chris Lee, Jonathan Ho, Tim Salimans, David Fleet, and Mohammad Norouzi. Palette: Image-to-image diffusion models. In ACM SIGGRAPHI 2022 conference proceedings, pp. 1-10, 2022a.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances neural information processing systems, 05-36479-364999 2022b.
Chitwan Saharia, Jonathan Ho, William Chan, Tim Salimans, David J Fleet, and Mohammad Norouzi. Image super-resolution via iterative refinement. IEEE transactions on pattern analysis and machine intelligence, 45(4):4713 2022c.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 15:25278-252991 2022.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871-8879, 2024.
Jing Shi, Ning Xu, Trung Bui, Franck Dernnncour, Zheng Wen, and Chenliang Xu. A benchmark and baseline for language-driven image editing. Proceedings of the Asian Conference on Computer Vision, 2020.
Nathan Silberman, Derek Hoiem, Pushmeet Kohli, and Rob Fergus. Indoor segmentation and support inference from rgbd images. In Computer Esion-ECCV 2012: 12th European Conference on Computer Vision, Florence, Italy, October 7-13, 2012, Proceedings, Part V 12, pp. 746-760. Springer, 2012.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. International conference on machine learning. pp. 2255-2265.. PMLR, 2015.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in neural information processing systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In International Conference on Learning Representations, 2021. URL https: http////pp///eeee///etppee net/forum? id-PxTIG12RRHS.
Xavier Soria, Yachuan Li, Mohammad Rouhani, and Angel D. Sappa. Tiny and efficient model for the edge detection generalization. In Proceedings of the IEFE/CVF International Conference on Computer Vision (ICCV) Workshops, pp. 1364-1373, October 2023.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: Enhanced transformer with rotary position embedding. Nearocomputing, 568: 127063, 2024.
18

---

Published as a conference paper at ICLR 2025
Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, and Xinlong Wang. Emu: Generative pretraining in multimodality.. In The Twelfth International Conference on Learning Representations, 2023.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Aleksei Silvestrov, Naejin Kong. Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution- robust large mask inpainting with fourier convolutions. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 2149-2159, 2022.
Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. Gemma: Open models based gemini research and technology. arXiv preprint 2xix:2203308225,, 2024.
Guy Tevet, Sigal Raab, Brian Gordon, Yoni Shafir, Daniel Cohen-or, and Amit Haim Bermano. Human motion diffusion model. In The Eleventh International Conference on Learning Represen- tations, 2023. URL https: wpwprevieww.e.ee/////p/.. net. uo iisermmmmddsJsssyy jwa.
Alexander Tong. Kilian FATRAS, Nikolay Malkin, Guillaume Huguet, Yanlei Zhang, Jarrid Rector- Brooks, Guy Wolf, and Yoshua Bengio. Improving and generalizing flow-based generative models with minibatch optimal transport. Trunsactions on Machine Learning Research, 2024. ISSN 2835- 8856. URL https:/ /tpp/pppnevviieeeeeee net forum? wwwrummiddcDD9eee 3AW. Expert Certification.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint vrXiv:2302.13971, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017
Longguang Wang, Yulan Guo, Yingqian Wang, Juncheng Li, Shuhang Gu, Radu Timofte, Ming Cheng, Haoyu Ma, Qiufang Ma, Xiaopeng Sun, et al. Ntire 2023 challenge on stereo image super-resolution: Methods and results. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1346-1372, 2023a.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-ssquenee learning framework. In International conference on machine learning, pp. 23318-27 2-34300 PMLR, 2022a.
Tengfei Wang, Ting Zhang, Bo Zhang, Hao Ouyang, Dong Chen, Qifeng Chen, and Fang Wen. Pretraining all you need for image-to-image translation. arXiv preprint arXiv:2205.122 12952. 2022b.
Xinlong Wang, Wen Wang, Yue Cao, Chunhua Shen, and Tiejun Huang. Images speak in images: A generalist painter for in-context visual learning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6830-6839, 2023b.
Zhaoqing Wang, Yu Lu, Qiang Li, Xunqiang Tao, Yandong Guo, Mingming Gong, and Tongliang Liu. Cris: Clip-driven referring image segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11686-11695 2022c.
Zhendong Wang, Yifan Jiang, Yadong Lu, Pengcheng He, Weizhu Chen, Zhangyang Wang, Mingyuan Zhou, et al. In-context learning unlocked for diffusion models. Advances in Neural Information Processing Systems, 36:8542-8566 2023c.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning.
19

---

Published as a conference paper at ICLR 2025
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to-image synthesis. arXiv preprint arXiv:2306. N.X.022306 2023.
Bin Xia, Yulun Zhang, Shiyin Wang, Yitong Wang, Xinglong Wu, Yapeng Tian, Wenming Yang. and Luc Van Gool. Diffir: Efficient diffusion model for image restoration. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 13095-13105, 2023.
Xiaoyu Xiang, Ding Liu, Xiao Yang, Yiheng Zhu, and Xiaohui Shen. Anime2sketch: A sketch extractor for anime arts with deep networks, 2021.
com/ mww.//meoeemee/eemmmmmmmm.///
ani anime2sketch.
Saining Xie and Zhuowen Tu. Holistically-nested edge detection. In IEEE International Conference on Computer Vision 2015.
Jiarui Xu, Sifei Liu, Arash Vahdat, Wonmin Byeon, Xiaolong Wang, and Shalini De Mello. Open- vocabulary panoptic segmentation with text-to-image diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2955-2966, 2023.
Lihe Yang. Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 10371-10381, 2024a.
Lihe Yang, Bingyi Kang, Zilong Huang, Zhen Zhao, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything v2. arXiv:2406. 4,4V:,,406.09414, 2024b.
Wenhan Yang, Robby T Tan, Jiashi Feng, Jiaying Liu, Zongming Guo, and Shuicheng Yan. Deep joint rain detection and removal from single image. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1357-1366, 2017.
Wenhan Yang, Wenjing Wang, Haofeng Huang, Shiqi Wang, and Jiaying Liu. Sparse gradient regularized deep retinex network for robust low-light image enhancement. IEEE Transactions on Image Processing, 30:2072 200-2022-206, 2021.
Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, and Philip HS Torr. Lavt: Language-aware vision transformer for referring image segmentation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 18155-18165, 2022.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, et al. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint arXiv:2408. 01800, 2024.
Hanrong Ye and Dan Xu. Inverted pyramid multi-task transformer for dense scene understanding. In European Conference on Computer Vision, pp. 514-530. Springer, 2022.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.0660 06721, 2023.
Ahmet Burak Yildirim, Vedat Baday, Erkut Erdem, Aykut Erdem, and Aysegul Dundar. Inst-inpaint: Instructing to remove objects with diffusion models. arXiv preprint arXiv:2304.032466 2023.
Jiahui Yu, Zhe Lin, Jimei Yang, Xiaohui Shen, Xin Lu, and Thomas S Huang. Free-form image inpainting with gated convolution. Proceedings of the IEEE/CVE international conference on computer vision, pp. 4471-4480. 2019.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg. and Tamara L Berg. Modeling context in referring expressions. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part II 14, pp. 69-85. Springer, 2016.
Xin Yu, Peng Dai, Wenbo Li, Lan Ma, Jiajun Shen, Jia Li, and Xiaojuan Qi. Towards efficient and scale-robust hitt--iigh-dffnnttion image demoiréing. In European Conference on Computer Vision, pp. 646-662. Springer, 2022.
20

---

Published as a conference paper at ICLR 2025
Syed Waqas Zamir, Aditya Arora, Salman Khan, Munawar Hayat. Fahad Shahbaz Khan, and Ming-Hun Yang. Restormer: Efficient transformer for high-resolution image restoration. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5728- 5739, 2022.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-uuie guided image editing. Advances Neural Information Processing Systems, 36, 2024a.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1836-3847, 2023a.
Mingyuan Zhang. Zhongang Cai, Liang Pan, Fangzhou Hong. Xinying Guo, Lei Yang, and Ziwei Liu. Motiondiffuse: Text-driven human motion generation with diffusion model. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024b.
Renrui Zhang, Zhengkai Jiang, Ziyu Guo, Shilin Yan, Junting Pan, Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. ICLR 2024, 2023b.
Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Alama-adaptee Efficient fine-tuning of large language models with zero-initialized attention. In ICLR 2024, 2024c.
Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et al. Mathverse: Does your multi-modal Ilm truly see the diagrams visual math problems? ECCV 2024, 2024d.
Renrui Zhang, Xinyu Wei, Dongzhi Jiang, Yichi Zhang, Ziyu Guo, Chengzhuo Tong, Jiaming Liu,
Aojun Zhou, Bin Wei, Shanghang Zhang, et al. Mavis: Mathematical visual instruction tuning. arXiv preprint (tt.::2400000733) 2024c.
Shiwen Zhang, Shuai Xiao, and Weilin Huang. Forgedit: Text guided image editing via learning and forgetting. arXiv preprint arXiv:2309. 10556, 2023c.
Haozhe Zhao, Xiaojian Ma, Liang Chen, Shuzheng Si, Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. Ultraedit: Instruction-batt fine-grained image editing at scale. arXiv preprint arXiv:2407.05288 2024.
Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene recognition using places database. Advances in neural information processing systems, 27, 2014.
Bolei Zhou, Agata I Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 million image database for scene recognition. IEEE transactions on pattern analysis and machine intelligence, 40(6): 1552-14644 2017a.
Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. Scene parsing through ade20k dataset. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 633-641, 2017b.
Yuqian Zhou, David Ren, Neil Emerton, Sehoon Lim, and Timothy Large. Image restoration for under-display camera. In Proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 9179-9188, 2021.
Yurui Zhu, Tianyu Wang, Xueyang Fu, Xuanyu Yang, Xin Guo, Jifeng Dai, Yu Qiao, and Xiaowei Hu. Learning weather-geneee generall and weather-specific features for image restoration under multiple adverse weather conditions. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 21747-21758, 2023.
Le Zhuo Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongjic Huang, Wenze Liu, Lirui Zhao, Fu- Yum Wang, Zhanyu Ma. et al. Lumina-ness next: Making lumina-t2 12x stronger and faster with next-dit. arXiv preprint axXiv:2406.185e 2024.
21

---

Published as a conference paper at ICLR 2025
APPENDIX
A RELATED WORK
Diffusion Models. Diffusion models estimate the data distribution by modeling the gradient of the noise- case-perturbed data distributions (Sohl-Dickstein et al., 2015; Song & Ermon, 2019; Ho et al., 2020; Dhariwal & Nichol, 2021; Song et al., 2021). They have demonstrated remarkable performance in various fields, ranging from text-to-image generation (Rombach et al., 2022a; Saharia et al., 2022b; Betker et al., 2023; Jiang al., 2024a: Zhang al., 2023b), controllable generation (Zhang et al., 2023a; Ye et al., 2023), and image editing (Avrahami et al., 2022; Brooks et al., 2023; Kawar et al., 2023) to video (Ho al., 2022; Brooks et al., 2024), audio (Kong et al., 2021; Huang al., 2023), 3D (Guo et al., 2023; 2024), and motion (Tevet et al., 2023; Zhang et al., 2024b) generation. Beyond generation, recent works have also exhibited diffusion models' capabilities in computer vision tasks, such as semantic segmentation (Baranchuk et al., 2022: Xu et al., 2023), depth estimation (Ke et al., 2024b; Lee et al., 2024), image restoration (Xia et al., 2023), and reasoning enhancement Guo et al. (2025). Benefiting from the visual knowledge learned from large-scale pretraining, these works open up the potential for adapting pretrained diffusion models to downstream tasks generative manner, with excellent capabilities like handling inherent uncertainties and zero-shot generalization.
Vision Generalists. Building a vision generalist capable of unifying various visual tasks has been a long-standing goal. Inspired by the success of scaling sequential modeling with transformers in natural language processing, many vision generalists (Wang et al., 2022a; Chen et al., 2022b; Lu et al., 2022b; 2024; Mizrahi et al., 2024; Bachmann et al., 2024; Bai et al., 2024) follow similar approach by converting inputs and outputs into sequences of discrete tokens (Van Den Oord et al., 2017: Esser et al., 2021), ), allowing joint modeling of different modalities within unified framework. With the rapid advancements in large language models (LLMs) (Achiam et al., 2023; Touvron et al., 2023; Zhang al., 2024c) and multi-modal large language models (MLLMs) (Gao et al., 2024; Zhang et al., 2024e;d; Jiang et al., 2024b; 2025), several works (Koh et al., 2024; Dong et al., 2024; Sun et al., 2023; Wu et al.; Han et al., 2023) have introduced task-specific tokens, aligning them with the embedding space of LLMs to equip text-only LLMs with the ability to perceive and generate images, However, a common drawback of these approaches is their limited performance on generation tasks, such as text-to-image generation and image editing. Additionally, their sampling efficiency is constrained by the next-token prediction paradigm, which worsens for saage-to-image tasks or when working with high-resolutie images. In contrast, diffusion models, known for their state-of-the-an generation performance and efficiency, are ideal candidates for image generation and manipulation tasks. Recent efforts (Wang et al., 2023c; Gan et al., 2024; Hu et al., 2024; Geng et al., 2024; Lei et al., 2025) aim to create a visual generalist by unifying multiple visual tasks through a natural language interface based on pretrained eext-to-image diffusion. However, these models generally focus limited set of tasks within narrow domains and are constrained by the scalability limitations of early foundation models (Rombach al., 2022b), limiting their potential practical visual assistants. To address these challenges, we propose a solution from both the model and data perspectives to develop more versatile visual assistant.
B MORE DETAILS FOR THE OMNI PEXEL-TO-PPEEL INSTRUCTION-TUNNNG DATASET
B.1 I
IMAGE RESTORATION
All the open-source datasets used for the image restoration task are listed below.
BSD (2001)
RealBlur (2020)
DPDD (2020)
GoPro (2017)
REDS (2019)
Denselaze (2019)
NII-HAZE (2020)
Reside-6K (2018)
UHDM (2022)
SIDD (2018)
Rain1 400 (2017)
Outdoor-Ran Rain (2019b)
SSID (2022)
RainDrop (2018)
RainDS (2021)
SRD (2017)
RealSnow (2023)
Snow 100K (2018)
CLWD (2021)
CelebA-Ho MlebA-HQUue (2015)
RealSR (2019)
LOO. ()
DIV2K (2017)
FFHQ (2019)
Flickr2K (2023a)
22

---

Published as a conference paper at ICLR 2025
B.2 IMAGE GROUNDING
(1) Segmentation Referring. We define referring segmentation as highlighting the target object specified by the user in the output image. For example, if the model is given instructions like. "Please mark the pixels in (color) based on the referring description: [caption)," the resulting image would display mask in the specified color over the appropriate object described in the caption. When constructing the data, we pre-set the mask to a specific color with 50% opacity and apply it directly to the original image. This method makes it easier for humans evaluate the accuracy of the predicted mask.
bounding boxes to highlight the target object specified by the user in the oulput image. Prompts for this task include instructions like, "Mark the specified area with a bounding box in [color]: [caption]." The model then frames the described object with bounding box in the specified color. Similar to referring segmentation, we pre-set the bounding box to a specific color and apply directly to the original image to produce the final output. During inference, we follow the post-processing methods outlined in InstruciCV (Sec. A.3) (Gan et al., 2024) to derive the coordinates of the specified region from the output image.
(2) Box Detection Referring. Instead of pixel-level grounding,
(3) Binary Mask Prediction. To promote the use of referring segmentation in real-world scenarios, we shift the objective from rendering images to directly predicting a binary mask. The prompt for this task might be: "Generate a binary mask for the described object: [caption]." The model is expected to produce a binary mask image where the object described in the caption is represented as a white region, with the background in black, excluding any original image content. Since the binary mask is a single-chamne image, we replicate the mask across three channels to convert it into RGB space during training.
B.3 CONTROLLABLE GENERATION
Canny Edge to Image. We use a Canny edge detector (Canny, 1986) (with random thresholds) and a Tiny and Efficient Edge Detector (TEED) (Soria et al., 2023) to obtain 1M canny-image-capee pairs from our collected natural images.
Holisticall-NNeseed Edge(HED) Boundary to Image. We use HED boundary detection (Xie & Tu, 2015) to obtain IM edge-image-caption pairs from our collected natural images (a part of images are source of the Canny Edge dataset.)
Depth Map to Image. Depth information is crucial for producing images with a sense of three- dimensionality. We used the Depth Anything V2 model (Yang et al., 2024b) obtain 1M depth-image- caption pairs, enabling accurate generation of depth maps across different visual scenarios.
User Sketch to Image. Following ControlNet (Zhang et al., 2023a), we generate human sketches from images by applying HED boundary detection (Xie & Tu, 2015) combined with strong data augmentations, including random thresholds, random masking of sketch portions, random morpho- logical transformations, and random non-man man--mmximmmmmmm suppression. This process results in 0.4 million sketch-image-captiie pairs.
Human Pose to Image. For human pose- based generation, employed the OpenPose model (Cao et al., 2017) for real-time
2D pose estimation. To ensure quality, only images where at least 30% of the key points of the whole body were detected were retained; those with fewer detected key points were discarded. We directly use visualized pose images with human skeletons as input condition. Finally, we obtain 0.25M pose-image-caption pairs.
Semantic Segmentation to Image. The semantic mask annotation is produced using OneFormer (Jain al., 2023), as adopted in 1ontrolNet-1.1 providing precise segmentation maps as conditions for image generation. This process results in 1M seg seg-image image-caption pairs.
Normal Map to Image. The surface normals are generated using DSINE (Bae & Davison, 2024). which contributed to the depiction of surface orientation and texture details in the generated images. Finally, we obtain 0.8M Normal-image-cape pairs.
'https:///oo
thub. http::/://ggitub..ooooo wawpm//ielllallllleeeamama / Net-vll 1-ni ghytly
23

---

Published as a conference paper at ICLR 2025
Line-art to Image. We use cartoon line drawing extraction method (Xiang et al., 2021) to generate line drawings from cartoons. This process yields 0.8M normal- maanlllamae-cattion pairs.
B.4 DENSE IMAGE PREDICTION
Depth Estimation. Monocular depth estimation is a dense prediction task to estimate the per-pixel depth value(distance relative to the camera) given an input RGB image. The datasets we use include Hypersim (Roberts al., 2021) and VKITTI 2 (Cabon et al., 2020), and our collected depth maps from controllable dataset. For different datasets, the ground-truth depth for each pixel may vary, being either absolute depth value a relative depth value. Here uniformly map the ground-truth value from real-valued range to the integer RGB space with range [o, 255), and let the three channels be the same ground truth. During inference, we directly average the outputs of the three channels, and then perform an inverse linear transformation specific to each dataset to obtain a depth estimate in the real-valued range.
Surface Normal Estimation, For surface normal estimation, we train the model to generate RGB images that encode pixel orientations differently. We convert the x/y/z orientations into r/g/b values to create a normal map visualization. The datasets we use include NYUv2 (Silberman et al., 2012) and our collected surface normals maps from controllable dataset.
Semantic Segmentation. Semantic segmentation is a dense prediction task to predict the per-pixe pixel semantic label given an input image. Given semantic segmentation task, we formulate different semantic categories using different colors in RGB space. Specifically, we define the background and ignore areas as black, i.e., pixels in color (0, 0, 0), and generate the colors for foreground categories using the pre-defined color-label dictionary. During inference, to decode the output image single channel map where each pixel represents a class label, we compute the L2 distance between each output pixel and the pre-defined colors for each semantic category, and take the label of the closest color as the predicted category. The datasets we use is ADE20K (Zhou et al., 2017b), which covers a broad range of 150 semantic categories.
B.5 INSTRUCTION PROMPT EXAMPLES.
Handling free-form user prompts, rather than relying on fixed task-specific prompts, significantly enhances Pix Wizard'ss s flexibility and usability general visual assistant. In Fig. 10, we showcase various examples where task speciife templates are dynamically rephrased and adapted by GPT-4o
C MORE DETAILS FOR THE ARCHITECTURE
C.I PRELIMINARIES
Diffusion Transformers. Diffusion models (Sohl-Dickstein et al., 2015; Ho al., 2020; Dhariwal & Nichol, 2021; Song et al., 2021) are a family of generative models demonstrating remarkable performance in modeling data distributions. These models trained to estimate clean data samples (or added noise) from their noisy version perturbed by a pre-defined Gaussian noise schedule. During sampling, they can generate data samples by iterative denoising starting from prior Gaussian distributions. Recent advancements in diffusion models, such as SoRA (Brooks et al., 2024), SD3 (Esser al., 2024), and PixArt (Chen et al., 2024b;a), exhibiting paradigm shift from the classic diffusion U-Net architecture to diffusion transformers (DiTs) (Peebles & Xie, 2023). DiTs appear be unified architecture with minimal modifications to original transformers (Vaswani et al., 2017). Therefore, these models demonstrate better scaling properties and can be naturally extended to more modalities, such as integrating text and image conditions through cross-attention.
Flow-based Models. Another line of generative models that extends the definition of diffusion models is ODE-based continuous normalizing flows (Chen et al., 2018), 1)), which are also known as flow matching (Lipman et al., 2023; Tong et al., 2024; Albergo & Vanden-Eijnden 2023) or rectified flows (Liu et al., 2023b). Specifically, given the data space Rd with samples x E Rdd flowbbaee models aim to learn time-dependenn velocity field v [0. 1] x Rd + Rd leading to flow 6: [0, 1] x Rd - Rd that can push noise 2n ~ Po (Ip) from source distribution to data 인 ~ p1 (x1) from target distribution. This velocity field and associated flow can be defined by ordinary
24

---

Published as a conference paper at ICLR 2025
Denoise
Dehaze
Derain
Remonve the nake and from this trage
Fix this pleture
sway riin
Makz this photo cleane gating of the nake.
Clear the laze reved the desail this pictuee
Get ran pateers show desals bettrr.
Redacee the roise gain this thiephttme
Retsove the mist foo picture.
Mikke this placto sleaes filerag the nandiops.
Deblur
Demolre
Demoire
Make this uncleu shuaper
Elimitate the Maint pattenss in this image.
Eliniiaae the Micré pullerss this image.
Fix dhe biluniness this inage.
Rodace the riplee like distottion this phaco
Redaes the cipple like distetioon in this phote.
Reatiyy the oagarnenc of this Narmod smagge
Smoooh out the lextured disontion it this sicture.
Smooff out the teanued distetioon in this pénure.
Deshadow
Desnow
Dewatermark
Bamone the shadkne make this pienne lovic hoteer
Erase the frost resiare visbiiiy this image.
Erese watemaal view of tiis
Get rn the cark srox for cleate
Remuove the asew get clearer view
Get of wenemenk fix the ciezal inverfrnnee
Remosiigg the diddow
Iepooe the image taking the
LighLight Enhancement
Flush Remove
Super-Res
Fix lighting
Bamove the flee reflectians.
Use super-ccaiatiinn to make the image sharper and more
Enhance the visiblity daniy lit photograph.
Clear the bright glace from caneera
Enbance the image morcaing brigitees and
Garrid of the inmovie light reflectioe fir cleacer look.
Inprove reselution image clearer and detalled
Restoration (Color)
Restoration
Restoration w/ cap
Rodoos the original solon in this image
Make this image clearce to show in features.
Echonce this
desciipii::: (capiio)).
Correct the ddddd colors revell the roal complexioor...
Fix the Narry image impeove vishiliy and doail
Rocover image detals using the caption: [eaption)
Improve the color belance for a nore socurite view
Restvee
Elinitate fuzzines poctuee
of ecosyten
(oggaa)).
Grounding (Box)
Grounding (Seg)
Grounding (Mask)
Derect, [color]: lcsption)
Locate the sibject, [coow];: loogtior]
Extect the bingy mukk image besed or the objoct
deseciptio:: (sogtios)
Drew a bourding box aroard the object, using (colect.
Idecnity and lighlight the aleas (oolor): [eagtion]
Create representing
Use (coloc) outlise the (saption)
(ceprizn)
Mark the piscle of (captize)) (enke) and love the rest unchanged.
described (eaptoo)
Coounding extract hinary mask: (cepion)
Inpainting
Outpainting
Text-to-Image
Completo inagy bacod deseription (raption)
Ourpoin the image with the descrppion: (ceprion)
Imsage aeteration: (caption)
Ingaining (saption)
Excand kerping syle theee
Produce on image sccondig the following descriptio:: (caption)
cxprion pride iepoircing (caption)
specified the description (ragtion)
Doawing: [caption]
Dense Image Pred.
Controllable Generation
Proces the image with Casny edge decction and show the olees.
Cuanty slae
inqured this [catios:
Generute as dkpul that shows for distaner of objecte the imaage.
Create inage based on the mep and test instuctions (caption]
map
image.
edge mep to image: (oagtion)
Generate human pose map flom the image, showing the postree and postiions
Geserate un image using human pase conzol: (caption)
Process the image show the categoraztion of scene's parts
Create a deneled image basal on the inpuf's control and tox: (caplion)
Tim the image into suatc that outiinee the scrne's mmmprncemi
Condition map image: (csptian)
imags map ontina major comporaess.
Using proviad bosiin mnate image (eaptimn)
Figure 10: Examples of GPT40 parapppaaee user prompts for different task.
differential equation (ODE):
Ze (xr) = ve (x)) 1) where do(x)) = x. Similar to the denoising network in diffusion models, the Flow Matching (FM) objective trains a time-dependent network (0(2,,t) t) to regress against the ground truth velocity field ue(x,, t). However, direct computation of this FM objective is intractable in practice, since there is closed-form solution of u, (z,, t). Instead, we can minimize the tractable Conditional Flow Matching (CFM) objective defined as:
LCTM (0) = Ex (e))) (ooo) (10 f) ue (z 1121)112;
where t ~ 24(0, 1), 21 ~ pa (z1), and x4 ~ pr ). It has been validated that the FM and CFM objectives share identical gradients with respect to 8, while CFM offers the flexibility to choose the design choices of u, (z,zzz) and p. (z.l21). ). A natural choice is to build the conditional probability paths straight paths between the source and target distributions (Liu et al., 2023b; Lipman et al.,
25

---

Published as a conference paper at ICLR 2025
2023), i.e. txy + (1 t) We can then use Equation 7 for training and solve the ODE from t == 0 to 1 to sample new data points.
C.2 TASK EMBEDDER
As shown in Fig. 1I, introducing the task embedder helps adaptively cluster similar task instructions in the latent space while separating those from different tasks, guiding the model's generation process in the correct direction.
Dersk-IIngge-rrudiccc
Image-Reztin AmssteResttsuttionn
Imoge-Grounding
Image Editiin
In/Outpanting
Condition Image
to-Image
1-SNE of embedddings before Task- Embedder
tNEE embedkdings after Lakk-mmeedderr
Figure 11: t-SNE visualization of the global text embeddings. Each dot represents a user instruction.
C.3 CLASSIFIER-FREE GUIDANCE FOR MCLTI-MODAL CONDITIONS
Classifier-free guidance (Ho & Salimans, 2022) is a technique that balances quality and diversity images generated by diffusion model. For our PixWizard the network eø cg c,, has two conditionings: the input image and text instruction 2023) and InstructCV (Gan et al., 2024), we employ tailored noise predictor that assigns different weights, wy and WT, to different conditionings, which can be adjusted to trade off how strongly the generated samples correspond with the input image and how strongly they correspond with the edit instruction. During training, we randomly set сд Ø er for 5% of examples, and both conditions are for 5% of examples. The process is as follows:
Similar to InstructPix2Pix (Brooks et al.,
Eg(2, Cr, CT) = ea(z,, 2)
+ (eo cr, 0) e0(21, 0))
(8)
+ur (co C1, oT) e0(2,, Ø))
In Fig. 12, illustrate the effects of the two parameters the generated samples. As shown, changes in both W, and Wr significantly influence the results. Specifically, in PixWizard, W1 strongly affects the color distribution of the generated images. An increase in W, leads more saturated and unrealistic colors. This can also negatively impact tasks such as depth estimation, where subtle depth variations may be represented as solid black regions. In contrast, the effects of Wr vary depending on the task. For example, in dense image prediction tasks, increasing Wy can degrade the quality of the estimation. However, for generative tasks, a higher Wyr typically has little effect.
UI
75
W: 16.5
parsuns handing and warting diferral oole Biooss
sup
Figure 12: Impact of classifier-free guidance.
26

---

Published as a conference paper at ICLR 2025
C.4 ANY RESOLUTION
Pix Wizard inherits the dynamic partitioning and padding scheme introduced by (Zhuo et al., 2024), allowing the model handle images of any resolution and aspect ratio during both fine-tuning and inference.
The dynamic partitioning and padding scheme is introduced to optimize image token handling, avoiding fixed resolutions. Specifically, given constraints on the maximum number of patches and the maximum aspect ratio, the method defines set of candidate patch partitions. For each input image size, it determines the optimal patch partition based on matching ratio between the input size and the candidate partition, selecting the best fit. The input image then resized accordingly using the chosen partition. This dynamic process leads to varying sequence lengths of patch tokens, so padding is applied to align the lengths across batches using a pad token. To prevent unwanted interactions between pad and regular tokens, attention masks are also introduced. This approach maximizes efficiency while preserving image integrity during embedding.
However, in practice, the required resolutions for different tasks can vary significantly. To support more flexible handling of arbitrary resolutions while preserving the original resolution as much as possible, we use [512², 768?. 1024?] as resolution centers generate candidate patch partitions. During training, we group data with similar resolutions into buckets, ensuring that sequence lengths within each batch comparable, minimizing padding tokens, and improving training efficiency. During inference, by incorporating NTK-Aware Scaled RoPE (Peng & Quesnelle, 2023) and sandwich normalization, Pizard demonstrates exceptional resolution extrapolation.
D MORE EXPERIMENTAL RESULTS
D.1 IMPLEMENTATION DETAILS
Dense Image Prediction. For semantic segmentation, we assign labels by identifying the nearest neighbor RGB color value, and accuracy is evaluated using the Mean Intersection over Union (mloU) metric. For monocular depth estimation, we average the output image across the three channels and apply the inverse of the linear transformation used during training to obtain depth estimates within the range of [0, 10) meters. Accuracy evaluated using the Root Mean Square Error (RMSE). For surface normal estimation,
recover the corresponding normal vectors from the output image and use the Mean Angle Error to assess accuracy.
Controllable Generation. We mainly evaluated Pix Wizard's ability based on two conditions; canny edge maps and depth maps. Following ControlNet++ (Li et al., 2024), we measured controllability using RMSE for depth maps and F1-Score for canny edges, comparing input conditions with features from the generated images. Image quality and text alignment were assessed using FID (Fréchet Inception Distance) and CLIP-Score. All experiments were conducted at resolution of 512 x 512.
Image Inpainting, We used latent diffusion (Rombach et al., 2022b) to measure FID and LPIPS, focusing on samples where 40-50% of the image area was inpainted. For outpainting, we followed MaskGIT (Chang et al., 2022a) settings, extending the image by 50% and evaluating performance with FID and Inception Score (IS) on 512 x 512 crops from the Places dataset (2017a).
Text-fo-image Generation. We evaluated Wizard with two methods: visual examples and automatic metrics, including the Human Preference Score (HPS) v2 (Wu al., 2023) and Zero-shot FID-30K on the MS-COCO dataset (Lin et al., 2014).
D.2 IMAGE RESTORATION
Following previous works (Conde et al., 2024; Potlapalli et al., 2024; Ai et al., 2024), besides the results from the deraining and denoising benchmarks, we selected six additional image restoration tasks to further evaluate the robustness of PixWizard: Snow (Liu et al., 2018) for desnowing, Reside (outdoor) SOTS (Li et al., 2018) for dehazing, LOLv2 (Yang et al., 2021) for low-light en- hancement, and GoPro (Nah et al., 2017) for deblurring. Additionally, to assess zero-shot capabilities on tasks not encountered during training, (UDC) image restoration and UIEB (Li et al., 2019a) for underwater (UW) image restoration. We evaluate performance using PSNR and SSIM distortion metrics.
use TOLED (Zhou et al., 2021) for under-display camera
27

---

Published as a conference paper at ICLR 2025
Table Quantitative results on restoration tasks with existing image restoration methods.
Desnowing Snow100K4
Dehazing SOTS
Low-light Enh.
Deblurring GoPro
[Zero-shot
[Zeroshhot]UW)IR
Methods
LOLv2
TOLED
UIEB
PSNR1 SSIM
PSNR1
SSIM PSNR SSIM
PSNR
SSIM] PSNR
SSIM
PSNR
SSIM
SwinlR (2021)
21.50
0.891
24.52
0.773
Restormer (2022)
0,914
24.09
0.927
20.77
0.851
27.22
0.829
27.74
0.841
17.34
0.770
NAFNet (2022a)
31.42
0.920
25.23
0.939
18.04
0.827
26.53
0.808
27.90
0.848
17.31
0.736
AirNet (2022)
30.14
0.907
21.04
0.884
19.69
0.821
24.35
0.781
26.76
0.799
17.09
0.761
PromptIR (2024)
30.91
0.974
21.23
0.860
27.02
0.798
DA-CLIP (2023)
28.31
0.862
30.16
0.936
21.76
0,762
24.65
0.703
Instruct|R (2024)
26.90
0.952
29.70
0.892
PixWizard 29.66
0.883
28.14
0.937
20.29
0.807
24.68
0.774
27.22
0.826
16.99
0.752
Results. Table 5 presents additional performance comparison with state-of-the (SOTA) task- specific and all-in-one restoration methods, As shown in the results, even though image restoration data constitutes only a small portion of the overall training data, our method still demonstrates competitive performance, even outperforming some task-specific methods. For example, it out- performs DA-CLIP in the desnowing task and exceeds NAFNet, Restormer, and AirNet in the dehazing task. Furthermore, when evaluated on tasks not included in the training phase, our method achieved performance comparable to that of specialized models, highlighting the generalizability of our approach.
D.3
IMAGE GROUNDING
We evaluate referring segmentation tasks on the gRefCOCO (gRef) (Liu et al., 2023a), RefCOCO, and RefCOCO+ validation and test sets. To assess the performance gap between our approach and specialized models, we report results from several expert methods and primarily compare our approach with two unified models: Unified-10 (Lu et al., 2022b) and InstructDiffusion (Geng et al., 2024). Unified-IO directly produces the corresponding binary mask, while InstructDifffsion requires post-pracessing network to extract masks from the output image. We use comparison methods: (i) We convert the original image to the HSV color space to enhance the mask's ' hue, then apply threshold for extraction. These results are reported w/ HSV. (ii) We directly generate the corresponding binary mask for comparison. Following standard practices (Liu et al., 2023a), we use cumulative IoU (cloU) to measure performance.
Table 6: Quantitative results referring segmentation in terms of cloU.
gRefCOCO
RefCOCO val
RefCOCO+
Method
val
test A
test B
val
test A
test B
CRIS 2022c
55.34
70.47
73.18
66.10
62.27
68.08
53.68
LAVT (2022)
57.64
72.73
75.82
68.79
56,86
62.29
48.14
ReLA (2023a)
62.42
73.21
75.24
68.72
56.10
62.26
47.89
Unified-IO (2022a)
17.31
46.42
46.06
48.05
40.50
42.17
40.15
InstructDiffusion w/ HSV (2024)
33.19
41.64
40.81
41.98
33.20
37.85
26.92
Pixai HSV
33.65
47.28
44.12
45.38
40.07
38.89
40.76
PixWizard
29.07
44.38
40.13
42.09
36.49
35.30
38.93
Results. Table 6 reports the results for referring segmentation. Our model demonstrates strong performance under two different mask extraction methods, outperforming InstructDiffusion across almost all evaluation datasets. In the various test sets of RefCOCO and RefCOCO+, our performance is comparable to Unified-IO, but on gReíCOCO, we significantly outperform Unified-1O (33.65 vs. 17.31). However, it is important to note that these unified methods still lag considerably behind those specifically designed for referring segmentation. Additionally, we observed that using generative model for grounding tasks presents significant challenge due to the model's limited perception and localization capabilities. This often results in failures to correctly locate objects when multiple targets present, highlighting the need for improvements in instruction and understanding in future work.
28

---

Published as a conference paper at ICLR 2025
D.4 MORE PRECISE EDITING
Based on the robust inpainting and grounding capability of PixWizard, we find that it allows user for more precise image editing tasks: (i) Remove Anything. It tackles the object removal problem (Crim- inisi et al., 2003; 2004; Elharrouss et al., 2020), enabling users to seamlessly remove specific objects. As shown in Fig. 13. PixWizard first generates target mask based on user instructions, then fills the area with appropriate background details. (ii) Replace Anything. It lets users swap objects in an image, replacing them with specified objects while maintaining background consistency. (iii) Add Anything. This allows users to insert objects into an image. By adding a mask and providing a text prompt, PixWizard generates the desired content using its advanced inpainting ability.
Remove Anything
Replace Anything
Add Anything
Extract mask of Corgi in the middlee
Construct binary mask image based object: rod paper
Adking masks by user
Impaining: A pathway along edge al'a body of water.
Inpaint the imagge: On the Jeft side of the imape, there is small duck
Complete the image: simple Line drawing of a dog playing on the grass.
Figure 13: Visualization results of Remove, Replace and Add Anything.
D.5 LIMITATIONS OF IMAGE EDITING TASKS
Image editing tasks can generally be divided into rigid and non-rigid edits. Rigid editing typically involves simpler modifications such as adding, removing, replacing, style transferring, or altering the texture of an object while maintaining the overall spatial structure of the image. On the other hand, non-rigid editing involves more complex transformations that require spatial manipulation, such as altering the pose of object, swap the position or changing the viewpoint.
Despite the versatility of Wizard in handling multiple types of rigid editing, we found that Pix ard currently exhibits limitations when dealing with non-rigid editing tasks, as several visualization examples shown in Figure. 14. When users attempt to provide more complex editing instructions, PixWiz often fails execute them successfully, instead keeping the original image largely un- changed. This limitation arises primarily due to two reasons: the design of the structural-aware guidance module and the lack of diverse training data for non-rigid transformations. PixWizard's structural-awes aware guidance is implemented by concatenating the VAE latent representation with ran- dom noise along the channel dimension. We found that this method would limit generation directions to follow the structure of the input image, constraining its ability to perform non-rigid transformations that demand substantial structural alterations. However, is worth noting that the "concat" method is efficient for other visual tasks such image estimation, low-level tasks, and image grounding, which require maintaining similar structural characteristics. This is one of the reasons we initially adopted it. Furthermore, the datasets used to train PixWizard-ss such UltraEdit (2024), MagicBrush (2024a), HQ-Edit (2024), Instruct P2P (2023), and SEED-X- (2024)- focus predominantly on rigid editing tasks and lack non-rigid editing data pairs.
29

---

Published as a conference paper at ICLR 2025
Input
Output
Make boy close eyes and smile.
Imput
Oulput
Move the iPhone in the image to the right side of the image.
Please help me close the book.
Figure 14: Bad cases of non-rigid editing.
How to perform non-rigid editing in PixWizard is indeed one of the directions we aim to address. Here are some of our current thoughts: (1) Constructing non-rigid editing data pairs is important. This could involve utilizing state-of-the-arr editing methods like Imagic (Kawar et al., 2023) and Forgedit (Zhang et al., 2023c), or advanced inpainting methods, complemented by GPT-assist and manual filtering to create high-quality non-rigid editing pairs. (2) Refining the structural-awan guidance. The primary objective is move away from directly concatenating the VAE latent in the channel dimension. In future work, we aim to explore the integration of flow-based image inversion techniques and shift towards fusing targeted image features within the self-attention layers of the DiT block. Further experiments will conducted to validate the effectiveness this approach.
It is also worth mentioning that we found that the quality of most existing open-source image editing datasets subpar. There are two main issues: first, the target images often contain artifacts, making them look unrealistic; second, there is lack of alignment between the target images and the editing instructions. Through experiments, we discovered that simply using more open-source data does not necessarily lead better performance. On the contrary, using small amount of high-quality editing data results in better performance in both evaluation metrics and user experience with Pix Wizard. Therefore, is both crucial and meaningful explore methods for effectively cleaning open- editing datasets. We are committed to addressing this issue in the future.
D.6 GENERALIZATION ABILITY TO UNSEEN TASKS AND PROMPTS
Zero-shot Examples. We have provided several zero-shot visual examples in Fig. 15. These include marking target objects using different symbols other than a box, such as a star, a point, or circle. Additionally, we can change the color of detection boxes to colors that were training. Furthcerermore, we can simultancously perform multiple tasks, such as joint deraining and watermark removal, deraining and super-resolution, or inpainting and outpainting, with masks of any shape and mixed Black-and-whit colors. These examples highlight the generalizability of our approach. However, getting Pix Wizard perform entirely new tasks that are completely unrelated to the training data, such as adding object and then detecting it or contour detection, is challenging. This limitation points to the boundaries of Pix Wizard's generalization capabilities.
seen during
Few-shot Examples. Following Emu Edit's experimental setup (Sheynin et al., 2024), we further validated Pix Wizard's generalizability from a few-shot learning perspective, We prepared an object contour detection task an illustration. We constructed 50 training samples for this task (all with red contours) and added them to our training dataset, then fine-tuned Pix Wizard. As shown in Fig. 15, Pix Wizard quickly acquired this capability. Additionally, changed the contour color and found that Pix Wizard could still follow the instructions, implying that the model can effectively generalize novel tasks.
30

---

Published as a conference paper at ICLR 2025
Zero-Shot: Using different marks
Add the hal the
eyes
specified oycle:
Zero-Shot: Using different colors
Detecion, purple bust the Jack o Lanteem
Detection, White bos: the Jack Lantem
Take away the rain and use super resolutiso make the imagee sharper.
Zero-Shot: Simultaneous Multi-tasking
BROES
Filtering mandrops, then the watermak Few-shot learning for contour detection
Inpainting and ocspaiming based the description: (image caption)
Highlight the largel arca with a vontour: the black dog
Grounding with red contour: the
Mark the area with blue contour: the dog
Figure 15: Visualization results of zero-shot and few-shot.
D.7 MULTI-HOT GUMBEL-SOFTMAN
In the implementation of the Multi-Hot Cumbel--oftmaa (MHGS), the pseudocode is defined as follows:
31

---

Published as a conference paper at ICLR 2025
def MHGS (legits, temp=1, dim=-1, sample tokens=16):
Add Gumbel noise and scale by temperature gumbels = (logits GumbelNoise (shape-logits.ehappe temp
Apply Softmax to obtain soft outputs y_soft Softmax (gumbels, dimension-dim)
Seiect top-k values for discrete output
inaices = Top sott, k=sample_tokens, dimension=dim)
Create hard multi-hon tensor from indices y_hard MultiHotiennor (indices, shape-logits.shenn
Combine hard and soft outputs while preserving gradients ret y hard StopGradient (y_soft) y_soft
return ret
32


Reviews:
[{"review_id": "0Shh3VP1Ov", "reviewer": "Reviewer_Vhoc", "summary": "This paper introduces PixWizard, a visual assistant that performs diverse image generation, manipulation, and translation tasks based on free-form language instructions. PixWizard unifies various vision tasks into a single image-text-to-image framework using a newly created Omni Pixel-to-Pixel Instruction-Tuning Dataset. Built on the DiT architecture, PixWizard supports dynamic resolutions, integrates structure- and semantic-aware guidance, and demonstrates competitive performance and generalization across unseen tasks and instructions, making it a powerful interactive tool for visual tasks.", "questions": "NA", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The paper effectively explores the unification of multiple tasks within a single framework.\n- The proposed model demonstrates strong performance across a wide range of tasks.\n- The method is adaptable to images of varying resolutions.", "weaknesses": "The reviewer does not have significant concerns but notes a few minor weaknesses:\n- Some implementation details are unclear; For example, in the statement, “In the first stage, we initialize the model by combining the weights of a pretrained text-to-image model with randomly initialized weights for the newly added modules,” it is unclear which model is used for initialization.\n- The paper reads more like an engineering report than a research paper with deep insights. However, it would be valuable if the model, code, and/or dataset were made publicly available.\n- A concluding analysis on how different tasks influence each other within the unified framework would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B78UnZovbJ", "reviewer": "Reviewer_g5qv", "summary": "PixWizard presents a unified approach to  conduct various kind of image generation, manipulation, and translation tasks with one DiT model.  The contributions are:\n\n1. The authors constructed an image-instruction-image triplet instruct tuning dataset to train this model.  \n\n2. The authors proposed Structural-Aware Guidance to fix the structure of reference image and Semantic-Aware Guidance to learn the instruction-based image to image capability.  Inspired by DynamicVit, the authors designed a task-aware dynamic sampler to squeeze the image tokens for different image to image tasks and utilized Multi-Hot Gumbel-Softmax to extract topK tokens in order to decrease computational cost in DiT.\n\n3. PixWizard inherits the dynamic ability of lumina-next model to handle images of any resolution and aspect ratio. In addition, a two-stage training strategy is designed to improve the performance of the model.\n\n4. Extensive experiments on various dataset and benchmarks concretely verified the generalization of PixWizard", "questions": "1. The current state-of-the-art general image editing methods, Imagic, followed by its speedup and less overfitting open-sourced version Forgedit, is capable of conducting general image editing tasks including the non-rigid editing. Is pixwizard capable of conducting non-rigid editing like Imagic and current state-of-the-art  Forgedit? If yes, I would like to see some examples compared with Imagic and  Forgedit with examples from TEdBench in the revised version. If not, I would like to see some discussions on this issue and how to solve it in the revised version.  Imagic and Forgedit require test-time fine-tuning, which costs at least 30 seconds. Is it possible for pixwizard to somehow distill and integrate these methods to tackle the non-rigid editing problem? Considering the time limit of rebuttal period, I won't require the authors to fix this issue in such a short time. Yet  discussions and possible solutions to this problem in the revised paper are compulsory . \n\n\n2. shown in table 6, the task-aware dynamic sampler do not bring significant performance gain. What are the computational savings then? They are not reported in the paper.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.  The task studied by this paper is the main-stream trend nowadays: unifying multiple image generation tasks in one DiT model.\n\n2.  Solid quantitative experiments demonstrate the generalization capability of the proposed method.\n\n3.  The authors designed a DiT version of DynamicViT. The Multi-Hot Gumbel-Softmax is also interesting. The proposed task-aware dynamic sampler, which sparsifies the image token to reduce computational cost and utilizes one of the two text encoders for task specific embedding, is a brilliant design. \n\n4. Many properties of pixwizard inherit from previous work lumina-next, including the DiT pretrained weigth, dynamic resolutions, which makes pixwizard a meaningful extension of the lumina-next-t2i DiT model in terms of omni image generation.", "weaknesses": "1. One of my main concerns is that pixwizard seems to be incapable of conducting non-rigid editing, which is a very common and important, yet challenging topic in image editing, for example, let the standing dog sit, let the polar bear raise its hand, close the open book, etc.  In figure 1, I cannot find  such a common task being presented. I doubt from two perspectives whether pixwizard is capable of conducting such non-rigid editings:  \n\nfirst, the authors designed the structural aware guidance, which concats the vae latent with random noise in  channel dimension. This is identical with instructpix2pix, which claims that \"our model is not capable of performing viewpoint changes, can make undesired excessive changes to the image, can sometimes fail to isolate the specified object, and has difficulty reorganizing or swapping objects with each other.\".  From my experiments, instructpix2pix completely fails in non-rigid editing. Thus I doubt whether pixwizard inherits such a incapability of instructpix2pix  due to the same concat operation.\n\nsecond, the datasets utilized for training image editing task are strongly biased and do not contain non-rigid editing data. In section 2, the authors listed the training datasets: UltraEdit (2024), MagicBrush (2024a), GQA-Inpaint (2023), Instruct P2P (2023), SEED-X-Edit (2024), GIER (2020), and HQ-Edit (2024). However, almost none of these datasets contain non-rigid editing data. Instead, these datasets focus on very simple editing tasks, which could simply be solved by inpainting models for adding, deleting and replacing objects or backgrounds,  or controlnet+text to image to change textures and style transfers (of course, pixwizard integrates all these capabilities in one model and demonstrates comparable or better quantitative results than separate models, which is still a contribution. I just want to convey my personal opinion that these tasks are generally considered simple image editing tasks).   These tasks are rather simple in general without changing spatial and structural features. Without training data on non-rigid editing, it is almost impossible for pixwizard to conduct such edits. \n\n2.  my second concern is that although the quantitative experiments are solid, the quantitative results in table 1 show that for many image to image tasks, there is still a significant gap between  pixwizard and task-specific model. In table 2 the image editing results are close, yet the testsets are very biased towards simple editing tasks. In table 3 the results on controlnet and t2i tasks are close though.\n\n\n3. shown in table 6,  the gain of task aware dynamic sampler and two stage training strategy  is minimal, which weakens the effectiveness of the propose method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LzSDP12vEd", "reviewer": "Reviewer_8hs4", "summary": "The paper proposes a generalist vision model that unifies various image generation and recognition tasks into an image-instruction-to-image problem. To address this problem, they collected a dataset, Omni Pixel-to-Pixel Instruction-tuning Dataset, consisting of 30 million image-instruction-image triplets. Fueled by this dataset, they train a conditional DiT model that takes the conditional image and instruction as the input. The paper shows extensive comparisons against specialized and generalist baselines on various benchmarks.", "questions": "1. \"To balance tasks, we assign manual sampling weights to each dataset, randomly selecting data when a weight is less than 1.0.\" - could the author elaborate on how this is done? Like, what does it mean by the weight is less than 1, when the weights are manually assigned?\n\n2. As one of the major contributions of the paper is the public data, will the processed dataset be released in any form? \n\n3. Which layers of the CLIP image features are used as the conditions, is that the 1D feature after pooling or 2D feature before pooling?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Collecting and preprocessing the Omni Pixel-to-Pixel Instruction-tuning Dataset is a huge effort, which is the largest of its kind to my best knowledge and can be a great contribution to the community.\n\n2. The paper is clearly written with rich details.\n\n3. Extensive evaluation is done to demonstrate the performance of the model on various visual tasks, from image editing to image grounding.\n\n4. Ablation studies are performed to verify the proposed methods, helping to understand the significance of each individual components.", "weaknesses": "1. While the authors claimed \"competitive performance compared to task-specific models and surpass general visual models,\" this is not directly reflected by the results: there is a clear gap in tasks like depth estimation, semantic segmentation, and image grounding with task-specific models. The general visual model also outperforms some of these tasks. The authors should state the contribution in a more clear way.\n\n2.  The other contribution claimed in the abstract that was not well-demonstrated in the paper is that \"the model exhibits promising generalization capabilities with unseen tasks.\" How much can the model generalize beyond instruction that it does not see during the training? For example, can the model follow the instruction \"Mark the specified area with a star in red: {caption}\"?\n\n3. The baselines chosen for image generation were not state-of-the-art anymore, e.g., UniControl is not compared, although their dataset (MultiGen-20M) was used.\n\n(Minor) Although some ablation studies are provided to justify the proposed methods, some other choices that seem novel are not well-proved, such as using Gemma-2 instead of T5 as the text encoder and RoPE as the positional embedding.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ED8UhHYWsc", "reviewer": "Reviewer_kiLh", "summary": "The authors propose PixWizard, a flow-based model that can tackle a range of visual tasks including text-to-image generation, inpainting and outpainting, as well as a long list of image-to-image translation tasks (sketch-to-image, depth-to-image, denoising, box-drawing based detection, segmentation and more).\n\nThe core of their approach revolves around teaching a pre-trained text-to-image model (Lumina-Next) to tackle multiple novel tasks by conditioning it on additional image input features and features describing the specific tasks the network should handle. The authors propose a set of blocks to extract said features and integrate them into the network, and propose a data curation and balancing strategy to ensure the model does not ignore rare tasks.\n\nFinally, the authors conduct a wide range of experiments on many different tasks and demonstrate that their approach is competitive (and sometimes even improves over) networks trained for each individual task, and also outperforms most generalist network approaches which aim to concurrently tackle many tasks.", "questions": "I tried to outline my concerns in the weakness section above. To sum them up in two main questions:\n\n1) What does this paper bring to the table that doesn’t already exist in prior published work like Emu Edit?\n\n2) What is the extent of generalization that your model actually supports? Can you back this up with experiments?\n\nI appreciate the effort the authors clearly put into the work, so if these are answered in a satisfactory manner, I’ll gladly increase my score.\n\nMinor clarification questions (skip these if you don’t have time):\n\n3) What are the details of the MHGS? How is it implemented?\n\n4) L100 says “We extend the dynamic partitioning and padding scheme to handle input images of any resolution, aligning closely with human perception.” but L279 says these resolution capabilities are inherited from [Zhuo et al., 2024]. Do you mean that Zhou support multiple resolutions but not good enough, and you improve this? Are you extending the scale of resolutions that they support? \n\n5) L298: “we assign manual sampling weights to each dataset” How? Can you describe your criteria?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The paper does a remarkable job in evaluating against a long range of baselines and across a wide list of tasks. It also demonstrates good performance across most of the tasks, showing that their approach can indeed lead to a fairly general multi-task model.\n\nThe authors also provide a fairly detailed overview of their network, block choices, and the sources and handling of their training data. In the appendix, they further analyze many of their contributions in an ablation study, and report on choices even when they do not meaningfully impact the results (which is good in my eyes, as it lets future work know which components they may want to discard in favor of simplicity).\n\nFinally, the appendix also contains an interesting application for the all-in-one approach, where they use the model to segment parts of the image and then edit them. Such interactive multi-step approaches that use different network skills to improve results can serve as a good reason for exploring multi-task models. Please see more on this in the weakness section below.", "weaknesses": "My chief concerns with the paper are as follows, ranked from most-important to least important:\n\n1) The paper does not appear to distinguish itself from prior, published multi-task approaches like Emu Edit [Sheynin et al, CVPR 2024], and it does not actually outperform them. More specifically, Emu Edit already introduced a multi-task approach based on a pre-trained text-to-image model. It already has learned task embeddings, and it shows a range of similar downstream tasks. \nI appreciate that outperforming closed models trained on unknown datasets is difficult, but when I ask myself what I learned from reading this paper that I did not already know from Emu Edit, I have a hard time coming up with an answer, and this is an issue.\n\n2) The paper claims to generalize to unseen tasks (e.g. L26). I may have missed the results that corroborate this claim, but the closest I could find are the zero-shot restoration experiments in the appendix. However, these are close enough to existing restoration tasks that even non-multi-task baselines can generalize well to them and outperform this model? I would have liked to see generalization to actual new tasks, even using inverted task embeddings like Emu Edit. Otherwise, this weakens the contribution.\n\n3) Related to (1) – most of the paper focuses on comparisons to other methods, to the point where ablation is relegated to the end of the appendix. I think ablation is crucial here and should be moved into the core paper, even at the cost of pushing out some experiments. since it is the only section that gives us validation for the specific new ideas introduced by the paper. This method uses a different base-model, is trained on different data, and has a significantly different number of parameters from the baselines. I understand that you beat them, but I also want to understand why. \n\n4) There is relatively weak motivation for why multi-task models are needed. Saving some memory compared to specialized approaches is okay, but its not a significant contribution. Again, this can be contrasted with Emu Edit which show that training on multiple tasks can improve each of them individually, and that training on discriminative tasks like segmentation can also improve performance on generative tasks like editing. I had a hard time finding similar motivations here.\n\n5) The paper can be better self-contained. Some components (e.g., dynamic partitioning and padding scheme) refer back to recent preprints with no citations, and if these parts are important then they should probably be given a bit more detail (even in the preliminaries in the appendix).\n\n\nMinor issues that did not affect my score:\n\n1) L121: without extra any -> without any extra\n\n2) “Next, we concatenate the image latent with the noise latent along the channel dimension” – Consider citing SR3 [Saharia et al, TPAMI 2022] or Palette [Saharia et al, SIGGRAPH 2022].\n\n3) Table 3 – Bold numbers are missing from the baselines when they win.\n\n4) L483 notes that the method outperforms the inpainting baselines based on FID and LPIPS scores, but the method seems to underperform the baselines on LPIPS?\n\n5) Appendix C.4. – It seems like the gains highlighted here are not related to any change in the proposed paper, but are just due to selecting a flow-matching baseline over a diffusion model? Did I miss something?\n\n6) L1544: less tokens requiring less inference time should be shown experimentally because it’s not trivial that the gain here is meaningful.  The method uses extra layers to actually select what tokens to drop, and the tokens enter through cross-attention (as opposed to self-attention) which means they scale compute linearly and not quadratically?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)