ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer
Authors: Zhen Han
Zeyinzi Jiang
Yulin Pan
Jingfeng Zhang
Chaojie Mao
Chen-Wei Xie
Yu Liu
Jingren Zhou
Venue: ICLR 2025 Poster (2025)
Keywords: Image Generation and Editing
Diffusion Transformer
Instruction Following
Unified Framework
Primary area: generative models
TL;DR: ACE is an All-round Creator and Editor

Abstract: Diffusion models have emerged as a powerful generative technology and have been found to be applicable in various scenarios. Most existing foundational diffusion models are primarily designed for text-guided visual generation and do not support multi-modal conditions, which are essential for many visual editing tasks. This limitation prevents these foundational diffusion models from serving as a unified model in the field of visual generation, like GPT-4 in the natural language processing field. In this work, we propose ACE, an All-round Creator and Editor, which achieves comparable performance compared to those expert models in a wide range of visual generation tasks. To achieve this goal, we first introduce a unified condition format termed Long-context Condition Unit (LCU), and propose a novel Transformer-based diffusion model that uses LCU as input, aiming for joint training across various generation and editing tasks. Furthermore, we propose an efficient data collection approach to address the issue of the absence of available training data. It involves acquiring pairwise images with synthesis-based or clustering-based pipelines and supplying these pairs with accurate textual instructions by leveraging a fine-tuned multi-modal large language model. To comprehensively evaluate the performance of our model, we establish a benchmark of manually annotated pairs data across a variety of visual generation tasks. The extensive experimental results demonstrate the superiority of our model in visual generation fields. Thanks to the all-in-one capabilities of our model, we can easily build a multi-modal chat system that responds to any interactive request for image creation using a single model to serve as the backend, avoiding the cumbersome pipeline typically employed in visual agents.

Published as a conference paper at ICLR 2025
ACE: ALL-ROUND CREATOR AND EDITOR FOLLOW- ING INSTRUCTIONS VIA DIFFUSION TRANSFORMER
Zhen Han'& Zeyinzi Jiang" & Yulin Pan" & Jingfeng Zhang" & Chaojie Mao Tongyi Lab, Alibaba Group
Chenwei Xie & Yu Liu & Jingren Zhou Tongyi Lab, Alibaba Group
ABSTRACT
Diffusion models have emerged as powerful generative technology and have been found to be applicable in various scenarios. Most existing foundational dif- fusion models are primarily designed for text-guided visual generation and do not support multi-modal conditions, which are essential for many visual editing tasks. This limitation prevents these foundational diffusion models from serving as a unified model in the field of visual generation, like GPT-4 in the natural language processing field. In this work, we propose ACE, an All-round Creator and Editor, which achieves comparable performance compared to those expert models in a wide range of visual generation tasks. To achieve this goal, we first introduce unified condition format termed Long-contex, Condition Unit (LCU), and pro- pose a novel Transformer-based diffusion model that uses LCU as input, aiming for joint training across various generation and editing tasks. Furthermore, we propose an efficient data collection approach to address the issue of the absence available training data. It involves acquiring pairwise images with synthesis- based or clustering-basee based pipelines and supplying these pairs with accurate textual in- structions by leveraging fine-tuned multi-modal large language model. To com- prehensively evaluate the performance of our model, we establish a benchmark of manually annotated pairs data across a variety of visual generation tasks. The extensive experimental results demonstrate the superiority of our model in visual generation fields. Thanks the all-in-one capabilities of our model, we can easily build a multi-modal chat system that responds to any interactive request for image creation using a single model to serve as the backend, avoiding the cumbersome pipeline typically employed in visual agents.
INTRODUCTION
In recent years, foundational generative models have made groundbreaking progress in natural lan- guage processing (NLP) (Anil et al. 2023 Anthropic 2023a by Ouyang (022). Conversational language models like ChatGPT (Brown al. 20200] OpenAl 2023b) offer unified framework for addressing various NLP tasks through a prompt-guided approach. By employing a unified input- output structure, these models can achieve dynamic multi turn interactions with users. Furthermore, by harnessing the knowledge of historical dialogues (Anthropic 2024 OpenAl 2024), they possess the capacity to comprehend intricate queries with greater nuance and depth. However, such unified architecture has not been fully explored in visual generation field. Existing foundational models of visual generation typically create images or videos from pure text, which is not compatible with most visual generation tasks, such as controllable image generation (Zhang et al. 2023b Jiang al. 2024) or image editing (Brooks et al. 2023). Thereby, specific visual generation tasks still require tailored tuning based on these foundational models, which is inflexible and inefficient. For this reason, the visual generative model has not yet become a powerful and unified productivity tool in
"Equl Contribution. Order is determined by random dice rolling.
Project leader and corresponding author.

---

Published as a conference paper at ICLR 2025
Control
Canny
4un
General
General
en
Scene
Text
Tayeer
Sertbble
Workflon flow
<00 Depth
ACE
Outpaini
Multi- Multi-Contei Control 4n Mosaie
Faoe
General
Object+Face
Repaint
Scene
Style
Figure 1: Multi-turn image editing results of ACE. ACE supports a wide range of image gener- ation and editing tasks through natural language instructions, allowing complex and precise editing requests to be easily accomplished through multi-turn interactions.
various application scenarios like large language models (LLMMs) (Abdin al., 2024 Dubey al,, 2024
Bai et al., 2023: Yang al.) 2024).
One major challenge building an all-in-one visual generation model lies in the diversity of multi- modal input formats and the variety of supported generation tasks. To address this, we design unified framework using a Diffusion Transformer generation model that accommodates a wide range of inputs and tasks, empowering it serve ACE. First, we analyze the condition inputs most visual generation tasks, and define Condition Unit (CU), which establishes unified input paradigm consisting of core elements such image, mask, and textual instruction. Second, for those CUs containing multiple images, we introduce Image Indicator Embedding ensure the order of the images mentioned in instruction matches image sequence within the CUs. Besides, we imply 3d position embedding instead of 2d spatial-level position embedding the image sequence, allowing for better exploring the relationships among conditional images. Third, we concatenate the current CU with historical information from previous generation rounds construct the Long-context Condition Unit (LCU). By leveraging this chain of generation information, we expect the model to better understand the user's ' request and create the desired image. As depicted in Fig. D ACE supports a range of generating and editing capabilities, allowing accomplish complex and precise generation tasks through multi-turn instructions.
All-round Creator and Editor, which we refer to as
To address the issue the absence of available training data for various visual generation tasks, we establish meticulous data collection and processing workflow to collect high-quality structured CU data at a scale of 0.7 billion. For visual conditions, we collect image pairs by synthesizing images from source images or by pairing images from large-scale databases. The former utilizes powerful open-source models to edit images to meet specific requirements, such as changing styles (Han al. 2024) or adding objects (Pan al. 2024), while the latter involves clustering and
2

---

Published as a conference paper at ICLR 2025
AC ACE
Input Paradigm
125
(1.) Mnt (CTT 371/1) wwggtnggg pe
V--m (0Ptaa
Tasks
Iimeration
Comeratinn
Eatting
Eihhing
Reyeinninn
Edlliig
Emmoratinn
tenpsoonet
(V)
Instruction
whoo mocern Aurwiimatl
ont Sallyy
Aer
Turget
Figure 2: The overview of generation and editing task types supported by ACE. These tasks are categorized into 8 basic types, multi-turn and long-context generation based on different input conditions (in green) and are formulated using the proposed input paradigm as 3 formats (in blue).
grouping images from extensive databases provide sufficient real data, thereby minimizing the risk of overfitting to the synthesized data distribution. For textual instructions, we first manually construct instructions for diverse tasks by building templates or requesting LLMs, then optimize the instruction construction process by training end-to-end cnttruction-laellinn
multi-modal large language model (MLLM) (Chen et al. 2024). thereby enriching the diversity of the instructions.
Our ACE provides more comprehensive coverage of tasks on single model compared to previ- ous approaches. Therefore, to thoroughly evaluate the performance of our generation model, we construct an evaluation benchmark that encompasses the main tasks. This benchmark incorporates inputs sourced from both the real world and gener--geeerated data, supporting global and local edit- ing tasks. It is larger in scale and broader in scope compared to previous benchmarks (Sheynin jet al,, 2224]] Zhang et al. 2023a). We conduct a user study to subjectively assess the quality of images generated by method and the adherence to instructions, revealing that our approach gen- erally aligns more closely with human perception across the majority of tasks. We summarize our main contributions follows:
We propose ACE, unified foundational model framework that supports a wide range of visual generation tasks. To our knowledge, this is the most comprehensive diffusion generation model to date in terms of task coverage.
By defining the CU for unifying multi-modal inputs across different tasks and incorporating long- context CU, we introduce historical contextual information into visual generation tasks, paving the way for CharGPT-like dialog systems in visual generation.
We design specific data construction pipelines for various tasks enhance the quality and effi- ciency of data collection, and we ensure the richness of multi-modal data through MLLM fine- tuning for automated instruction labeling.
We establish more comprehensive evaluation benchmark compared to previous ones, cover- ing the most known visual generation tasks. Evaluation results indicate that ACE demonstrates notable competitiveness in specialized models while also exhibiting strong generalization capa- bilities across a broader range of open tasks.
ALL-ROUND CREATOR AND EDITOR
ACE is image creation and editing model based the Diffusion Transformer that follows tex- tual instructions. It establishes unified framework that covers a wide range of tasks through the definition of standard input paradigm and strategy for aligning multi-modal information. With this
exquisite design, the model is capable of handling various single tasks, multi turituum tasks, and long- context tasks with historical information.
3

---

Published as a conference paper at ICLR 2025
2.1 PROBLEM DEFINITION
2.1.1 TASKS
When it comes to generation and editing, the input condition information varies significantly de- pending on the specific task types. This encompasses a diverse range of forms, including textual instructions, conditioning images in controllable generation, masks used in region editing, and im- ages in guided generation, among others. We analyze and categorize these conditions from textual and visual modalities respectively: (i) Textual modality: refer to all types of textual condi- tions as instructions and categorize them into Generating- Instructions, depending whether they describe the content of the generated image directly the difference from the input visual cues; (ii) Visual modality: we categorize all generation tasks into 8 basic types, as shown in Fig. 2
Instructions and Editing- based
Text-guided Generation. It only uses generating-baie bassdd text prompt as condition to create images, and none of the visual cues are adopted.
Low-level Visual Analysis. It extracts low-level visual features from input images, such as edge maps segmentation maps. One source image and editing- based instruction are required in the task to accomplish creation.
Controllable Generation. is the inverse task of Low-level Visual Analysis, which creates vivid images based on given conditions, e.g., edge map, contour image, doodle image, scribble image, depth map, segmentation map. low-resolution image, etc.
Semantic Editing. It aims to modify some semantic attributes of input image by providing editing instructions, such as altering the style of an image or modifying the facial attributes of character.
Element Editing. It focuses on adding, deleting, replacing specific subject in the image while keeping other elements unchanged. Repainting. It erases and repaints partial image content of input image indicated by given mask and instruction.
Layer Editing. It decomposes an input image into different layers, each of which contains a subject or background, reversely fuses different layers.
Reference Generation. generates an image based on one or more reference images, analyzing the common elements among them and presenting these elements in the generated image.
By leveraging the generation tasks of these fundamental units, we can combine them to create multi- turn scenarios. Furthermore, utilizing the historical information from every round makes it possible to tackle long-context visual generation tasks.
2.1.2 INPUT PARADIGM
significant obstacle to implementing different types of generation and editing task requests within one framework lies in the diverse input condition formats of tasks. To address this issue, we design a unified input paradigm defined as Conditional Unit (CU) that fits as many tasks as possible. The CUs composed of a textual instruction T that describes the generation requirements, along with visual information V, where V consists of set of images I that can be defined as I = 0 (if there source image) I = {2',12,..
NN (if there are source images) and corresponding masks N }. When there is no specific mask, M is set to a blank image. The overall
M = {M' M2....,.. MM formulation of the CU is as follows:
CU = (T,V), V= ([T:M]] (T; [T MM,),
(1)
where a channel-wiss connection operation is performed between corresponding I and M, N rep- resents the total number of visual information inputs for this task.
Furthermore, better address the demands of complex long-context generation and editing, histor- ical information can be optionally integrated into CU, which is formulated as:
LCU, {(T)-mm Ti ... T,), (Vim- Vi Vi}
where m denotes the maximum number of rounds of historical knowledge introduced in the current request. LCU, is Long-context Condition Unit used generate desired content for the i-th request.
4

---

Published as a conference paper at ICLR 2025
Casiase Tslesiong
Ramana
Jobb
Cuadiion Oui
Crallto Usi
UAI Ameake
Fall Narviee
Domsonak
fitentorg
Constban Toerunng
Packay
Posble
ALC
Aroage Perivake Eistesiingg
Ieape realstor Erahesiigg
Lapp
T
Log Cosker Sel Alassion
Mu
Lorg Unnker Drem Ateenin
Tume Cuadke Almosion
(a) Overall Architecture of ACE.
(b) Detailed Illustration of Main Blocks.
Figure 3: The illustration of ACE framework. Condition Tokenizing module tokenizes each input CU, concatenating them to obtain the visual token sequence and the text token sequence. The Image Indicator Embedding module employs pre-defined textual tokens to indicate the image order in textual instructions and distinguish various input images. The Long-context Attention Block ensures effective communication and integration of long-context sequences.
2.2 ARCHITECTURE
In this section, we introduce unified visual generation framework that can perform all visual gen- eration tasks within single model, and incorporate long-context conditions enhance compre- hension. As illustrated in Fig. Bh, the overall framework is built based on Diffusion Transformer model (Vaswani al., 2017, Peebles Xie 2023). and integrated with three novel components to achieve unified generation: Condition Tokenizing. Image Indicator Embedding, and Long-context Attention Block. We will provide a detailed description of them below.
Condition Tokenizing. Considering an LCU that comprises M CUs, the model involves three entry points for each CU: language model (T5) (Raffel et al.) 2020) to encode textual instructions, Variational Autoencoder (VAE) (Kingma & Welling 2014p compress reference image to latent representation, and a down-sampling module to resize mask to the shape of corresponding latent im- age. The latent image and its mask (an all-one mask if mask is provided) concatenated along the channel dimension. These image-mask pairs are then patchified into 1-dimensional visual token sequences um,,n. where denotes the spatial index in patchified latent images, Similarly, textual instructions are encoded into I-dimensional token sequences Vm- After processing within each CU, we separately concatenate all visual token sequences and all textual token sequences to form long-context sequence.
indexes for CUs and visual information Vs in each CU, while p
Image Indicator Embedding. As illustrated Fig. 3p. indicate the image order in tex- tual instructions and distinguish various input images, we encode some pre-defined textual tokens "[image}. {imagel]. [imageN]" into T5 embeddings as Image Indicator Embeddings (I-Emb). These indicator embeddings are added to the corresponding image embedding sequence and text embedding sequence, which formulated as:
Vnn, Vm I-Emb m,
(3)
um.pp Um,n.p + L-Embm,nn
(4)
In this way, image indicator tokens in textual instructions and the corresponding images implic-
itly associated.
Long-context Attention Block. Given the long-context visual sequence, first modulate it with the time step embedding (T-Emb), then incorporate a 3D Rotational Positional Encodings (RoPE) (Su et al. 2023) to differentiate between different spatial- and frame-level image embed- dings. During the Long Context Self-Attention. all image embeddings of each CU at each spatial location, equivalently and comprehensively interact with each other by l Attn(u a²). Next,
5

---

Published as a conference paper at ICLR 2025
Synthesiafng based
Puiring bosed
Newdarte
Pair Data
Dasa malli
Cluster
Gromp
Diejait Seea
Holluze
Conckttca
Peutures
Chusters
Levell Digaint Sore
Levelä Dijeint Sats
Template-bee
Mi..M-bneed
Instruction
specd dibllidts oe dehasiee ana Treakird
Commarted
untferstund
similarities
InternVL
Prompee
Image
procide
madil Dor Eifiii de Joone
InternVL
Menual Amnotation:
Add robbis doi wsaring pink oceralla
Owen-VL
Insteurtion
Figure 4: The pipeline of dataset construction and instructions labeling. In data construction, two methods are utilized: synthesizing using open-source expert models and mining from large- scale data. For instruction labeling, we combined templating with MLLM labeling, further training the Instruction Captioner to achieve large-scale instruction labeling.
unlike the cross-attention layer of the conventional Diffusion Transformer model, where each visual token attends to all of the textual tokens, we implement cross-attention operation with each condi- tion unit. That means image tokens in m-th CU will only attend to the textual tokens from the same CU. This can be formulated as:
um Attn(Humm H',,)
(5)
This ensures that, within the cross-attention layer, the text embeddings and image embeddings aligm on a frame-by- Frame-by-frame basis.
3 DATASETS
3.1 PAIR DATA COLLECTION
A critical challenge of training foundational visual generation model lies in how to acquire pair- wise images for various tasks. In this section, we introduce two ways to efficiently build high- quality datasets for most of the generation and editing tasks: (i) Synthesizing from source image: thanks to the rapid development in the field visual generation, there have been many of powerful open-source models designed to solve one specific problem. Leveraging these powerful single-point technologies, we could synthesis plenty of image pairs for lots of generation and editing tasks, such controllable generation, style editing, object editing, and on. (ii) Pairing frrm massive databases: though the synthesis-basee based method efficient and straightforward in acquiring pairwise data. However, It still possesses two drawbacks. First, some editing problems have not been fully ex- plored, and there are no powerful open-source models available for these tasks. Second, using only synthetic data easily cause over-fitting and reduce the quality of generated images. Therefore, is essential to provide sufficient real data to address the aforementioned drawbacks. We propose hierarchically aggregating pipeline for pairing content-related images from massive databases to build pairs of data for training, as illustrated in Fig. 4 We first extract semantic features using SigLIP (Zhai et al., 2023) from large-scale datasets (e.g., LAION-5B (Schuhmann al.) 2022). OpenImages (OpenImage 2023), and our private datasets). Then leveraging K-means clustering technology, coarse- coarse-grained clustering is implemented to divide all images into tens of thousands of clusters. Within each cluster, we implement two-turn union-find algorithm to achieve fine-grained image aggregation. The first turn is based on the SigLIP feature and the second turn uses a similarity score tailored for specific tasks. For instance, editing task and the object consistency score for the general editing task. Finally, we collect all possible pairs from each disjoint set and implement cleaning strategies to filter high-quality pairs.
calculate the face similarity score for the facial

---

Published as a conference paper at ICLR 2025
Benefiting from these two automatic pipelines, we construct a large-scale training dataset that con- sists of nearly 0.7 billion image pairs, covering basic types of tasks, multi-turn and long-context generation. We depict its distribution in Fig. and provide a detailed description of the specific data construction methods for each task, please refer to appendix
3.2 INSTRUCTIONS
In addition to collecting image pairs, it is essential label clear natural language instructions that indicate how to transform one image into another. Compared to the caption generation commonly used in text-to-image task, instruction labeling is generally more challenging, as requires analyzing not only the semantics of individual images, but also the discrepancies across multiple images. We employ both Template-based and MLLM-based methods to tackle this challenge. Template-basee method constructs instruction templates for specific vision tasks by leveraging human knowledge priors. However, the instructions generated by this method lack diversity, which can lead to signifi- cant overfitting problems. MLLM-based method generates unique instructions for each given editing pair, leveraging off-the-shelf MLLMs. Nonetheless, current MLLMs exhibit limitations in produc- ing precise instructions for editing tasks involving non-natural images, such depth-controlled image generation and image segmentation. Thus, we combine these two methods and design an effective strategy mitigate the aforementioned drawbacks. For tasks that contain non-natural im- ages, we utilize a template-ban based method to generate instruction templates. These templates are then combined with the generated captions to produce the final instructions. To address the issue of insufficient diversity, we employ LLMs to reformulate instructions multiple times, and tune prompts ensure that each rewritten version is distinct from all preceding instructions. For tasks that con- tain natural images, we employ an MLLM to predict the differences and commonalities between the images in the input pair. Then an LLM is used to generate instructions focusing on semantic distinctions according to the analysis of the differences and commonalities. Further, the collected instructions generated by these two methods undergo human annotation and correction. The revised instructions used for fine-tuning open-source MLLM, enabling it to predict instructions for any given image pair. Specifically, we collect a dataset of approximately 800,000 curated instruc- tions and train Instruction Captioner by fine-tuning the VL2-22L (Chen et al., 2024). Once trained, the Instruction Captioner is able to take any two images as input and generates the instruction for transforming the source image to the target image. It can also be further extended to the processing of cluster data, by entering a set of images, obtaining the similarity description among images within the cluster, and the differences between each pair within the cluster. The above process is illustrated in Fig.
4 EXPERIMENTS
4.1 BENCHMARKS AND METRICS
Existing Benchmarks. We first evaluate the commonly used benchmark MagicBrush (Zhang ct al. 2023a). It contains an overall 1,053 edit turns and 535 edit sessions for single-turn and multi- turn image editing respectively. It compares the output images with groundtruth images and the provided target text descriptions. Following the setting proposed in the MagicBrush benchmark, we calculate the LI distance, L2 distance, CLIP (Radford et al., (021) similarity, DINO (Liu al.) 2023ab similarity between the generated image and groundtruth image, and CLIP similarity between the generated image and textual prompt. We also evaluate the Emu Edit benchmark (Sheynin al. 2024), please see appendix F For details.
ACE Benchmark. To thoroughly evaluate the performance of various visual generation tasks, we build a benchmark dataset that covers all types of tasks the aforementioned. ACE benchmark consists of both real and generated images. The real images are primarily sourced from the MS- COCO (Lin et al. (2014) dataset and the generated images are created by Midjourney (Midjoumny) 2023), using prompts obtained from JourneyDB (Sun et al. 2023a). For each task type, we manu- ally craft instructions and masks to closely resemble actual user input patterns, reaching total of 12,000 entries. The detailed statistics of ACE benchmark can be found in Fig. 24 We evaluate image quality and prompt following scores through user study. The image quality score assesses the aesthetic quality of the generated images, while the prompt following score measures how well the images align with the provided textual instructions.
7

---

Published as a conference paper at ICLR 2025
lepert
Labe
JCB
Faod
Tarrerter
Fiom Blitiig
Thxs Roweao
STOP
Teah Boakee
Shopp
Shop
Auginaly Serige
of Dawrey
Osen Bezunal
Latradion
Deqeite
Secleneee Cormutinn Thee grine Emanoood
Satredies: Amartt, Smauth Smaeiif susteee
Mew Amannd Smaart Amapri wn Coeend ana bockent beclonend leeur
Terracten:
Termerken
Clouer dimeptho sireew Sinent
Figure 5: Comparison and visualization of ACE performance with expert models in different
tasks. ACE demonstrates adaptability to multi-task and achieves superior performance.
8

---

Published as a conference paper at ICLR 2025
Table Results on the MagicBrush benchmark. LC denotes long-context generation with history.
Settings
Methods
Lll
L24
CLIP-It
DINO!
CLIP-TY
Global Description-giieed
SD SD.Siid [Meng 2021
0.1014
0.0278
0.8526
0.7726
0.2777
Null Text Inversion Mocady et 2022
0.0749
0.0197
0.8827
0.8206
0.2737
GLIDE (Nichol 2022
3 3,4973
115.8347
0.9487
0.9206
0.2249
Single-tum
Blended Diruston Availani et
2022
3.5631
119.2813
0,9291
0.8644
0.2622
(Ours)
0.0505
0.0160
Instructionguided
HIVE [hang et al.
2024
0.1092
0.0380
0.8519
0.7500
InstructPix22PXx
0.1122
0.0371
0.8524
0.2764
MagicBrush Zhang (2225))
0.0625
0.0203
0.9332
0.8987
0.2781
UltraEdit ZTha
ACE (Ours)
0.0507
0.0165
0.9453
0.9215
0.2841
Global Desrritiio-guiited
SD-SDLidit Meng
0.0602
0.7933
0.2694
Null Text Inversion Moxaay (Nichol
2022
0. 1057
0.0335
0.8468
0.7529
0.2710
11.7487
1079.5997
Bleoded Dilfusion Avaliamii al.
2022
14.5439
1510.2271
0,8782
0.7690
0.2619
(Our))
0.0778
0.0290
0.9124
0.8611
0.28443
ACE (Ours w/ LC)
0.0768
0.0285
0.9136
0.8635
0.28199
Molti-
Instruction-gided
[Zhang et 2024
0.0557
0.2673
InstructPixxPP 2PX Brocks
2023
0. 1584
0.0598
0.7924
0.6177
0.2726
MagicBrush Zhang el 2023g
0.0964
0.0353
0.8924
0.8273
0.2754
Ultradit 7hao
0.0745
0.0236
0.9045
0.8505
ACE (Ours)
0.0773
0.0293
0.9128
0.8661
0.2855
w/
0.0284
4.2 QUALITATIVE EVALUATION
In our qualitative evaluation, we present comparison of our method with SOTA approaches across various tasks, including ControlNet (Zhang et al.) 2023b), InstruetPix2Pix (Brooks
2023).
MagicBrush (Zhang et al. 2023a), CosXI. AI, 2024). SEED-X Edit (Ge et al.
2024a
UltraEdit (Zhao et al., (2024). StyleBooth (Han et al. 2024). SDEdit Meeg et al. 2021) 1), LoRA (Hu
al. 2022). SD-Inpaint (AI, 2022b), LaMa (Suvorov et al. 2022). IP-Adapter (Ye et al. 2023),
Instant (Wang et al. 2024b), FaceChain (Liu et al., 2023b). Any Tuoo et al. 2023). UDiff-
Text (Zhao & Lian 2024). In Fig. 5 we present qualitative comparisons between single ACE
model and 16 other methods across 12 subtasks. Overall, our method not only addresses diverse range of tasks but also performs superior compared to task-specific methods. Additionally, also show some extra tasks that the comparison methods do not perform well in the last three lines. Please see appendix H for more examples of qualitative evaluation.
4.3 QUANTITATIVE EVALUATION
Evaluation Existing Benchmarks. We first compare method with baselines the Mag- icBrush benchmark. Results are present on Tab. [ For single-turn image editing, ACE significantly outperforms other methods under an instruction-guided setting while demonstrating comparable per- formance under description-guide employ the same inference way MagicBrush, performing independent and continuous edits single image. The results show that our approach has significant advantages. Furthermore, we con- struct a long sequence using the historical information from each editing round, achieving a certain improvement in performance compared to not using it. This also demonstrates the effectiveness of LCU and architecture design.
setting. For each setting of multi-turn image editing, we first
Evaluation on ACE Benchmark. We conduct a comprehensive human evaluation using our bench- mark to assess the performance of generated images, employing image scoring as the evaluation metric. Specifically, quality. The prompt following metric measures the image compliance with text instructions or text descriptions, and is categorized into five levels. The image quality metric encompasses various pects such as generated color, details, layout, and visual appeal, and is scored on a scale from to 5. Considering the broad capabilities of our method, compare with several common approaches
score each image considering two aspects: prompt following and image

---

Published as a conference paper at ICLR 2025
Table 2: User study results on ACE benchmark. For each method in every supported task, we evaluate both prompt following and image quality, reporting the two scores in single cell, separated by a "/" *_" means this task does not exist or not supported by the current method.
TxtZing
Controllable
Semantic
Element
Repainting
Txt2imn
Scribbi
Text
Add Obj
Run Obj.
Ouipaint
Canny
Depth
Powe
Foee
Style
Add
Inpaint
Global Editing
SD155
A1 20224
3.3/2.2
SDXL
StamnivAl
2022
4.1/2.8
ChrNee
Zhang
217 9b
2.5/2.0
3.824 1,92 2.9/19
StyleBooin
(Han ei
2024
3.M2.6
IP-Adapter
2.0/2.2
1.725
InstantID
Wang cl
2024b
2.3/2.7
PaceChain Liu at
20235
2.0/3.0
SDEdit
Meng
2021
1./199
1.311
1.161.
1.2/14
1.3/2.1 1.1177
1.52.
1
1.5/2.1
1.1/2.11.1122 1,12.0
IP2P Brocks el
202.3
1.9/20
1.7/2.0
14/1
1/2626
1.361.32.666
2.02
1.5/244
MB
Zhang
2112
1.31.8
1.M1.7
1,3/19
11/
2,4/33
142.0222. 1.2.2.2.222222.20
1,5224
2.2/5
3.12
172.4
SEED-X
(Ge a
21744
1.6/2 1.7/2.0
1.7/2.2
5/
212/222222222222 2.0/2
1.3/2.6
2.26
1.962.6
2.52
CosXL (SabiiitA
4.1/2.8
3.7/2
3.233/3
2.82.5
UltraEsht Zhaa et
2024
1.7/2.2 1,2/188
1.3/2.3
1.1/1
2.3/2.5
2.1/224
1.7/2.6
1.122.7
2.7/2.3
1.5/2.6
ACE (Ours)
3.7/2.5
4.6/2.7
4.5/2.8
4.8/2.9
4.1/2.3 2 2.42
2.82
4,4/2.
2.6/244
3.9/2
Local Editing
LaMa (Suvonov
2022
1.6/2.8
4,5/2.8
1.5/2.81.6/2..
1.6/2.1 3.0/2.4
SDInpaint
3.6/66
CurNNet Zhing et
2023b
2.9/2.7
1,92.5
2.6/2.2
3.02.1 3.2/111
AnyTeal [oo el al
3.5/2.7
UDifTea Zhao
2024
3.6/2.7
Ultradit Zhan ef al
2124
1//1.9 1,2/18 1.2/2.0
1.1/2.8
1.2/2.9
2.9/2
1.4/25
1. 122.1
ACE (Ours)
4.8/2.6 4.3/2.5 4.8/2.6
4.5/2.9
4.5/2.9 3.7/2.5
4.323
4.4/2.7 4.6/8.8
and some experts designed for specific tasks. We engaged 5 professional designers as evaluators to carry out these assessments. For each task, the data is evenly distributed among the evaluators in an anonymous manner, and scores are aggregated for analysis.
As shown in Tab. we compare our approach across multiple global editing tasks and local editing tasks. The prompt following score and image quality score are presented together, separated by "/" pattern. The bold numbers represent the best, and the underlined numbers indicate the second best. Our method achieves the highest prompt following scores in 7 of 12 global editing tasks and 8 of 10 local editing tasks, which demonstrates that ACE fully understands the intention the instruction and is able to correctly generate an image that meets the instruction. Furthermore, ACE achieves the best image quality scores in results indicate that ACE excels at generating high aesthetic images across various image editing tasks. Nonetheless, our method performs unsatisfactorily in certain tasks, such as general editing and style editing. One possible reason is that images generated by methods using larger models, such as those producing 1024-resolution images based on the SDXL model, are more preferred by evaluators compared those produced by our model, which has size of 0.6B parameters and an output resolution of around 512.
global editing tasks and 10 local editing tasks. These
5 CONCLUSION
We propose ACE, versatile foundational generative model that excels at creating images, and following instructions across wide range of generative tasks. Users can specify their generation intentions through customized text prompts and image inputs. Furthermore, we advance the explo- ration of capabilities within interactive dialogue scenarios, marking significant step forward in the processing of long contextual historical information in the field of visual generation. Our work aims to provide a comprehensive generative model for the public and professional designers, serving as a productivity enhancement tool to foster innovation and creativity.
Acknowledgments. We sincerely appreciate the contributions of many colleagues for their insight- ful discussions, valuable suggestions, and constructive feedback, including: Haiming Zhao, Yuntao Hong, You Wu, Jixuan Chen, Yuwei Wang, and Sheng Yao for their data contributions, and Lianghua Huang, Kai Zhu, and Yutong Feng for their discussions, suggestions, and the sharing of resources.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, Alon Benhaim, Misha Bilenko, Johan Bjorek, Sébastien Bubeck, et al. Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone. arXiv preprint arXiv:2404.142119 2024.
Runway AI. Stable Diffusion v1.5 Model Card, bttpss//auggLnggfacccc/// 2022a.
st stable-diffusion-1
Runway AI Stable Diffusion Inpainting Model Card, httpss:/huuggingaace co/
Furwayml /ssabssaallseeee ffusiiifuoooooo innnannnntt..m 2022b.
Rohan Anil, Andrew M. Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, et al. PaLM Technical Report. arXiv preprint arXiv:2305.10403, 2023.
Anthropic.
Introducing Claude, BtDps//ww.antaargiiccooo////
introducing-claune 2023a.
Anthropic. Claude 2. Technical report,
productions ion
2023b.
Anthropic. The Claude 3 Model Family: Opus. Sonnet, Haiku, https:/
anthropic com/ de www.eeebbbbbbllllec http:////...........00000000.0.1 Card_ Clw_eelaa_ee__e pdf 2024.
Omri Avrahami, Dani Lischinski, and Ohad Fried. Blended Diffusion for Text-driven Editing of Natural Images. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 18208-18218, 2022.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Ge, Yu Han, Fei Huang. Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, et al. Qwen Technical Report. arXiv preprint arXiv:2309.16609 2023.
Rumeysa Bodur, Erhan Gundogdu, Binod Bhattarai, Tye-Kyun Kim, Michael Donoser, and Loris Bazzani. iEdit: Localised Text-guided Image Editing with Weak Supervision. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 7426-7435, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A. Efros. InstructPix2Pix: Learning To Follow Image Editing Instructions. In IEEE Conf. Comput. Vis. Pattern Recog,, pp. 18392-18402, 2023.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, et al. Language Models are Few-Shot Learn- ers. arXiv preprint arXiv:2005.1416. 2020.
John Canny. A Computational Approach to Edge Detection. IEEE Trans. Pattern Anal. Mach. Intell., pp. 679-66986699 1986.
Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh. OpenPose: Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. IEEE Trans. Pattern Anal. Mach. Intell., 1721):172-186, 2021. ISSN 0162-8828, 2160-9292, 1939-3539.
Caroline Chan, Frédo Durand, and Phillip Isola. Learning To Generate Line Drawings That Convey Geometry and Semantics. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 7915-7925, 2022.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. PixArt-o: Fast Training of Diffusion Transformer for Photorealistic text-to-Image Synthesis. arXiv preprint arXiv:2310.00422, 2023a.
Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao. AnyDoor: Zero-shot Object-level Image Customization. arXiv preprint arXiv:2307.0948// 2023b.
11

---

Published as a conference paper at ICLR 2025
Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, and Jifeng Dai. InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistie Tasks. In IEEE Conf. Comput. Vis. Pattern Recog- pp. 24185-241999 2024.
Seunghwan Choi, Sunghyun Park, Minsoo Lee, and Jaegul Choo. VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalization. In IEEE Conf. Comput. Vis. Patlern Recog-, pp. 14131-14140, 2021.
Alibaba Cloud. Tongyi Wanxiang, /ttp::s//ooooppilllllyy.o.ocoo www.aa/aaala...oomoacaaaaaaaaa 2023.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. ArcFace: Additive Angular Margin Loss for Deep Face Recognition. In IEEE Conf. Comput. Vis. Puttern Recog, pp. 4690-4699, 2019a.
Jiankang Deng, Jia Guo, Debing Zhang, Yafeng Deng, Xiangju Lu, and Song Shi. Lightweight Face Recognition Challenge. Int. Conf. Comput. Vis., pp. 0-0, 2019b.
Yuning Du, Chenxia Li, Ruoyu Guo, Xiaoting Yin, Weiwei Liu, Jun Zhou, Yifan Bai, Zilin Yu, Yehua Yang, Qingqing Dang, and Haoshuang Wang. PP-OCR: A Practical Ultra Lightweight OCR System. arXiv preprint (rXi:22009.09941, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, et al. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783, 2024.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, Dustin Podell, Tim Dockhorn, Zion En- glish, Kyle Lacey, Alex Goodwin, Yannik Marek, and Robin Rombach. Scaling Rectified Flow Transformers for High- Resoluteso Image Synthesis. In Int. Conf. Mach. Learn., 2024.
FLUX. FLUX, httpps://blacccoreeeetteee
ai/ 2024.
Yuying Ge, Sijie Zhao, Chen Li, Yixiao Ge, and Ying Shan. SEED-Data-Eda Technical Report: A Hybrid Dataset for Instructional Image Editing. arXiv preprint arXiv:2405. 2024a.
Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. SEED-X: Multimodal Models with Unified Multi-granularity Comprehension and Generation. arXiv preprint 2rXiv:2404.144 /4396, 2024b.
Zigang Geng, Binxin Yang, Tiankai Hang, Chen Li, Shuyang Gu, Ting Zhang, Jianmin Bao, Zheng Zhang, Houqiang Li. Han Hu, Dong Chen, and Baining Guo. InstructDiffusion: A Generalist Modeling Interface for Vision Tasks. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 12709- 12720, 2024.
Zhen Han, Chaojie Mao, Zeyinzi Jiang, Yulin Pan, and Jingfeng Zhang. StyleBooth: Image Style Editing with Multimodal Instruction. arXiv preprint arXiv:2404. /2/54, 2024.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhr -Zu,, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. LoRA: Low-Rank Adaptation of Large Language Models. In Int. Conf. Learn. Represent., 2022.
Jiehui Huang, Xiao Dong, Wenhui Song, Hanhui Li, Jun Zhou, Yuhao Cheng, Shutao Liao, Long Chen, Yiqiang Yan, Shengcai Liao, and Xiaodan Liang. ConsistentID: Portrait Generation with Multimodal Fine-Grained Identity Preserving. arXiv preprint arXiv:2404.16771 2024a.
Lianghua Huang, Di Chen, Yu Liu, Yujun Shen, Deli Zhao, and Jingren Zhou. Composer: Creative and Controllable Image Synthesis with Composable Conditions. In Int. Conf. Mach. Learn., 2023.
Yuzhou Huang, Liangii Xie, Xintao Wang. Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, and Ying Shan. SmartEdit: Exploring Com- plex instruction-basees Image Editing with Multimodal Large Language Models. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 8362-8371, 2024b.
12

---

Published as a conference paper at ICLR 2025
Tao Jiang, Peng Lu, Li Zhang, Ningsheng Ma, Rui Han, Chengqi Lyu, Yining Li, and Kai Chen. RTMPose: Real-Time Multi-Persoe Pose Estimation based on MMPose. arXiv preprint WrXiv:2303307399. 2023.
Zeyinzi Jiang, Chaojie Mao, Yulin Pan, Zhen Han, and Jingfeng Zhang. SCEdit: Efficient and Controllable Image Diffusion Generation via Skip Connection Editing. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 8995-9004, 2024.
Diederik P. Kingma and Max Welling. EncooEngg Variational Bayes. Int. Conf. Learn. Represent., 2014.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C. Berg, Wan-Yen Lo, Piotr Dollár, and Ross Girshick. Segment Anything. In Int. Conf. Comput. Vis., pp. 4015-4026, 2023,
KOLORS, KOLORS,
Koloooooooooooooo holae//oooooaaaaooooooooooooooo 2024.
com
Pengzhi Li, QInxuan Huang, Yikang Ding, and Zhiheng Li. LayerDiffusion: Layered Controlled Image Editing with Diffusion Models. arXiv preprint 2.K.i:::305.186766 2023.
Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong. Yanxin Long. Xinchi Deng. Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, et al. Hunyuan-DiT: A Powerful Multi- Resolution Diffusion Transformer with Fine-Grained Chinese Understanding. arXiv preprint arXiv:2405.08748, 2024.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Mierosoft COCO: Common objects in context. In Eur: Conf. Comput. Vis., pp. 740-755, 2014.
Shilong Liu, Zhaoyang Zeng. Tianhe Ren. Feng Li, Hao Zhang, Jie Yang. Chunyuan Li, Jianwei Yang. Hang Su, Jun Zhu, and Lei Zhang. Grounding DINO: Marrying DINO with Grounded Pre-Traine Pre-Training for Open-Set Object Detection. arXiv preprint arXiv:2303.05449 2023a.
Yang Liu, Cheng Yu, Lei Shang, Yongyi He, Ziheng Wu, Xingjun Wang. Chao Xu, Haoyu Xie, Weida Wang, Yuze Zhao, Lin Zhu, Chen Cheng, Weitao Chen, Yuan Yao, Wenmeng Zhou, Jiaqi Xu, Qiang Wang. Yingda Chen, Xuansong Xie, and Baigui Sun. FaceChain: A Playground for Human-centric Artificial Intelligence Generated Content. arXiv preprint arXiv:2308. 14256, 2023b.
Ilya Loshchilov and Frank Hutter. Decoupled Weight Decay Regularization. In Inl. Conf. Learn. Represent., 2018
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun- Yan Zhu, and Stefano Ermon. SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. In Int. Conf. Learn. Represent., 2021.
Midjoumey. Midjoumey, httpt:t//////..... http:://wwwwwt/tm www.:ddddooureree com 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yacl Pritch, and Daniel Cohen-Or. Null-text Inversion for Editing Real Images using Guided Diffusion Models. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 6038-6047, 2022.
Davide Morelli, Matteo Fincato, Marcella Cornia, Federico Landi, Fabio Cesari, and Rita Cucchiara. Dress Code: High-Rea Reshhueesoltionn Multi-Category Virtual Try-On. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 2230-2234, New Orleans, doi: 10. 1099CVPPW5647..02200022222 ieee http:///e.e.e...oooooooooooeeeee
LA, USA, 2022. IEEE. ISBN 978 1-611---66448---9999
URL httpp:://eeexpplpcccc
Chong Mou, Xintao Wang. Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, Ying Shan, and Xiaohu Qie. A2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to- Image Diffusion Models, arXiv preprint 2302 2023.
13

---

Published as a conference paper at ICLR 2025
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. GLIDE: Towards Photorcalistic Image Generation and Editing with Text-Guidee Diffusion Models. arXiv preprint arw.:: 2022.
OpenAI. DALL-E http:://ppeaal..cm///llll1-1 2022.
OpenAL. DALLE 3, https://pperaa..com/ca.ll.. 2023a.
OpenAl. GPT-4 Technical Report. prwpxxxrrrprrrnnxxxxxx200000.770 2023b.
OpenAl. Hello (Ft.a//ttts//peeaaaa...////////d 40/ 2024.
Openlmage. index. html 2023.
Long Ouyang, Jeffrey Wu, Xu Jiang. Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, et al. Training language models to follow instructions with human feedback. Adv. Neural Inform. Process. Syst. pp. 27730-27744, 2022.
Yulin Pan. Chaojie Mao, Zeyinzi Jiang, Zhen Han, and Jingfeng Zhang. Refine: Taming Customized Image Inpainting with Text-Subject Guidance. arXiv preprint 1rXiv:2403.19534 2024.
Locate, Assign,
William Peebles and Saining Xie. Scalable Diffusion Models with Transformers. In Int. Conf. 2023.
Comput. Vis., pp.
Can Qin, Shu Zhang, Ning Yu, Yihao Feng, Xinyi Yang, Yingbo Zhou, Huan Wang. Juan Car- los Niebles, Caiming Xiong, Silvio Savarese, Stefano Ermon, Yun Fu, and Ran Xu. UniCon- trol: Unified Diffusion Model for Controllable Visual Generation In the Wild. arXiv preprint arXiv:2305.1 147, 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv preprint arXiv:2103.0002. 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the Limits of Transfer Learning with Unified Text-to- Text Transformer. J. Mach. Learn. Res., pp. 1-67, 2020.
René Ranftl, Katrin Lasinger, David Hafner, Konrad Schindler, and Vladlen Koltun. Towards Ro- bust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer. 1EEE Trans. Pattern Anal, Mach. Intell., pp. 1623-1637, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE Conf. Comput. Pattern Recog pp. 10684-10695, 2022.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfrr Aberman. DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generation. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 22500-22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang. Emily Denton, Seyed Kam- yar Seyed Ghasemipour, Burcu Karagol Ayan, S. Sara Mahdavi, Rapha Gontijo Lopes, Tim Sal- imans, Jonathan Ho. David J. Fleet, and Mohammad Norouzi. Photorealistic Text-to-lmage Dif- fusion Models with Deep Language Understanding. In Adv. Neural Inform. Process. Syst., 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W. Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R. Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5B: An open large-scale dataset for training next generation image-text models. In Adv. Neural Inform. Process. Syst., 2022.
14

---

Published as a conference paper at ICLR 2025
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu Edit: Precise Image Editing via Recognition and Generation Tasks. In IEEE Conf. Comput. Vis. Pattern Recog., pp. 0871-8879 2024.
Yujun Shi, Chuhui Xue, Jun Hao Liew, Jiachun Pan, Hanshu Yan, Wenqing Zhang, Vincent Y. F. Tan, and Song Bai. DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing. In IEEE Conf. Comput. Vis. Puttern Recog., pp. 8839-8849, 2024.
Gowthami Somepalli, Anubhav Gupta, Kamal Gupta, Shramay Palta, Micah Goldblum, Jonas Geip- ing, Abhinav Shrivastava, and Tom Goldstein. Measuring Style Similarity in Diffusion Models. arXiv preprint arXiv:2404.01292, 2024.
StabilityAI. Stable Diffusion XL Model Card, http:://wggiinnaaee.cco/aaaaal/ stablle-diffuaaioo----baaass
2022.
StabilityAI. CosXL Model Card, httpss//wuggiiffacccco/tttaatatp 2024.
Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu. RoFormer: En- hanced Transformer with Rotary Position Embedding. arXiv preprint arXiv22104008864 2023.
Kegiang Sun, Junting Pan, Yuying Ge, Hao Li, Haodong Duan, Xiaoshi Wu, Renrui Zhang, Aojun Zhou, Zipeng Qin, Yi Wang, Jifeng Dai, Yu Qiao, Limin Wang, and Hongsheng Li. JourneDD:: A Benchmark for Generative Image Understanding. In Adv. Neural Inform. Process. Syst., 2023a.
Ya Sheng Sun, Yifan Yang, Houwen Peng, Yifei Shen, Yuqing Yang, Han Hu, Lili Qiu, and Hideki Koike. ImageBrush: Learning Visual In-Context Instructions for Exemplar- Image Manip- ulation. In Adv. Neural Inform. Process. Syst., 2023b.
Roman Suvorov, Elizaveta Logacheva, Anton Mashikhin, Anastasia Remizova, Arsenii Ashukha, Alcksei Silvestrov, Naejin Kong, Harshith Goka, Kiwoong Park, and Victor Lempitsky. Resolution-Robust Large Mask Inpainting With Fourier Convolutions. In IEEE Winter Conf. Appl. Comput. Vis., pp. 2149-2159, 2022.
Yuxiang Tuo, Wangmeng Xiang, Jun-Yan He, Yifeng Geng, and Xuansong Xie. AnyText: Multilin- gual Visual Text Generation and Editing. In Int. Conf. Learn. Represent. 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, f. ukasz Kaiser, and Illia Polosukhin. Attention is All you Need. In Adv. Neural Inform. Pro- cess. Syst., 2017.
Haofan Wang. Matteo Spinelli, Qixun Wang, Xu Bai, Zekui Qin, and Anthony Chen. In- stantStyle: Free Lunch towards Stylc-Preserving in Text-to-Iman to-mmage Generation. arXiv preprint arXivv2404.02733,, 2024a.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. InstantID; Zero-shot Identity- Preserving Generation in Seconds. arXiv preprint arXiv:2401.0755 2024b.
Xintao Wang, Liangbin Xie, Chao Dong, and Ying Shan. Real-ESRGAN: Training World Blind Super-Resolution with Pure Synthetic Data. In Int. Conf. Comput. Vis., pp. 1905-1914, 2021.
Zhizhong Wang, Lei Zhao, and Wei Xing. StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models. In Int. Conf. Comput. Vis., pp. 7677-7689, 2023.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang. SmartBrush: Text and Shape Guided Object Inpainting With Diffusion Model. In IEEE Conf. Comput. Vis. Pattern Recog. pp- 2023.
Yunyang Xiong, Bala Varadarajan, Lemeng Wu, Xiaoyu Xiang, Fanyi Xiao, Chenchen Zhu, Xiao- liang Dai, Dilin Wang, Fei Sun, Forrest landola, Raghuraman Krishnamoorthi, and Vikas Chan- dra. EfficientSAM: Leveraged Masked Image Pretraining for Efficient Segment Anything. In IEEE Conf. Comput. Vis. Pattern Recog, pp. 1611 16111-16121, 2023.
15

---

Published as a conference paper at ICLR 2025
Jiacong Xu, Zixiang Xiong, and Shankar P. Bhattacharyya. PIDNet: A Real-time Semantic Seg- mentation Network Inspired by PID Controllers. In IEEE Conf. Comput. Vis. Pattern Recog., pp- 19529-19539, 2023.
An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, et al. Qwen2 Technical Report. arXiv preprint arXiv:2407.10671 2024.
Yukang Yang, Dongnan Gui, Yuhui Yuan, Weicong Liang, Haisong Ding, Han Hu, and Kai Chen. GlyphControl: Glyph Conditional Control for Visual Text Generation. In Adv. Neural Inform. Process. Syst. 2023.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. IP-Adapter: Text Compatible Image Prompt Adapter for text-to-Image Diffusion Models. arXiv preprint arXiv:2308.067. 2023.
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid Loss for Language Image Pre- Trainiggg In Int. Conf. Comput. Vis., pp. 11975-11986, 2023.
Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, and Haifeng Wang. ERNIE-ViL.G: Unified Generative Pre-training for Bidirectional Vision-Language Generation. arXiv preprint arXiv:2112.15288 2021.
Hua Zhang. Si Liu, Changging Zhang, Wenqi Ren, Rui Wang, and Xiaochun Cao. SketchNet: Sketch Classification with Web Images. In IEEE Conf. Comput. Vis. Patlern Recog,, pp. 1105- 1113, 2016a.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. MagicBrush: A Manually Annotated Dataset for nusided Guided Image Editing. In Adv. Neural Inform. Process. Syst., 2023a.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. Joint Face Detection and Alignment Using Multitask Cascaded Convolutional Networks. IEEE Sign. Process. Letters, pp. 1499-1503, 2016b.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding Conditional Control to Text-to-Image Diffusion Models. In Int. Conf. Comput. Vis., pp. 3836-3847, 2023b.
Shu Zhang, Xinyi Yang, Yihao Feng, Can Qin, Chia-Chih Chen, Ning Yu, Zeyuan Chen, Huan Wang, Silvio Savarese, Stefano Ermon, Caiming Xiong, and Ran Xu. HIVE: Harnessing Human Feedback for Instructional Visual Editing. In IEEE Conf. Comput. Vis. Pattern Recog.. pp. 9026- 9036, 2024.
Youcai Zhang. Xinyu Huang, Jinyu Ma, Zhaoyang Li, Zhaochuan Luo, Yanchun Xic, Yuzhuo Qin, Tong Luo, Yaqian Li, Shilong Liu, Yandong Guo, and Lei Zhang. Recognize Anything: A Strong 2023c.
Image Tagging Model. arXiv preprint arXiv:2306.03::
Haozhe Zhao, Xiaojian Ma. Liang Chen, Shuzheng Si. Rujie Wu, Kaikai An, Peiyu Yu, Minjia Zhang, Qing Li, and Baobao Chang. UltraEdit: Instruction-based Fine-Grained Image Editing at Scale. arXiv preprint arXiv:2407.05228V) 2024.
Shihao Zhao, Dongdong Chen, Yen-Chun Chen, Jianmin Bao, Shaozhe Hao, Lu Yuan, and Kwan Yee K. Wong. Uni-ControLNet All-in-One Control to Text-to-Image Diffusion Models. Adv. Neural Inform. Process. Syst., 2023.
Yiming Zhao and Zhouhui Lian. UDiffText: A Unified Framework for High-quality Text Synthesis in Arbitrary Images via Character-aware Diffusion Models. In Eur. Conf. Comput. Vis., 2024.
16


Reviews:
[{"review_id": "Ut7V2TYefS", "reviewer": "Reviewer_NXSb", "summary": "The paper presents a method to train a unified model for 8 different tasks: Text-guided Generation, Low-level Visual Analysis, Controllable Generation, Semantic Editing, Element Editing, Repainting, Layer Editing and Reference Generation. The idea is intuitive. The main contribution of the paper is the framework for generating paired training data. The source of the data generation comes from two aspects: 1. synthetic generation and 2. from publicly available datasets (LAION-5B). To verify the results of this task, authors also create a new benchmark called ACE Benchmark.", "questions": "Indeed, as I mentioned in the weakness, how those tasks affect each other?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The dataset generated in this paper is beneficial to the community. This will help other researchers follow this series of research works.\n2. A unified model for all tasks is also more efficient compared to have several individual models specific to certain type of tasks.", "weaknesses": "1. It seems the author does not have clear discussions on how those tasks affect each other. Are they beneficial to each other? Or some of the tasks are reducing the performance of other tasks? How to select the most reasonable tasks that should be unified with the single model? I believe adding this type of discussion with corresponding experiments will make the paper more solid.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "u6jviIDAma", "reviewer": "Reviewer_vGPZ", "summary": "This work proposes an All-round Creator and Editor as a unified foundation model for visual generation tasks. The main technical contribution lies in introducing a Long-context Condition Unit that standardizes diverse input formats. Built upon diffusion transformers, the architecture incorporates condition tokenizing, image indicator embedding, and long-context attention blocks to achieve unified visual generation capabilities. To address the scarcity of training data, the authors develop a data collection pipeline that combines synthesis/clustering-based approaches. Additionally, they establish a comprehensive benchmark for evaluating model performance across various visual generation tasks.", "questions": "1. Regarding Figure 6, the authors are encouraged to elaborate on the empirical or theoretical basis for the chosen data distribution and its specific advantages for the ACE model.\n\n2. The paper would benefit from addressing the practical challenges of model updates. Specifically, how might one efficiently incorporate new functionalities without complete model retraining? This consideration is crucial for the model's practical deployment and ongoing development.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1) The framework unifies multiple image generation and editing tasks through a single model, avoiding the hassle of calling separate specialized models. The proposed LCU provides a structured approach to incorporating historical context in visual generation.\n\n2) The paper presents systematic methodologies for data collection and instruction construction, which contributes to the development of all-in-one visual generative foundation models.\n\n3) The evaluation benchmark provides comprehensive coverage across diverse image manipulation and generation tasks, enabling thorough performance assessment.", "weaknesses": "Technical Issues:\n1. Formatting inconsistencies: in lines 417-418, the image placement obscures instruction text.\n\n2. The authors are encouraged to provide discussions on task-specific performance trade-offs during training, specifically how optimizing for one task might affect the performance of others.\n\n3. It would be helpful to provide methodological details regarding parameters in data preparation (lines 321-325), such as cluster number determination and data cleaning criteria.\n\n4. The qualitative results in Figure 5 reveal some limitations. 1) Row 1 (left): ACE generates a distorted hand. 2)  Row 2 (right) and Row 4 (left): The model exhibits undesired attribute modifications not specified in the instructions, including unintended gesture alterations / head rotation changes, and camera perspective shifts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "R7HHx8W7gp", "reviewer": "Reviewer_vndP", "summary": "1. Propose ACE, a unified foundational model framework that supports a wide range of visualgeneration tasks, achieve a best task coverage.\n2. Define the CU for unifying multi-modal inputs across different tasks and incorporate long context CU.\n3. Design specific data construction pipelines for various tasks to enhance the quality and eff-ciency of data collection.\n4. Establish a more comprehensive evaluation benchmark compared to previous ones, cover-ing the most known visual generation tasks. \n\nIt's a lot of work, from method to data to data construction pipelines to benchmark, very systematic and complete work.\nAnd all-in-one models are really interesting and is consistent with the general trend of generate model development.\nBut,\ndrawings are terrible, and the method is a little weak. Maybe you could use the all-in-one methods in low-level works as reference.", "questions": "Please DRAW better.\nI don't find the computing resource? I think it would be big, maybe you could have a discuss.\nRelated work? i think there should be other works that make try building an all-in-one visual generation model. Maybe you could list them clearly, I'm not an expert on this.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Propose ACE, a unified foundational model framework that supports a wide range of visualgeneration tasks, achieve a best task coverage.\n2. Define the CU for unifying multi-modal inputs across different tasks and incorporate long context CU.\n3. Design specific data construction pipelines for various tasks to enhance the quality and eff-ciency of data collection.\n4. Establish a more comprehensive evaluation benchmark compared to previous ones, cover-ing the most known visual generation tasks\n5.Analyze and categorize these conditions from textual and visual modalities respectively, includeTextual modality and Visual modality.", "weaknesses": "1. The drawings are terrible!!! In particular, Figure 3. Incongruous text proportions and strange colour scheme...It's in the lower-middle range of T2I work. \n2. The method is a little weak. All-in-one methods have been far dicussed in the field of low-level and it's ripe for the picking. Compared with them, the ACE module is not that impressive.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tk23KP68k4", "reviewer": "Reviewer_bd3V", "summary": "This work proposes a unified visual generation and editing framework that supports a wide range of predefined tasks. To train and evaluate the proposed ACE, this work also introduces a data curation pipeline and an overall benchmark. Experimental results and numerous use cases demonstrate the superiority of the proposed method.", "questions": "1. What would happen if the Text Encoder T5 were replaced with an LLM? Would it be able to understand more diverse instructions?\n2. Will the collected data be made public?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The method provides a unified visual generation and editing framework that supports a wide range of predefined tasks.\n The benchmark is comprehensive, designed to evaluate visual generation and editing models effectively.", "weaknesses": "The paper lacks some ablation studies to help readers understand the authors' design choices. Additionally, the results in Table 2 may not be entirely fair, as the superiority of ACE might be attributed to the scale of the data.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7BxETnEyE3", "reviewer": "Reviewer_7X1X", "summary": "This paper introduces ACE (All-round Creator and Editor), a unified foundational model capable of handling a diverse array of visual generation tasks. By incorporating Long Contextual Units (LCU) and an efficient multimodal data collection methodology, ACE demonstrates exceptional performance in multi-task joint training, encompassing a wide range of tasks from text-guided generation to iterative image editing. Experimental results indicate that ACE significantly outperforms existing methods across multiple benchmark tests, showcasing its robust potential for practical applications.", "questions": "Discussion on Model Efficiency and Scalability: Could you provide more details on the model's performance across different scales of data? This would help in understanding its computational efficiency and scalability.\n\nIn-depth Analysis of Specific Tasks: For key tasks, could you offer a detailed comparison with state-of-the-art models specifically designed for those tasks? This would provide a clearer picture of the model's relative performance.\n\nEnhancing Model Interpretability: Could you explore the decision-making process of the model and provide an interpretability analysis of the generated results? This would help in understanding how the model arrives at its outputs.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "ACE introduces LCU, a novel approach that unifies various modal conditions, enabling the model to handle complex multimodal tasks. LCU allows ACE to flexibly adapt to different tasks, including generation and editing, which is lacking in current models. By integrating historical information into LCU, ACE can handle multi-turn editing tasks, enhancing its practicality in continuous interaction scenarios. ACE covers eight basic generation tasks and supports multi-turn and long-context tasks, establishing a comprehensive evaluation benchmark, significantly outperforming existing methods, especially in image editing tasks. User studies show that ACE is more in line with human perception. This paper not only makes significant contributions and proposes a practical and innovative solution but also excels in writing and figure drawing, with clear diagrams and rigorous logic, providing an excellent reading experience for the audience.", "weaknesses": "Model Efficiency and Scalability:\nThe paper should include a more detailed discussion on the computational efficiency and scalability of the model:\nIt is important to evaluate the model's performance when processing large-scale data to understand its practical applicability.\n\nIn-depth Analysis of Specific Tasks:\nThe paper should provide a thorough performance analysis for specific tasks.\nThis analysis should include comparisons with models that are specifically designed for those tasks, such as image editing models and inpainting models.\n\nData Annotation Quality:\nWhile MLLM-assisted annotation improves efficiency, the quality of automatic annotations may not always be on par with manual annotations.\nA quantitative analysis of the data annotation quality would enhance the credibility of the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)