ACE: All-round Creator and Editor Following Instructions via Diffusion Transformer

Authors: Zhen Han, Zeyinzi Jiang, Yulin Pan, Jingfeng Zhang, Chaojie Mao, Chen-Wei Xie, Yu Liu, Jingren Zhou
Primary area: generative models
Keywords: Image Generation and Editing, Diffusion Transformer, Instruction Following, Unified Framework

Abstract:
Diffusion models have emerged as a powerful generative technology and have been found to be applicable in various scenarios. Most existing foundational diffusion models are primarily designed for text-guided visual generation and do not support multi-modal conditions, which are essential for many visual editing tasks. This limitation prevents these foundational diffusion models from serving as a unified model in the field of visual generation, like GPT-4 in the natural language processing field. In this work, we propose ACE, an All-round Creator and Editor, which achieves comparable performance compared to those expert models in a wide range of visual generation tasks. To achieve this goal, we first introduce a unified condition format termed Long-context Condition Unit (LCU), and propose a novel Transformer-based diffusion model that uses LCU as input, aiming for joint training across various generation and editing tasks. Furthermore, we propose an efficient data collection approach to address the issue of the absence of available training data. It involves acquiring pairwise images with synthesis-based or clustering-based pipelines and supplying these pairs with accurate textual instructions by leveraging a fine-tuned multi-modal large language model. To comprehensively evaluate the performance of our model, we establish a benchmark of manually annotated pairs data across a variety of visual generation tasks. The extensive experimental results demonstrate the superiority of our model in visual generation fields. Thanks to the all-in-one capabilities of our model, we can easily build a multi-modal chat system that responds to any interactive request for image creation using a single model to serve as the backend, avoiding the cumbersome pipeline typically employed in visual agents.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 6, 6

Reviews and discussion:
<thread>
creation date:
1730373379636

thread_type:
official_review

<thread_object>
creator: reviewer_7x1x

<content>
summary:
This paper introduces ACE (All-round Creator and Editor), a unified foundational model capable of handling a diverse array of visual generation tasks. By incorporating Long Contextual Units (LCU) and an efficient multimodal data collection methodology, ACE demonstrates exceptional performance in multi-task joint training, encompassing a wide range of tasks from text-guided generation to iterative image editing. Experimental results indicate that ACE significantly outperforms existing methods across multiple benchmark tests, showcasing its robust potential for practical applications.

strengths:
ACE introduces LCU, a novel approach that unifies various modal conditions, enabling the model to handle complex multimodal tasks. LCU allows ACE to flexibly adapt to different tasks, including generation and editing, which is lacking in current models. By integrating historical information into LCU, ACE can handle multi-turn editing tasks, enhancing its practicality in continuous interaction scenarios. ACE covers eight basic generation tasks and supports multi-turn and long-context tasks, establishing a comprehensive evaluation benchmark, significantly outperforming existing methods, especially in image editing tasks. User studies show that ACE is more in line with human perception. This paper not only makes significant contributions and proposes a practical and innovative solution but also excels in writing and figure drawing, with clear diagrams and rigorous logic, providing an excellent reading experience for the audience.

weaknesses:
Model Efficiency and Scalability:
The paper should include a more detailed discussion on the computational efficiency and scalability of the model:
It is important to evaluate the model's performance when processing large-scale data to understand its practical applicability.

In-depth Analysis of Specific Tasks:
The paper should provide a thorough performance analysis for specific tasks.
This analysis should include comparisons with models that are specifically designed for those tasks, such as image editing models and inpainting models.

Data Annotation Quality:
While MLLM-assisted annotation improves efficiency, the quality of automatic annotations may not always be on par with manual annotations.
A quantitative analysis of the data annotation quality would enhance the credibility of the paper.

questions:
Discussion on Model Efficiency and Scalability: Could you provide more details on the model's performance across different scales of data? This would help in understanding its computational efficiency and scalability.

In-depth Analysis of Specific Tasks: For key tasks, could you offer a detailed comparison with state-of-the-art models specifically designed for those tasks? This would provide a clearer picture of the model's relative performance.

Enhancing Model Interpretability: Could you explore the decision-making process of the model and provide an interpretability analysis of the generated results? This would help in understanding how the model arrives at its outputs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730561256800

thread_type:
official_review

<thread_object>
creator: reviewer_bd3v

<content>
summary:
This work proposes a unified visual generation and editing framework that supports a wide range of predefined tasks. To train and evaluate the proposed ACE, this work also introduces a data curation pipeline and an overall benchmark. Experimental results and numerous use cases demonstrate the superiority of the proposed method.

strengths:
The method provides a unified visual generation and editing framework that supports a wide range of predefined tasks.
 The benchmark is comprehensive, designed to evaluate visual generation and editing models effectively.

weaknesses:
The paper lacks some ablation studies to help readers understand the authors' design choices. Additionally, the results in Table 2 may not be entirely fair, as the superiority of ACE might be attributed to the scale of the data.

questions:
1. What would happen if the Text Encoder T5 were replaced with an LLM? Would it be able to understand more diverse instructions?
2. Will the collected data be made public?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730711131431

thread_type:
official_review

<thread_object>
creator: reviewer_vndp

<content>
summary:
1. Propose ACE, a unified foundational model framework that supports a wide range of visualgeneration tasks, achieve a best task coverage.
2. Define the CU for unifying multi-modal inputs across different tasks and incorporate long context CU.
3. Design specific data construction pipelines for various tasks to enhance the quality and eff-ciency of data collection.
4. Establish a more comprehensive evaluation benchmark compared to previous ones, cover-ing the most known visual generation tasks. 

It's a lot of work, from method to data to data construction pipelines to benchmark, very systematic and complete work.
And all-in-one models are really interesting and is consistent with the general trend of generate model development.
But,
drawings are terrible, and the method is a little weak. Maybe you could use the all-in-one methods in low-level works as reference.

strengths:
1. Propose ACE, a unified foundational model framework that supports a wide range of visualgeneration tasks, achieve a best task coverage.
2. Define the CU for unifying multi-modal inputs across different tasks and incorporate long context CU.
3. Design specific data construction pipelines for various tasks to enhance the quality and eff-ciency of data collection.
4. Establish a more comprehensive evaluation benchmark compared to previous ones, cover-ing the most known visual generation tasks
5.Analyze and categorize these conditions from textual and visual modalities respectively, includeTextual modality and Visual modality.

weaknesses:
1. The drawings are terrible!!! In particular, Figure 3. Incongruous text proportions and strange colour scheme...It's in the lower-middle range of T2I work. 
2. The method is a little weak. All-in-one methods have been far dicussed in the field of low-level and it's ripe for the picking. Compared with them, the ACE module is not that impressive.

questions:
Please DRAW better.
I don't find the computing resource? I think it would be big, maybe you could have a discuss.
Related work? i think there should be other works that make try building an all-in-one visual generation model. Maybe you could list them clearly, I'm not an expert on this.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721580926

thread_type:
official_review

<thread_object>
creator: reviewer_vgpz

<content>
summary:
This work proposes an All-round Creator and Editor as a unified foundation model for visual generation tasks. The main technical contribution lies in introducing a Long-context Condition Unit that standardizes diverse input formats. Built upon diffusion transformers, the architecture incorporates condition tokenizing, image indicator embedding, and long-context attention blocks to achieve unified visual generation capabilities. To address the scarcity of training data, the authors develop a data collection pipeline that combines synthesis/clustering-based approaches. Additionally, they establish a comprehensive benchmark for evaluating model performance across various visual generation tasks.

strengths:
1) The framework unifies multiple image generation and editing tasks through a single model, avoiding the hassle of calling separate specialized models. The proposed LCU provides a structured approach to incorporating historical context in visual generation.

2) The paper presents systematic methodologies for data collection and instruction construction, which contributes to the development of all-in-one visual generative foundation models.

3) The evaluation benchmark provides comprehensive coverage across diverse image manipulation and generation tasks, enabling thorough performance assessment.

weaknesses:
Technical Issues:
1. Formatting inconsistencies: in lines 417-418, the image placement obscures instruction text.

2. The authors are encouraged to provide discussions on task-specific performance trade-offs during training, specifically how optimizing for one task might affect the performance of others.

3. It would be helpful to provide methodological details regarding parameters in data preparation (lines 321-325), such as cluster number determination and data cleaning criteria.

4. The qualitative results in Figure 5 reveal some limitations. 1) Row 1 (left): ACE generates a distorted hand. 2)  Row 2 (right) and Row 4 (left): The model exhibits undesired attribute modifications not specified in the instructions, including unintended gesture alterations / head rotation changes, and camera perspective shifts.

questions:
1. Regarding Figure 6, the authors are encouraged to elaborate on the empirical or theoretical basis for the chosen data distribution and its specific advantages for the ACE model.

2. The paper would benefit from addressing the practical challenges of model updates. Specifically, how might one efficiently incorporate new functionalities without complete model retraining? This consideration is crucial for the model's practical deployment and ongoing development.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733484180

thread_type:
official_review

<thread_object>
creator: reviewer_nxsb

<content>
summary:
The paper presents a method to train a unified model for 8 different tasks: Text-guided Generation, Low-level Visual Analysis, Controllable Generation, Semantic Editing, Element Editing, Repainting, Layer Editing and Reference Generation. The idea is intuitive. The main contribution of the paper is the framework for generating paired training data. The source of the data generation comes from two aspects: 1. synthetic generation and 2. from publicly available datasets (LAION-5B). To verify the results of this task, authors also create a new benchmark called ACE Benchmark.

strengths:
1. The dataset generated in this paper is beneficial to the community. This will help other researchers follow this series of research works.
2. A unified model for all tasks is also more efficient compared to have several individual models specific to certain type of tasks.

weaknesses:
1. It seems the author does not have clear discussions on how those tasks affect each other. Are they beneficial to each other? Or some of the tasks are reducing the performance of other tasks? How to select the most reasonable tasks that should be unified with the single model? I believe adding this type of discussion with corresponding experiments will make the paper more solid.

questions:
Indeed, as I mentioned in the weakness, how those tasks affect each other?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732201701019

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Dear all, 

We would like to express our gratitude to our reviewers for their valuable comments. For positive comments, 
- significant contribution (R-vGPZ, R-7X1X),
- unified & broad task support & efficient framework (R-NXSb, R-vGPZ, R-bd3V),
- systematic and complete work (R-vGPZ, R-vndP),
- comprehensive evaluation benchmark (R-vGPZ, R-bd3V, R-7X1X),
- well writing and drawing with excellent reading experience (R-7X1X)

we appreciate them and will carry them forward.

We would like to further clarify our goal and contributions, address the common concerns raised by the reviewers, and outline the revisions made to the manuscript in response to these comments.

**1. Goal**

We aim to create an all-in-one model that supports a wide range of visual generation and editing tasks, which we have named ACE: an All-round Creator and Editor. Currently, it covers eight basic types: Text-guided Generation, Low-level Visual Analysis, Controllable Generation, Semantic Editing, Element Editing, Repainting, Layer Editing, and Reference Generation. To achieve this, we define a universal LCU input paradigm, design specific data construction pipelines, and propose a comprehensive evaluation benchmark. We hope to continually expand task capabilities and improve generation quality, providing momentum for the development of the open-source community.

**2. Task Interdependence**

We add a discussion about this in the *"Supplementary Materials, Section IMPLEMENTATION DETAILS"*. Below, we briefly describe the issue:
In an all-in-one visual generation model, there exist multiple interactions between tasks, similar to that in large language models, and this relationship can be viewed as a complex balancing action. To achieve an optimal performance balance, we handle the data preparation and model training processes to ensure that the model can fully leverage the complementarity between different tasks while controlling for any potential negative impacts.

**3. Design choices**

We add an additional section in the *"Supplementary Material, Section ARCHITECTURE DESIGN"* to clarify our design considerations and provide relevant visual analyses, which, in brief, include the following modules:
Long-context Attention Block is specifically designed to handle input image sequences of varying lengths, addressing the limitations of the conventional attention block employed in DiT, which cannot effectively manage sequences of disparate lengths. 
Image Indicator Embeddings facilitate the alignment between images in the sequence and their corresponding mentions within the text prompt.

**4. More implementation details**

Based on the reviewers' comments, we have supplemented relevant content in the *"Supplementary Material, Section IMPLEMENTATION DETAILS"*, including a further description of the training process and adding the corresponding parts of data preprocessing, computational efficiency, checkpoints evaluation, and visualization of the editing process to address the reviewers' concerns.

For other concerns, we address them in the respective comments to the reviewers.


Thanks and best regards,

Authors of Submission 8711
</content>
</thread_object>

</thread>
<thread>
creation date:
1732776945924

thread_type:
official_comment

<thread_object>
creator: area_chair_kfbz

<content>
title:
Reviewer feedback and discussion

comment:
Dear Reviewers,

As the discussion period will end next week, please take some time to read the authors' rebuttal and provide feedback as soon as possible. For reviewers vndP, bd3V, and 7X1X, did the author address your concerns, and do you have further questions?

Thanks,

Area Chair
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes an all-in-one model that supports a wide range of visual generation and editing tasks. Reviewers recognize the contribution of the unified framework and extensive evaluations. Questions are raised regarding design choices, more analysis, and experiments. The authors addressed most of the concerns during the rebuttal, and all reviewers give positive scores. Therefore, the area chair recommends accepting the paper.