Crafting Layered Designs from Pixels

Authors: Jingye Chen, Zhaowen Wang, Nanxuan Zhao, LI ZHANG, Difan Liu, Jimei Yang, Qifeng Chen
Primary area: generative models
Keywords: Graphic Design, Layered Image Generation

Abstract:
Graphic designs play a vital role in communicating ideas, values, and messages.
During the design process, designers typically organize their work into layers of
text, objects, and backgrounds to facilitate easier editing and customization. However, creating design in such a format requires significant effort and expertise. On
the other hand, with the advancement of GenAI technologies, high quality graphic
designs created in pixel format have become more popular and accessible, while
with the inherent limitation of editability. Despite this limitation, we recognize
the significant reference value of these non-layered designs, as human designers
often derive inspiration from these images to determine layouts or text styles. Motivated by this observation, we propose Accordion, a graphic design generation
framework built around a vision language model playing distinct roles in three
key stages: (1) reference creation, (2) design planning, and (3) layer generation.
By using the reference image as global design guidance, distinct from existing
methods, our approach ensures that elements within the design are visually harmonious. Moreover, through this three-stage framework, Accordion can benefit
from an unlimited supply of AI-generated references. The stage-wise design of
our framework allows for flexible configuration and various applications, such as
starting from user provided references directly with the later two stages. Additionally, it leverages multiple vision experts such as SAM and element removal
models to facilitate the creation of editable graphic layers. Experimental results
show that Accordion generates favourable results on the DesignIntention benchmark, including tasks such as text-to-template, adding text on background, and
text de-rendering. Furthermore, we fully explore the potential of Accordion to
facilitate the creation of design variations, validating its versatility and flexibility
in the whole design workflow.

Decision: reject
Reviewer scores: 5, 5, 3

Reviews and discussion:
<thread>
creation date:
1730354495645

thread_type:
official_review

<thread_object>
creator: reviewer_vf8x

<content>
summary:
Based on existing Vision Language Model, the paper presented a layers-aware graphics design system, named Accordion. The system can be used for reference creation, design planning, and layer generation. The pipeline is comprised of three stages: reference creation, design planning, and layer generation. The experiments demonstrate applications of text to template, adding text to background, text de-rendering, and design variation creation.

strengths:
The system is comprised of various existing components, and seems to work well. It is able to ease the graphics design work.

The result images are delightful. I hope the system could be accessed publicly in the future.

weaknesses:
The innovation is limited. The work integrates various existing tools and large models, such as SAM, VLM, to compose the system. In the Section 3 (Methodology), there is no specific content describing the work, but there is only high-level summary. Hence, I cannot find enough specific academic contribution.

This is an engineering work. I consider it is not suitable for an ICLR research paper.

questions:
I'd like to see the runtime table for Stage 1, 2, 3 and various editing tasks, as waiting time highly has a big impact on an editing system.

Is the system adaptable to different resolution images. If given a high resolution image (>5M pixels), how about the running performance.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730445413980

thread_type:
official_review

<thread_object>
creator: reviewer_rr7v

<content>
summary:
This paper proposes some components for visual designs. It has stages for layer extraction, background processing, element planning, etc, to help the design generation.

strengths:
The problem and research direction seem solid and has many real-world applications. Some recent commercial products like ChatGPT Canvas also validate the importance of these research directions.

The paper has shown many results with different application types.

weaknesses:
It is not very clear which part is proposed by this paper: The Reference Creation is an existing Vision LLM. (This paper proposed a prompt that works well for it). The OCR, background removal, SAM, inpainting are all existing models. The Vision LLM is trained with an internal dataset Design39K, which seems the contribution of this work. But this contribution (presenting a finetuned VLM) seems relatively weak.

questions:
I think this paper may need a major revision to make the exposition much clearer. The authors may want to show at the very beginning that the intended use case of this application is mainly converting AI-generated design images into real design files with layers and texts, as a post-processing for image generators like Flux and SD.

Currently the first stage of this method is “reference creation”, making readers to think that the references are outputs. But the intended application is mainly using existing reference as inputs.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730513049404

thread_type:
official_review

<thread_object>
creator: reviewer_rplh

<content>
summary:
This paper presents a workflow for generating layered graphic designs using visual language models (VLMs). The workflow involves three stages from generating reference, design planning, and layers. The proposed workflow can facilitate creation of design variations.

strengths:
* the overall workflow successfully generate nice layered graphic design.
* the workflow is versatile for different design scenario
* the proposed method is straightforward as long as the training data is provided.

weaknesses:
* Although the results seem better than other compared method, it is not clear whether this is because the better image generative model. I recommend to have a fair comparison, e.g., applying the planning and layering method to the design image generated by other image generative models.
* the novelty of the method seems limited. I think the main contribution is the process of generating design planning before layering decomposition. But I think it lacks the comparison on combining different previous works with the proposed unified workflow.
* lack of experiment and results on applying the proposed method on graphic design in the wild.

questions:
* it is still very unclear to me why the method need to generate the reference image in the first place instead of using an existing deisgn?
I think it makes the overall proposed workflow like a system or a product, without a concentrated technical contribution. 
* I think the second and the third stages can be achieved by combining other existing works. Therefore, I want to ask is there any specific reason why those existing works for text de-rendering and layer decomposition or code generation from a given design cannot work?
</content>
</thread_object>

</thread>


Metareview: