Decomposition of Graphic Design with Unified Multimodal Model
Authors: Hui Nie
Zhao Zhang
Yutao Cheng
Maoke Yang
Gonglei Shi
Qingsong Xie
Jie Shao
Xinglong Wu
Venue: ICML 2025 poster (2025)
Keywords: Layer Decomposition
Graphic Design
Unified Multimodal Model
Primary area: applications->computer_vision

Abstract: We propose Layer Decomposition of Graphic Designs (LDGD), a novel vision task that converts composite graphic design (e.g., posters) into structured representations comprising ordered RGB-A layers and metadata. By transforming visual content into structured data, LDGD facilitates precise image editing and offers significant advantages for digital content creation, management, and reuse. This task presents two core challenges: (1) predicting the attribute information (metadata) of each layer, and (2) recovering the occluded regions within overlapping layers to enable high-fidelity image reconstruction. To address this, we present the Decompose Layer Model (DeaM), a large unified multimodal model that integrates a conjoined visual encoder, a language model, and a condition-aware RGB-A decoder. DeaM adopts a two-stage processing pipeline: first generates layer-specific metadata containing information such as spatial coordinates and quantized encodings, and then reconstructs pixel-accurate layer images using a condition-aware RGB-A decoder. Beyond full decomposition, the model supports interactive decomposition via textual or point-based prompts.  Extensive experiments  demonstrate the effectiveness of the proposed method. The code is accessed at https://github.com/witnessai/DeaM.

Decomposition of Graphic Design with Unified Multimodal Model
Hui Nie Zhao Zhang 22 Yutao Cheng? Maoke Yang"
Gonglei Shi? Qingsong Xie Jie Shao Xinglong Wu
Abstract
image editing, layers employed to isolate different com- ponents of an image, functioning analogously to transparent sheets that can be stacked to achieve various visual effects or construct new compositions. Understanding the under- lying layer structure of such images is essential for a wide range of applications, including content editing, material archiving, and image reconstruction. However, despite sig- nificant progress in computer vision and computer graphics, automatically decomposing composite image into its con- stituent layers remains a challenging and largely unsolved problem.
We propose Layer Decomposition of Graphic Designs (LDGD), a novel vision task that con- verts composite graphic design (e.g., posters) into structured representations comprising ordered RGB-A layers and metadata. By transforming vi- sual content into structured data, LDGD facilitates precise image editing and offers significant advan- tages for digital content creation, management, and reuse. This task presents two core challenges: (1) predicting the attribute information (metadata) of each layer, and (2) recovering the occluded regions within overlapping layers to enable high- fidelity image reconstruction. To address this, we present the Decompose Layer Model (DeaM), a large unified multimodal model that integrates a conjoined visual encoder, a language model, and a condition-awee RGB-A decoder. DeaM adopts two-stage processing pipeline: first generates layer-specific metadata containing information such as spatial coordinates and quantized encod- ings, and then reconstructs pixel-accurate layer images using condition-aware RGB-A decoder. Beyond full decomposition, the model supports in- teractive decomposition via textual or point-based prompts. Extensive experiments demonstrate the effectiveness of the proposed method. The code is accessed at http://iihubb.com/witeessse
This paper introduces Layer Decomposition of Graphic Designs (LDGD), as shown in Figure The goal is to decompose composite image into a set of semantically meaningful and individually discernible layers with well- defined order-akin to peeling back the layers of an onion. Each layer, whether it represents the background, primary imagery, atmospheric elements, text, or other visual com- ponents, is annotated with structured metadata, including attributes such as position, size, and text color. The primary challenge of this task lies in the intricate visual overlap and interdependence among layers, which makes single- purpose vision models such as standard image segmentation inadequate for achieving accurate and structured layer de- composition.
We introduce Decompose Layer Model (DeaM), purpose- built framework for this challenging task, inspired by the success of large unified multimodal models in understand- ing complex visual patterns and generating multimodal out- puts. The DeaM incorporates three critical components: conjoined visual encoder, comprehensive large language model, and a sophisticated condition-awar RGB-A decoder. The conjoined visual encoder is designed to perceive and integrate visual information across different semantic hierar- chies. The large language model then processes this visual information along with textual instructions execute layer decomposition tasks. Following this, the condition-aware RGB-A A decoder takes the tokens associated with the im- age layers derived from this decomposition, producing the relevant visual outputs. DeaM operates by producing meta- data for each dissected layer, represented as a JSON format. In Figure DeaM demonstrates the ability to forecast a wide variety of details for the textual layers, such as font
1. Introduction
In the era of digital media production, multi-layer composite images serve as a fundamental structure in visual design, enabling the conveyance of rich and complex information through the integration of visual elements and text. In digital
'Work done during internship at ByteDance. 'Equal Contribu- tion& & Project Lead. 'nniversity of Chinese Academy of Sciences. ByyteDacce Intelligent Creation, China. 'OPPO AI Center. Corre- spondence to: Zhao Zhang <zzhang @ www.alaaaaaiiiccommmmmm www.kii.ed..c....... Jie Shao <shaojie. mail @ bomm
Proceedings of the 42nd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

Decomposition of Graphic Design with Unified Multinodal Model
Points
Anything
Structure info. JSON
Layoul Show
coffee with chacolate beans
Mosttepoy" Ting.. Ten" 64 0 w: 1640, TM" 924 -....
default
Input Image
DeaM
Decomposed RGB-A layer
Here
Order Now!
Coffee Morning
Discount 20% OFF Coffee Momung bonda
Discount 20% OFF Everyday
Discount 20%
Discount 20% OFF
Figure 1. We introducc new task, Layer Decomposition of Graphic Designs (LDGD), which involves decomposing composite image into multiple layers (including the background, primary imagery, atmosphere graphics, text. etc.) in logical layer order. To address this task, we develop a decomposition model, DeaM, which supports both full-layer decomposition and selective layer extraction guided by diverse forms of input, including textual prompts and point-hased instructions.
style, color palette, textual content, alignment, and forth, thereby facilitating text layer reconstruction and secondary edits. For the image layers, DeaM predicts an additional field, capturing the visual tokens' indices essential for the decoder to regenerate RGB-A images.
2. Related Work
2.1. Layered Image Decomposition
Earlier work (Monnier et al., 2021; Sbai et al., 2018; Du et al., 2023) explored similar tasks of decomposing images into layers by learning object prototypes along with param- eters for occlusion and transformation, combining them to reconstruct complete images. While these studies have made notable progress, their problem formulations fall short of capturing the complexity and diversity inherent in real- world scenarios. To address these limitations, we extend their task definition and propose new formulation that is more aligned with practical applications.
In addition to fully automated layer decomposition, DeaM supports interactive prompts, including both text instruc- tions and point-based inputs, allowing users to selectively guide the decomposition of specific layers.
To support the LDGD task, we construct an in-house dataset named CreatiLD, comprising over 200,000 poster images annotated with rich information necessary for layer decom- position. To further enhance the model's capabilities, we leverage the linguistic proficiency of GPT-4 generate open-ended instruction-tune totioonntttinnggggo data, thereby broadening the model's 's understanding of diverse user queries.
Our task setting differs in several key aspects: (1) Our input images contain text, which is typically not regarded as an object. We explicitly decompose the text as separate visual layer. (2) Our output includes not only the image of each individual layer but also associated metadata for each layer.
Our contributions are multifold:
We formalize the novel and practical task, LDGD, which aims to decompose composite image into in- dividual and complete RGB-A layers with metadata, even if the layers are obstructed by each other.
Overall, our formulation represents unified understanding and generation task, making it inherently more challenging than previous approaches.
In addition, existing derendering methods (Ma et al., 2022: Rodriguez et al., 2025) typically convert images into SVG format. However, SVG representations struggle to capture complex visual details, and these methods are generally lim- ited to decomposing simple graphical content while lacking the ability to accurately decompose the text.
We present the DeaM, a large unified multimodal model that performs end-to-end decomposition of an image. DeaM also accommodates user interactions, allowing for the targeted decomposition of layers spec- ified through text or point-based prompts.
Our DeaM achieves consistently superior results com- pared to the baseline and a variety of existing ap- proaches, highlighting its effectiveness in the challeng- ing task of layer decomposition.
2.2. Large Multimodal Models
Over the recent year, there has been a notable surge in in- terest surrounding large language models (LLMs) (Devlin
2

---

Decomposition of Graphic Design with Unified Multinodal Model
alal, 2019; Radford et al., 2019; Raffel et al., 2020; Ouyang et al., 2022; Zhang al., 2022; Zeng al., 2023; Chiang et al., 2023) in the realms of natural language processing and computer vision. This surge is attributable to the supe- rior capabilities of LLMss, which have demonstrated excel- lence in multifaceted applications, notably their comprehen- sively detailed global knowledge base and multifunctional utilities. Large multimodal models (LMMs) (Team al., 2023; Zhang et al., 2024a; OpenAI, 2023; Wang et al., 2023; Alayrac et al., 2022; Li et al., 2025; Gong et al., 2023; Huang et al., 2023; Dai et al., 2023; Zhu et al., 2024; Driess et al., 2023; Ye et al., 2023; Bai et al., 2023; Wang et al., 2024a) integrate the powerful linguistic capabilities of LLMs and extends its multimodal processing abilities, breaking down barriers between visual and linguistic modalities, and even more beyond. Two notable contributions in vision-language learning are BLIP-2 (Li et al., 2023) and LLaVA (Liu et al., 2023). BLIP-2 integrates a frozen image encoder with large language models using a lightweight Q-Former for efficient cross-modal alignment. LLaVA pioneers the use of GPT- 4 (OpenAI, 2023) to generate instruction tuning data for multimodal tasks. In multimodal localization, Shikra (Chen et al., 2023) excels at spatial reasoning, while Graphist (Cheng et al., 2025) extends localization to poster layout de- sign, showcasing strong structural understanding. Recently, emerging unified multimodal models (UMM) (Zheng et al., 2023; Zhan et al., 2024; Sun et al., 2024; Zhou et al., 2025; Xie et al., 2025; Wu et al., 2024c; Wang et al., 2024b; Fang et al., 2024; Wu et al., 2024a; Ma et al., 2024; Qu et al., 2024; Wu et al., 2024b; Li et al., 2024; Shi et al., 2024: Tong et al., 2024) have demonstrated the ability jointly generate text and images, enabling unified multimodal gen- eration. Inspired by these advances, we explore leveraging a UMM for layer decomposition of graphic designs, aiming to simultaneously generate textual elements and multiple visual layers.
model, while keeping the unmasked areas the same in each denoising step. Another kinds of inpainting methods (Xie et al., 2023a;b; Yang et al., 2023a; Yu et al., 2023) fine-tune a specially designed image impainting model to integrate corrupted image and mask. However, these methods cannot decompose multiple layers simultaneously and struggle to handle cases where layers occlude each other.
3. Task Formulation
Given a composite image x E X' that includes various image elements (such as the background, primary imagery, decorations, and text), the goal of Layer Decomposition of Graphie Designs (LDGD) is to decompose the image into an ordered series of RGB-A layers I = {T, E T xwxx4) Jisi with their metadata (such as position, text color and so on) from the original image. We hope that each layer is as close as possible to the corresponding part in the original image and that re-rendering these layers in accordance with their metadata will yield the original picture. For the sake of simplicity, we treat text elements as RGB-A images, because once our model recognizes the text, color, font, and size, we can render them into RGB-A images. The position information for each layer contains five numerical elements (Ii; Vs, Wi, hús 4:). where x; and y: denote the coordinates of the upper left cornrr of the layer in relation to the original image, wi; and hú represent the width and height of the layer within the original image, and L; indicates the layer order. If the ordering of the layers is illogical, then re-rendering the image using the obtained layers would result in implausible occlusions (Cheng et al., 2025).
Ji=l
4. Baseline
Layer decomposition of graphic designs is a novel and un- derexplored task for which no existing methods provide a satisfactory solution. For example, traditional image seg- mentation can separate foreground objects but fails to re- cover occluded background content, and often treats text as non-editable image regions. Similarly, image inpainting is typically limited to generating a single-layered output, such as simply erasing text, without structural decomposition.
2.3. Image Editing
Although contemporary studies have not yet concentrated on the concurrent decomposing of multiple image layers, a subset of prevalent image generation and editing techniques demonstrates partial capability in isolating and decompos- ing individual layers. Instruction-basee image editing: In- structPix2Pix (Brooks et al., 2023), HIVE (Zhang et al., 2024b), MGIE (Fu et al., 2024), SmartEdit (Huang et al., 2024), and SEED-X (Ge et al., 2024) can sometimes achieve layer decomposition effects by facilitating editing of des- ignated layers. However, these methods are unable well decompose multiple elements of an image. Several inpaiting methods (Liu et al., 2024; Lugmayr al., 2022; Yang et al., 2023b; Corneanu et al., 2024) can modify the specified re- gions in image with extra mask condition. These methods only sample the masked regions from a pre-trained diffusion
To create basic approach to this task, we develop a baseline combining image inpainting. matting, and OCR. We first extract text data, font attributes, and bounding boxes using OCR, then form mask for inpainting. Image matting separates the primary imagery, whose mask aids in further inpainting to restore the background via the Volcano Engine API. This method can only decompose the input image into three layers: background, primary imagery, and text. It cannot decompose the image into more layers.
3

---

Decomposition of Graphic Design with Unified Multinodal Model
RGB-A VQ-GAN
Phase Decoder Training
Text layer
Di MASCAE DESIGNS
Image layer
''categor": 'img".
"len" 144,
Alpha
RGB
204,
204,
"category": "text"
Y: 15,
'y: 15,
"Tont":
W: 852,
Condition-Aware RGB-A Decoder
VQ-Decoder
WW: 652,
"content:: "DRE...
"h": 223
"h"" 223,
"color": (253,255,25))
"quant" >cvt-01022>
"size": 12.
"align": "leff").
<vb- 2v8-15892>
Codebook
7.
<vt-00000>
00000000000000000000000000000000
quanthidd
<vt-00001>
Large Language Model (LLM)
2v5-24575>
continuuus
Conjoined Visual Encoder
Instructions
VQ-Encoder
Layer Decomposition (LD):
"Please decompose the input."
BEEWICAPE
Alpha
Text-Guided LD:
"Decompose layers matching <descriptions"
RGB
Point-Guided LD:
"Decompose layers in <x1, y1>, ... <xn, yn>"
Phese 1: Train VQ-GAN for
quantizing encoded image layers
Phase 2: Instruction Tuning
Figure 2. DeaM Pipeline. The training process of DeaM consists of three phases: (1) VQ-GAN Training: Train an RGB-A VQ-GAN, then use to encode all image layers with "quant" annotations. (2) Instruction Tuning: Train DeaM to generate detailed image descriptions (metadata), performing layer decomposition tasks including element positions, layor order, indices in the VQ-GAN codebook, and so on. (3) Decoder Training: Train a condition-aweer aware RGB-A decoder using ResNet as the condition encoder, keeping the VQ encoder frozen. This phase is decoupled from the instruction tuning.
5. Proposed DeaM
the case in reality. The RGB information is actually more important. To eliminate the model's encoding ambiguity, we adjusted the VQ-GAN reconstruction loss by applying different weights to the RGB and the alpha channel. The training data of RGB-A VQ-GAN includes image layers from all poster images, the ImageNet training set (Deng et al., 2009), and a subset of several million images from LAION (Schuhmann et al., 2021; 2022).
As shown in Figure Decompose Layer Model (DeaM) comprises three key components: conjoined visual en- coder, a large language model backbone, and condition- aware RGB-A decoder. The conjoined visual encoder is capable of encoding semantically multi-layered visual infor- mation. The large language model processes the input visual and textual instruction information and generates layer de- composition results. The condition-awar RGB-A decoder then decodes the tokens corresponding the image layers in the decomposition results to produce the respective images.
5.2. Conjoined Visual Encoder.
Visual images contain visual elements with different levels of semantic information, such as object categories with pro- nounced semantic information (e.g., people, watches) and decorative elements with less semantic information (e.g.. geometric element patterns). Existing LMMs (Liu et al., 2023) typically employ CLIP as the sole visual encoder, encoding generally higher-level semantic information. To enable the model focus on both high-level semantic ob- ject categories and lower-level visual elements, design conjoined visual encoder. And we utilize two different types of visual encoders: the CLIP Vision Encoder (Radford et al., 2021), and DINO v2 (Oquab et al., 2024). After encoding, we concatenate the visual features along the channel dimen-
5.1. RGB-A VQ-GAN.
We modify the RGB VQ-GAN into an RGB-A VQ-GAN by changing the number of channels in the convolutional ker- nels of the first and last layers from 3 to 4 to accommodate the alpha channel. When using the default training strategy (Esser et al., 2021), there is ambiguity in the model's encod- ing of RGB-A images. For instance, the model produces the same encoding for some black lines that are similar but not identical. We found that this issue arises because the model treats the reconstruction of the alpha channel and the RGB channels as equally important during training, which is not

---

Decomposition of Graphic Design with Unified Multinodal Model
sion, thereby obtaining visual features with dimensions of
correct number of indices, these index sequences cannot be reshaped into two-dimensional matrix (12 12 or 8 8), and thus, we would be unable to decode and generate images using the decoder. To address this issue, when constructing training image token data for the layer, after encoding with VQ encoder to get a two-dimensional matrix of indices, we append a learnable newline token ('\n²) at the end of each row of indices before flattening the sequence. This approach introduces more regularity to the sequence, thereby reducing the prediction difficulty for the unified multimodal model.
N (1024 2). Both visual encoders here use an input resolution of 336 x 336 to ensure consistent number of tokens after encoding.
5.3. Large Language Model Backbone.
Vocabulary Expansion. In the architectural design of our model, a crucial component involves the utilization of semantic tokens transformed by the VQ-GAN model (Esser et al., 2021). This architecture integrates an encoder and a decoder, augmented by a specialized quantization layer whose role is to translate image information into a sequence of tokens predefined in a codebook. Both the encoder and the decoder adopt convolutional layer structures to effec- tively process images of varying resolutions. Specifically, the encoder compresses the spatial dimensions of the in- put image through a series of downsampling operations, while the decoder reconstructs the image back to its orig- inal size via matching upsampling operations. To purely represent the image information of each layer and avoid the generated layer images containing meaningless background information, the model training utilizes images with four channels, namely RGB-A. That is, areas not associated with this layer will be transparent. This RGB-A image format (Zhang & Agrawala, 2024) can support the re-rendering of decomposed layers and facilitate range of subsequent applications such as more sophisticated image editing and creation. We trained the VQ-GAN model with a downsam- pling ratio of f = 16. Higher image resolutions lead to clearer reconstructions, but they also make the training se- quences of the model much longer and significantly increase computational cost. Given the limitations on the length of the model's output tokens, when VQ-GAN generates tokens for each image layer, we set the input resolution for semanti- cally rich natural images to 192 192 and for semantically sparse decorative elements to 128 128. In the end, we obtain either 144 or 64 tokens. In our method, the VQ-GAN encoder is employed to encode the image layers (i.e., not text layers), and the obtained codebook IDs serve as the ground truth for training. Subsequently, we incorporate an equivalent number of special tokens like '<vt- <ett-numberr)>> > to the LLM,, reflecting the size of the VQ-GAN's code- book. The number represents the index of the VQ-GAN codebook embedding. In this work, we use Inter (Team, 2023) as the LLM backbone.
5.4. Condition-Aware RGB-A Decoder
After obtaining the output of the decomposing results, for image layers containing the image tokens '<vt- [number]>* we reshape them into a 12 x 12 or 8 x 8 wwo-dimensional array based on the length of the image to- kens. We can retrieve the corresponding index's 's VQ-GAN codebook embedding and use the VQ decoder to decode and restore them back into images. However, we found that directly using the pretrained VQ decoder to decode the gen- erated images results in less clarity. To further enhance the quality of decoded images, we designed a Condition-Aware RGB-A Decoder, as shown in Figure 3. Motivated by the observation that the input images are of high clarity, but not utilized during decoding, we aim for the decoder to lever- age the input images as conditional information to enhance image generation quality as much as possible. The decoder takes two types of inputs: one is the original layers, and the other is the corresponding regions of the input images asso- ciated with these layers. We noted that the unobscured parts of the layer images are relatively easy to restore, while the obscured parts are more challenging. To focus the model's 's attention on the obscured regions, we force the model to learn the mask of the obscured areas. Simultaneously, to reduce the impact of unnecessary foreground information in the occluded regions on the final reconstruction quality, we multiply the predicted mask with the features of the con- ditional information, thereby erasing the information of the occluded areas. Finally, these modified data are concate- nated with the latent features to serve as the input for the next part of the network. The training loss for the decoder is the same as that for VQ-GAN((ssser et al., 2021). During inference, we use the features indexed by the codebook from the special token predicted by the unified multimodal model as the features encoded by the VQ-Encoder, combined with the images of the crop areas corresponding to the modified layer as input.
Enhancing Prediction Regularity. We have discovered that predicting the output image VQ-GAN codebook index '<vt-(number)>> can be lengthy (for instance, semantically rich images or graphic elements may have 144 tokens, while semantically sparse decorative elements might have only 64). This poses a substantial challenge for unified multi- modal models, which struggle to accurately infer the precise number of indices in a single forward pass. Without the
5.5. Training Strategy
The training process of DeaM is divided into three phases: VQ-GAN training, instruction tuning, and decoder training. In the first phase, we train an RGB-A VQ-GAN. After the training is complete, use this VQ-GAN to encode all im-
5

---

Decomposition of Graphic Design with Unified Multinodal Model
Condition Eno.
Occlusion Mask
Crap
Poster
VQ-Encoder
Codebook
Input
Reconstruction
RGB-A
condinuuus quantiffed
Condition-Aware RGB-A Decoder
Figure Condition-Aware RGB-A Decoder. During training, the decoder utilizes VQ-GAN decoder for initialization accepts images corresponding the poster areas of the layers as conditional information to further enhance the image generation quality. To ensure decoder concentrates on generating occluded regions, model compelled to predict the mask the occluded areas. During inference, the features encoded by the VQ-Encoder replaced with the quantized encodings predicted by the DeaM.
age layers, adding annotations for the "quant" field. In the second phase, focus training DeaM to generate de- tailed descriptions of images (metadata), that is, to perform the layer decomposition task, which includes the positions, layer order, corresponding indices of image elements in the VQ-GAN codebook and so on. By providing the model with specific task instructions and images, DeaM learns to au- toregressively generate the JSON structure information for each layer, encompassing all the details necessary to render the different image elements. We also include text-guided, point-guided and GPT-4 generated instruction cettuctioo-tuning data for training. In the third phase, we train a condition-awand aware RGB-A decoder. Here, we use ResNet as the condition encoder and keep the VQ encoder frozen during training, only training the decoder and condition encoder. Addition- ally, the training of the decoder module is decoupled from the training of the unified multimodal model. We use 16 NVIDIA A800 GPUs for training.
color statistics are extracted. For text layers, we apply Opti- cal Character Recognition (OCR) to extract textual content, use a font classification model to identify typefaces, and compute font sizes based on the layer coordinates. During training, each input consists of a single-layer poster image, and the model is trained to output detailed metadata and a structured representation for each corresponding layer.
6.2. Instruction Data
The construction of instruction data aimed to enable the DeaM to adhere to user instructions and accurately decom- pose one or more layers in alignment with user requirements. We contemplate how to design user instructions with greater freedom support broader range of application scenarios. We crafted two types of instruction-tuning data with consid- erable flexibility, and in addition, we employed GPT-4 to collect a large corpus of open instruction-tuning data.
Text-Guided Instruction-Tunigg Data. We curated a set of instruction data for each layer endowed with cap- tion. Adhering to the specified format, each instruction read: "Please decompose the layer that manifests follows: [caption].". The '[caption]' was replaced with descrip- tion pertinent to a specific layer, which the model was then tasked to decompose. Ultimately, we collected 200,000 annotated data entries.
6. Training Datasets
6.1. Human-Annotated Data
curate large-scale in-house dataset of over 200,000 multi-layer structured poster images, containing layer se- quences, category labels, and coordinate information. This dataset, named CreatiLD, is collected from the internet and comprises 224.054 graphic designs with complete layer- level annotations. The majority of samples are posters cov- ering domains such as holiday events, retail, dining, and corporate communications. On average, each sample con- tains 10.30 layers, with an approximate image-to-text ratio of 6.3:3.7. To support the LDGD task, enrich the dataset with additional annotations. For image layers, captioning model is employed to generate descriptive captions, and
Point-Guided Instruction Eestructoo-Tuunng Data. To facilitate wider range of interactive modalities, have also gathered instruction-tuning data based point interactions. Users imput points at various locations on an image, and the model then interprets which layer corresponds to each specified point. Expanding upon this, our approach allows for a more intuitive user experience, enabling individuals to en- gage with the visual content directly and receive immediate.
6

---

Decomposition of Graphic Design with Unified Multinodal Model
context-aware feedback from the model. In the end, we collected 100,000 annotated data entries.
ing all the layers would exhibit a high degree of similarity to the original image. As for the prediction accuracy of bound- ing boxes and the accuracy of layer order prediction, we draw on the Hungarian matching algorithm used in object detection (Carion et al., 2020) to view the problem as one of predicting the accuracy between two sets of bounding boxes. We denote the ground truth set of objects as b, and the collection of N predictions b5 {0+, Di. W0, h.]. In order to establish bipartite matching between these two groups, we look for a permutation of a E Px items that yield the minimal cost:
GPT4-Generated Data. Drawing inspiration from the LLaVA (Liu et al., 2023) and Shikra (Chen et al., 2023) models, we employed the GPT-4 model to generate a more versatile collection of instructions, catered to address open- ended user inquiries. By inputting the JSON format infor- mation (including information on the types and positions of each layer) of posters into GPT-4, the model autonomously devised five question-answer pairs relevant to each poster's s content. Model-generated questions were constrained to those answerable with the metadata, ensuring the identifi- cation of related layers was possible. We collected 15,000 question-answer -nnswer pairs through GPT-4 for the fine-tuning of the model.
â arg min aEPx
Σ Looz (b. bo(t));
(1)
where we define Ltor as Liou ba(t)): Liou is loU loss (Zhou al., 2019) in this work. We use the mean IoU (Intersection over Union) of the matched bounding boxes as metrie Loc,, to measure the accuracy of the bounding box prediction. Considering that there might be various ways to decompose some decorative elements in an image, such as two lines that could be decomposed either as two separate layers or as a single layer, it inconvenient to perform box calculations. Therefore, we only focus on the box detection capabilities of the core elements (such as primary imagery).
7. Experiments
7.1. Test Datasets
To facilitate more open and transparent comparison with other methods, we utilize a publicly available academic dataset Crello for evaluation.
Crello (Yamaguchi, 2021) is now referred to VistaCreate?, provides collection of visual designs originating from online design tool. This comprehensive compilation encompasses a wide variety of design types, tailored for multiple uses such as infographics for social platforms, digital advertising banners, headers for blogs, and templates for print posters. Within this dataset, each individual design is accompanied by intricate details specify- ing the order of layers, the precise location of each element within the space, and the classifications of the various de- sign components. The test set of this dataset contains over 2,000 images.
Crello dataset':
Con. Vis. Enc.
Enh. Pre. Reg.
Con. Dec.
FID
105.524
101.132
95.488
70.629
Table 1. Ablation Study. Here, Con. Vis. Enc. denotes the conjoined visual encoder, Enh. Pre. Reg. stands for enhancing prediction regularity, and Con. Dec. refers to the condition-aware RGB-A encoder.
7.2. Evaluation Metrics
method
FID
Loca 1
Baseline
99.603
0.7069
In order to systematically evaluate the effectiveness of layer decomposition, we have designed comprehensive set of evaluation metrics. We focus primarily on two aspects: (1) the quality of the image reconstruction, (2) the prediction accuracy of the layer position (that is, the planar position of the layer on the original image and the order of the layers). For assessing the quality of layer reconstruction, we refer to the evaluation methods used for image reconstruction qual- ity. We primarily utilize the image reconstruction quality metric FID (Fréchet Inception Distance). Motivated by the premise that if the method excels in decomposing all layers with high fidelity, the composite image generated by merg-
DeaM(ours)
70.629
0.7 128
Table 2. Comparison with the baseline.
7.3. Quantitative Results
Ablation Study. We perform ablation studies on key com- ponents and strategies of our model design, with results summarized in Table 1. Using image reconstruction qual- ity as the evaluation metric, we observe consistent perfor- mance gains contributed by each component. Notably, the condition- RGB-A decoder yields the most substantial improvement, primarily by significantly enhancing image clarity. Additionally, the conjoined visual encoder strength- ens the model's 's detection capability and reduces missed detections, further boosting reconstruction quality.
'nttpe://hugginggtte cyberagent/crello 'nttpe://ereate, vista.com. com/
7

---

Decomposition of Graphic Design with Unified Multinodal Model
Input
Reconstruction
Layers
Christmas OFFERS CLABSEAND SELL
Christmuns
OFFERS ATTEARSINDANG
Pizza delivery
Pizza deliverv Now
nco
deliverv
Figure 4. Layer decomposition results of DeaM on the Crello datasct.
Comparison of Baseline. As shown in Table 2, in terms of image reconstruction quality, our method significantly outperforms the baseline. Moreover, unlike the baseline which only parses images into three layers (background, main subject, and text), our method can parse to more layers, enhancing its capability.
cating that the reconstructed image is close the original image in terms of image quality. For each row in the figure, the leftmost image is the input, the middle column shows the reconstructed image based on the predicted layer order, and the rightmost section displays the decomposed layers of the input. These layers are arranged from left to right in hierarchical order, where layers further to the left represent lower levels in the visual hierarchy, with the leftmost layer typically corresponding the background. From these fig- ures, it can be seen that the bounding box predictions for each layer are quite accurate, and the text content is largely correct, although the accuracy of the font prediction not exemplary. The example at the top of the Figure 4 is quite interesting: when there are many decorative elements and their spatial relationships lack clear hierarchical order, the model tends predict them being on the same layer. Moreover, there is still room for improvement in our image decoding and generation quality, which calls for a more powerful image tokenizer.
GT
Baseline
DeaM(Ours)
9opp connd hps rish andl compat vo
Ston,counde countnr risk nnd ceunt en us
SApcocssi tareria And
Order
Ordee
CYBER MONDAY
CYBER MONDAY
Comparison of Baseline. As illustrated in Figure 5, we compare the reconstruction quality between the baseline and our proposed DeaM. DeaM exhibits strong performance in reconstructing text regions, effectively recovering key de- tails such as content, font, size, and color in most cases, In contrast to the baseline's 's impainting-based approach, which often introduces artifacts or unnatural patterns, DeaM sub- stantially reduces these issues, resulting in cleaner and more faithful reconstructions.
HONDH
Figure 5. Comparison with the reconstruction results of the base-
line.
Text-Guided and Point-Guided Layer Decomposition. As shown in Figure 6, we showcase the capabilities of our model through the lens of point-guided and text-guided layer decomposition, revealing its versatility and robust under-
7.4. Qualitative Results
Layer Decomposition. We present the results of layer decomposition and composition shown in Figure 4, indi-
8

---

Decomposition of Graphic Design with Unified Multinodal Model
in differentiating between various layer types. It does merely recognize the presence of text imagery; it under- stands their distinct roles within composite image. This distinction is crucial for applications that demand selective editing or targeted adjustments within an image. The il- lustrative examples provided, particularly the image in the third row, not only validate the model's precision in pin- pointing locations denoted by user-specified points but also highlight its predictive prowess in envisioning the result- ing image once certain layers are omitted. This is evident in the model's is ability to infer what the image would look like without the textual elements that overlay the picture, a task that requires a nuanced understanding of both spatial relationships and the hierarchy of visual elements.
Sprring Clean
MUTTONEE AM ROA NEVEAL
DILLE
Too: Qulodd (Duam)
Aod
Clean Boot Tocke
BEGRETS
Opanaer Supers wook Teer
Port stee DDaak
10FM
Comparison with Image Vectorization Method. We com- pare our approach with image vectorization method, LIVE (Ma et al., 2022), which generates compact SVG representations featuring layer-wise structures that align se- mantically with human perception. As shown in Figure 7, LIVE tends to perform well on simple logo images but struggles with more complex natural images.
Figure 6. Results of point-guided and text-guided layer decompo- sition. For each column, left image represents the input picture, and the right image shows the results decomposed by the model.
Input
LIVE
DeaM(Ours)
8. Conclusion
In summary, we introduce the novel task of layer decom- position of graphic design, and present DeaM, dedicated framework that achieves significant progress in automatic layer decomposition, fine-grained image understanding, and manipulation. Leveraging a novel RGB-A VQ-GAN, DeaM effectively decomposes composite images into distinct lay- ers, substantially improving both encoding and decoding efficiency-cruciall for high-fidelity image editing. The model's 's decomposing capability is further enhanced by the integration of multiple carefully designed components and strategies. DeaM also supports a more user- friendly inter- action paradigm, allowing control via both point-basee and text-based instructions. In addition, we construct new dataset specifically for the layer decomposition task, which facilitates future research in this area. Extensive experi- ments demonstrate the effectiveness of DeaM, highlighting its potential to inspire a wide range of computer vision applications.
Valeatino's Day
Vnentirss Day
ASIAN FOOD TASTY AND FRESH
ASIAN FOOD
BOOK
BOOK
Figure 7. Comparison with LIVE. The first column shows the input images, while the second and third columns present the reconstruc- tion results from LIVE and our method DeaM, respectively.
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here.
standing of complex image structures. The first row presents the inference results of the instruction-based image editing method MGIE. In the second row, corresponding textual prompts for text-guided layer decomposition are shown be- neath each image. The third row visualizes the prompt points used for point-guided layer decomposition on the input images, indicated by red dots. The text-guided layer decomposition outcomes illustrate the model's adeptness
References
Alayrac J.-B., Donahue, J., Luc, P., Micch, A., Barr, I., Hasson,
Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et

---

Decomposition of Graphic Design with Unified Multinodal Model
Flamingo: visual language model for few-shot learning. Ad- vances in Neural Information Processing Systems (NeurIPS), 35:23716 267366 2022.
Esser, P., Rombach, R., and Ommer, B. Taming transformers for high- high-resolution image synthesis. In Proceedings of IEEE Con- ference on Computer Vision and Pattern Recognition (CVPR), Pp. 12873-12883 2021.
Bai, J., Bai, S., Yang. S., Wang. S., Tan, S., Wang, P. Lin, J., Zhou, C., and Zhou, J. Qwen-vl: A frontier large model with versatile abilities. arXiv preprint arXiv:2308. 2023.
Fang, R.. Duan, C., Wang, K., Li. H., Tian, H., Zeng. X., Zhao, R.. Dai, J.. Li, H., and Liu, X. Puma: Empowering unified milm with multi-granular visual generation. arXiv preprint arXiv:2410. 13861, 2024.
Brooks, T., Holymski, A., and Efros, A. A. Instructpix2pix: Learn- ing to follow image editing instructions. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 18392-18402, 2023.
Fu, T., Hu, W., Du, X., Wang, W. Y., Yang. Y., and Gan, Z. Guiding instruction- image editing via multimodal large language models. In International Conference on Learning Representa- tions (ICLR), 2024.
Carion, N., Massa, F., Synnaeve, G., Usunier, N., Kirillov, A., and Zagoruyko, S. End-to-end object detection with transformers. In Proceedings of European Conference on Computer Vision (ECCV). pp. 213-229, 2020.
Ge, Y., Zhao, S., Zhu, J., Ge, Y., Yi, K., Song, L., Li, C., Ding, X., and Shan, Y. Seed-x: Multimodal models with unified multi- gualii-grauulartyy comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024.
Chen, K., Zhang, Z., Zeng, W. Zhang, R., Zhu, F., and Zhao, R. Shikra: Unleashing multimodal I1m's 's referential dialogue magic. arXiv preprint arXiv:2306. 15195, 2023.
Gong. T., Lyu, C., Zhang, S., Wang. Y.. Zheng. M., Zhao, Q. Liu, K., Zhang. W., Luo. P.. and Chen. K. Multimodal-gpt A vision and language model for dialogue with humans. arXiv preprint arXiv:2335 200.004994
Cheng, Y., Zhang, Z, Yang. M., Nie, H., Li, C., Wu, X., and Shao, J. Graphic design with large multimodal model. In Proceedings of AAAI Conference on Artificial Intelligence (AAAI), pp. 2473- 2481, 2025.
047904 2023.
Huang, S., Dong, L.,, Wang, W., Hao, Y., Singhal, S., Ma, S., Lv, T., Cui, L., Mohammed, O. K., Patra, B., Liu, Q.. Aggarwal, K., Chi, Z.. Bjorck, N. J. B., Chaudhary, V., Som, S., Song, X., and Wei, F. Language is not all you need: Aligning perception with language models. In Advances in Neural Information Processing Systems (NeurlPS), 2023.
Chiang, W.-L.,, Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, E., et al. Vicuna: An open-souree chatbot impressing gpt-4 with 90%* chatgpt quality. See htlps://vicuaa. Imsys. org (accessed 14 April 2023), 2(3):6, 2023.
Huang. Y., Xie, L., Wang. X.., Yuan, Z., Cun, X.,, Ge, Y., Zhou, J.. Dong. C., Huang, R., Zhang, R., and Shan, Y. Smartedit: Exploring complex instruction based image editing with multi- modal large language models. In Proceedings of IEEE Confer- ence Computer Vision and Pattern Recognition (CVPR). pp. 8362-8371, 2024.
Corneauu, C., Gadde, R., and Martinez, A. M. Latentpaint: Image inpainting in latent space with diffusion models. In Proceddings of the IEEE Winter Conference on Applications of Computer Vision (WACV), pp. 4334-4343, 2024.
Dai, W., J., Li, D., Tiong, A. M. H., Zhao, J., Wang, W., Li, B., Fung, P., and Hoi, S. C. H. Instructblip: Towards general- purpose lisionlangguag models with instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS). 2023.
Li. B., Zhang, Y., Chen, L.,, Wang, J., Pu, F., Cahyono, J. A., Yang. J., Li, C., and Liu, Z. Otter: A multi-modal model with in-context instruction tuning. IEEE Transactions on Pattern Analysis and Machine Intelligence (T-PAMI), 2025.
Li, H., Tian, C., Shao, J., Zhu, X., Wang, 7., Zhu, J., Dou, W., Wang, Li, H., Lai, I.., et al. Synergen-vi: Towards synergistic image understanding and generation with vision experts and token folding. arXiv preprint arXiv:2412.0966 2024.
Deng, J., Dong, W., Socher, R., Li, L.-J., Li, K., and Fei-Fei, L. Imagenet: A large-scale hierarchical image database. In Pro- ceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248 228--2552222 2009.
Li, J., Li. D., Savarese, S., and Hoi. S. Blip-2: Bootstrapping language -igge-imae pre- tre-training with frozen image encoders and large language models. In Proceedings of International Confer- ence Machine Learning (ICMI.) pp. 19730-19742. 2023.
Devlin, J., Chang, M., Lee, K., and Toutanova, K. BERT: pre- training of deep bidirectional transformers for language under- standing. In Annual Conference of the Nations of the Americus Chapter of the Association for Computational Linguistics, pp. 1171-4186, 2019.
Liu, A., Niepert, M., and den Broeck, G. V. Image inpainting via tractable steering of diffusion models. In International Conference on Learning Representations (ICLR), 2024.
Driess, D., Xia, F. Sajjadi, M. S. M., Lynch, C., Chowdhery, A., Tchter, B., Wahid, A., Tompson, J., Vuong. Q., Yu, T., Huang, W. Chehotar, Y., Sermanet, P., Duckworth, D., Levine, S., Vanhoucke, V., Hausman, K., Toussaint, M., Greff, K., Zeng, A., Mordatch, I., and Florence, P. Palm-e: An embodied multimodal language model. In P Proceedings of International Conference on Machine Learning (ICML), volume 202. pp. 8469-8484 8488, 2023.
Liu, H., Li, Wa, Q., and Lee, Y. Visual instruction tuning. In Advances in Neural Information Processing Systems (NeurIPS), 2023.
Lugmayr, A., Danelljan, M., Romero, A., Yu, F., Timofte, R., and Gool, L. V. Repaint: Inpainting using denoising diffusion probabilistic models. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1 11451- 1 11461, 2022.
Du, Z.,, Kang, L., Tan, J., Gingold, Y. I., and Xu, K. Image vectorization and editing via linear gradient layer decomposition. ACM Trans. Graph., 42(4):97:1-97 139::1,, 2023.
10

---

Decomposition of Graphic Design with Unified Multinodal Model
Ma, X., Zhou, Y., Xu, X., Sun, B., Filev, V., Orlov, N., Fu, Y., and Shi, H. Towards layer-wise image vectorization. In Pro- ceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp. 16293-16302, 2022.
Schuhmann, C., Bcaumont, R., Vencu, R.,, Gordon, C., Wightman, R. Cherti, M., Coombes, T., Katta, A., Mullis, C., Wortsman, M., ct al. Laion-5b; An open large-scale datasct for training next generation image-text models. Advances in Neural Information Processing Systems (NeurIPS), 35:25278-25294, 2022.
Ma, Y., Liu, X., Chen, X., Liu, W., Wu, C., Wu, Z.., Pan, Z., Xie, Z.., Zhang. H., Zhao, L., et al. Janusflow: Harmonizing autoregres- sion and rectified flow for unified multimodal understanding and generation. arXiv preprint arXiv:2411 11.0777 2024.
Shi, W. Han, X., Zhou, C., Liang, W., Lin, X. V. Zettlemoyer. L.,, and Yu, L. Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint uxiv:2442..55188 2024.
Monnier, T., Vincent, E., Ponce, J., and Aubry, M. Unsupervised layered image decomposition into object prototypes. In Pro- ceedings of IEEE International Conference on Computer Vision (ICCV), pp. 8640-8650, 2021.
Sun, Q. Yu, Q., Cui, Y., Zhang, F., Zhang, X., Wang, Y., Gao, H., Liu, J.. Huang, T., and Wang, X. Emu: Generative pretraining in multimodality. In International Conference on Learning Representations (ICLR), 2024.
OpenAL. Gpt-4v(ision) system card, 2023. URL https://
Team, G., Anil, R., Borgeaud, S.,, Wu, Y., Alayrac, J.-B., Yu, J., Soricut, R., Schalkwyk, J., Dai, A. M., Hauth, A., et al. Gemini: family of highly capable multimodal models. arXiv preprint arXiv:2312.1 1105, 2023.
openai , aeteaai.ccm/eeeee wtt.c///ttte.cccceteeeee 4v- -ysttee-ccad
Oquab, M., Darcet, T., Moutakanni, T., Vo, H. V., Szafraniec, M., Khalidov. V., Fernandez. P., Haziza, D., Massa. F., El-Nouby. A., Assran, M., Ballas, N.. Galuba. W., Howes, R., Huang, P., Li, S., Misra, I., Rabbat, M., Sharma, V. Synnaeve, G., Xu, H. Jégou, H., Mairal, J., Labatut, P., Joulin, A., and Bojanowski, P. Dinov2: Learning robust visual features without supervision. Transactions on Machine Learning Research, 2024.
Team, I. Internlm: A multilingual language model with progres- sively enhanced capabilities, 2023.
Tong, S., Fan, D., Zhu, J., Xiong, Y., Chen, X., Sinha, K., Rabbat, M.. LeCun, Y., Xie, S., and Liu. Z. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint Xix::4412.141666 2024.
Ouyang. I.,, Wu, J., Jiang, X., Almeida, D., Wainwright, C.,, Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. Advances in Neural Information Processing Systems (NeurIPS), 35:27730-27744, 2022.
Wang, W., Chen, 7., Chen, X., Wu, Zhu, X., 7eng, G., Luo, P., Lu, T., Zhou, J., Qiao, Y., and Dai, J. Visionllm: Large language model is also an open-ended decoder for vision-centric tasks. In Advances in Neural Information Processing Systems (NeurIPS). 2023.
Qu. L., Zhang, H., Liu, Y., Wang, X., Jiang, Y., Gao, Y., Yc, H., Du, D. K., Yuan, Z., and Wu, X. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv 2024.
Wang, W., Lv. Q.. Yu, W.. Hong. W., Qi. J.. Wang. Y., Ji. J., Yang. Z.., Zhao, L.. Song. X.. Xu, J., Chen, K., Xu, B., Li, J.. Dong, Y., Ding, M., and Tang, J. Cogylm: Visual expert for pretrained language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024a.
preprint arXiv:2412.03066
Radford, A.. Wu, Child, R., Luan, D., Amodei. D., Sutskever, I. et al. Language models are unsupervised multitask learners. OpenAl blog, 18)); 2019.
Wang, X., Zhang, X., Luo, Z.,, Sun, Q., Cui, Y., Wang. J., Zhang, F., Wang. Y.. Li, Z.. Yu, Q. et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409. 18869, 2024b.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agar- wal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. Learning transferable visual models from natural language su- pervision. In Proceedings of International Conference on Ma- chine Learning (ICML). pp. 8748 8863-876,,22 2021.
Wu, C., Chen, X., Wu, Z. Ma. Y., Liu, X.,, Pan, Z.. Liu, W., Xie, Z., Yu, X., Ruan, C., et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint ar Xiv:2410. 13848, 2024a.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer lcarning with unified text-to-text transformer. Journal of Muchine Leurning Research, 21140):1167, 2020.
Wu, J., Jiang, Y., Ma. C., Liu, Y., Zhao, H., Yuan, Z., Bai, S., and Bai, X. Liquid: Language models are scalable multi-modal generators. arXiv preprint 2xXv:24112.04322 2024b.
Wu, Y., Zhang, Z., Chen, J., Tang, H., Li, D., Fang. Y., Zhu, L.. Xie, E., Yin, H., Yi, L., et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint 2:Xiv:2409.044/ 2024c.
Rodriguez, J. A., Puri, A., Agarwal, S., Laradji. I. H.. Rajeswar, S.,. Vázquez, D., Pal, C., and Pedersoli, M. Starvector: Generating scalable vector graphics code from images and text. In Proceed- ings of AAAI Conference on A Artificial Intelligence (AAAI), pp- 29691-29693, 2025.
Xie, J. Mao, W., Bai, Z., Zhang, D. J., Wang, W., Lin, K. Q., Gu, Y., Chen, Z., Yang. Z., and Shou, M. Z. Show-o: One sin- gle transformer to unify multimodal understanding and genera- tion. In International Conference on Learning Representations (ICLR), 2025.
Sbai, O., Couprie, C., and Aubry, M. Vector image generation by learning parametric layer decomposition. arXiv preprint arXiv:1812.05... 2018.
Schuhmann, C. Vencu, R., Beaumont, R., Kaczmarczyk, R., Mullis, C., Katta, A., Coombes, T., Jitsev, J., and Komatsuzaki, A. 44ion-400m: Open dataset of clip-filtere 400 million image- text pairs. arXiv preprint arXiv:2 / 11.02144 04,111120000 2021.
Xie, S., Zhang, Z., Lin, Z., Hinz, T., and Zhang, K. Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of IEFE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 222422,,7, 2023a.

---

Decomposition of Graphic Design with Unified Multinodal Model
Xie, S., Zhao, Y., Xiao, Z., Chan, K. C., Li, Y., Xu, Y., Zhang, K., and Hou, T. Dreaminpainter: Text-guided subject- driven image inpainting with diffusion models. arXiv preprint arXiv:2312.03771, 2023b.
Zhou, D., Fang, Song, X., Guan, C., Yin, Dai, Y., and Yang. lou loss for 2d/3d object detection. In International Conference on 3D Vision (3DV), pp. 85-94, 2019.
Zhu, D., Chen, J., Shen, X., Li, X., and Elhoseiny, M. Minigpt-4: Enhancing lasion-language understanding with advanced large language models. In International Conference on Learning Representations (ICLR), 2024.
Yamaguchi, K. Canvasvae: Learning to generate vector graphic documents. In Proceedings IEEE International Conference on Computer Vision (ICCV), pp. 5481-5489, 2021.
Yang, S., Chen, X., and Liao, Uni-paint: A unified framework for multimodal image inpainting with pretrained diffusion model. In ACM International Conference on Multimedia (ACM MM), pp. 3190-3199, 2023a.
Yang. S., Zhang. L.. Ma. L.. Liu, Y., Fu, J., and He, Y. Magicre- mover: Tuning-free free text-guided image inpainting with diffusion models. arXiv preprint arXiv:2310.02848 2023b.
Q. Xu, H., Xu, G., Ye, J., Yan, M.. Zhou, Y.. Wang, J., Hu, A., Shi, P. Shi, Y. et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint 2304. 14178, 2023.
Yu, T., Feng, R., Feng. R., Liu, J., Jin, X., Zeng, W., and Chen, Z. Inpaint anything: Segment anything meets image inpainting. arXiv preprint arxiv:2304.00 06790, 2023.
Zeng. A., Liu, X., Du. Z.. Wang. Z., Lai. H., Ding, M., Yang. Z., Xu, Y., Zheng, W., Xia, X., Tam, W. I.., Ma, Z., Xue, Y., Zhai, J., Chen, W., Liu, Z., Zhang, P., Dong, Y., and Tang, J. GLM- 130B: an open bilingual pre-trained model. In International Conference on Learning Representations (ICLR), 2023.
Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z.,, Zhang, X., Yuan, R., Zhang, G., Li. L.. Anygpt: Unified multimodal Ilm with discrete sequence modeling. In Annual Meeting of the Association for Computatonnal Linguisties., 2024.
Zhang, D., Yu, Y., Dong, J., Li, C., Su, D., Chu, C., and Yu, D. Mm-llms: Recent advances in multimodal large language mod- els. In Annual Meeting of the Association for Computational Linguistics. pp. 12401- 2024a.
Zhang, L. and Agrawala, M. Transparent image layer diffuusion using latent transparency. ACM Trans. Graph., 1-(4) 2024.
Zhang, S., Roller, S., Goyal, N., Artetxe, M., Chen, M., Chen, S., Dewan, C., Diab, M., Li, X., Lin, X. V., et al. Opt: Open pre-trained transformer language models, arXiv preprint arXiv:220.001663 2022.
Zhang, S., Yang, X., Feng, Y.. Qin, C., Chen, C.-C., Yu, N., Chen, Z., Wang, H., Savarese, S., Ermon, S., et al. Hive: Hareessing human feedback for instructional visual editing. In Proceedings of IEEE Conference Computer Vision and Pattern Recogni- tion (CVPR), pp. 9026-9036, 2024b.
7heng, K., He, X., and Wang, X. F. Minignt-5: Interleaved vision- and-lang generation via generative vokens, arXiv preprint arX0::230..0229,. 2023.
Zhou, C. Yu, L.. Babu, A., Tirumala, K., Yasunaga, M., Shamis, L., Kahn, J. Ma. X., Zettlemoyer, L., and Levy, O. Transfusion: Predict the next token and diffuse images with one multi-modal model. In International Conference Learning Representa- tions (ICLR), 2025.
12


Reviews:
[{"review_id": "MJaXk7gbam", "reviewer": "Reviewer_6rxP", "summary": "The paper proposes the problem setup Layer Decomposition (LD), and an approach Decompose Layer Model (DeaM), that can take the rendition (image) of a single page graphic design, and \"decompose\" it to its constituent components. The problem setup that the paper introduces has immense practical value, for instance, a user can scan a graphic design, and make changes to them, after it is broken down into its constituent components by DeaM.\n\n\n\n### Update after Rebuttal\n\nI thank the authors for their efforts in clarifying my queries and concerns. I am raising my score from weak reject to weak accept. The reason why I am not increasing the scores even further are:\n1. As pointed out earlier in my review, I genuinely feel that the way to evaluate the quality of each layer of the decomposed design should improve. Measuring FID of each component layer cannot serve a proxy to ensure whether each component that should have been decomposed into each layer has been done so. \n2. I share concerns with Reviewer NQZP on the quality of the output and with Reviewer ZHaU on not having much qualitative results. I appreciate that the authors shared 6 results during rebuttal phase, but that is too less for a computer-vision oriented paper. Combined with the fact that FID is the main quantitative metric (which my fellow reviewers has noted that it is not ideal to be used), makes it hard to form an informed decision. \n\nAs the paper introduces a new problem setting, it would greatly benefit the community if they release the model and the dataset, so that fellow researchers can build upon them. Thank you!", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your constructive feedback, which gives me the opportunity to clarify the ambiguities in the content of this paper.\n\n**Dataset Information:**  This dataset is collected from the internet and consists of single-page graphic designs (including all layers of the materials), comprising 224,054 samples. The samples are primarily posters and include areas such as holiday events, retail, dining, and corporate domains. The average number of layers in the dataset is 10.30, with an approximate image-text ratio of 6.3:3.7.\n\n**VQ-GAN Details:** The modification is very simple by adjusting the number of channels in the convolutional kernels of the first and last layers from 3 to 4 to accommodate the alpha channel. The training data for the poster images consists of over 200,000 samples collected as previously mentioned. However, we use the materials from all image layers for training, whereas MLLM uses the final poster images for training.\n\n**DINOv2 Details:** We were inspired by the [COMM](https://arxiv.org/abs/2310.08825) work. The visual encoder of CLIP is evidently well-aligned with the word embedding space, but due to the global supervision from image captions, it fails to learn more detailed pixel-level information. This might hinder the fine-grained perception capabilities in MLLM. Therefore, we added a visual encoder based on self-supervised training, DINOv2, whose self-supervised training approach enables it to focus more on pixel-level details (e.g., simple geometric element ).\n\n**Resolution Settings:** Higher image resolutions lead to clearer natural reconstructions, but they also make the training sequences of the model much longer and significantly increase computational cost. Based on empirical observations from VQ-GAN's experimental results on natural images, we chose a resolution of 192. Decorative elements are generally simpler than natural images, so a smaller resolution of 128 is sufficient to ensure adequate quality.\n\n**Regarding the evaluation:**\n1. The Crello training data is quite small (approximately 20k), making it challenging to obtain a reasonably good decomposition model. Due to time constraints, we first re-divided our dataset, randomly selecting 1000 images as a test set for evaluation. The results are [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/quan_res).\n2. We added evaluation results for single layers to this [link](https://github.com/anonymous-icml25-0328/rebuttal/blob/main/quan_res_fid_layer).\n3. We presented qualitative results on the test set of our dataset in [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_ours_vis).\n4. We will revise the description here. Currently, the model tends to generate more hallucinations in the predictions for small text and artistic text, whereas the attribute predictions for large printed text are relatively feasible.\n5. We supplemented the presentation with many failure cases, primarily highlighting issues where the reconstructed layers differ significantly from the original image, the cases are [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_crello_fail).\n\n**Regarding the writing:**\n1. CARD stands for Condition-Aware RGB-A Decoder. We will further refine the content of the introduction and methodology sections.\n2. Thank you for your suggestion. \"Layer Decomposition of Graphic Designs (LDGD)\" indeed better fits the task of this paper, and we will make the necessary modifications.", "claims_and_evidence": "1. The paper claims to introduce a new problem setup and the first approach towards the same, which is true. \n2. The paper claims to have collected a new dataset, but is largely silent about its key attributes:\n- How was it collected?\n- Are they all single page graphic designs?\n- How many datapoints are there (the paper vaguely says over 200,000 designs, but what is the exact number)? \n- What all domains of graphic designs does it cover: are they fashion, retail, corporate, and what else?\n- What is the average number of layer in the dataset?\n- What is the proportion of text and image layers in the dataset and on on...", "methods_and_evaluation_criteria": "The proposed approach is logical, but lot of details are missing:\n1. Sec 5.1 abruptly starts with the discussion on VQ-GAN. From the context, it seems that VQ-GAN has to be adapted to take in the alpha channel also. How was this adaption done? Was the Encoder and Decoder of VQ-GAN modified to include additional layers to consume the additional channel? \n2. Line 203 says that the VQ-GAN is trained on \"poster images\". Which dataset are these coming from?\n3. In sec 5.2, DINO v2 features are used to focus on \"lower-level visual elements\" (Line 217). Its very unclear how adding DINO v2 features helps in getting more attention to the elements like graphic lines and shape. \n4. In line 234, 235, the input resolution of natural images and decorative elements are set to 192 × 192 and 128 × 128. This choice should be validated by an ablation experiment. Its unclear why this choice improves performance. \n\nOn the evaluation:\n1. The model is trained on their new dataset, while evaluated on public dataset (Crello). This doesnt seem like a good evaluation protocol. It would have been ideal to train-and-test on the proposed dataset and Crello separately, this would showcase the mettle of the proposed approach on two datasets. \n2. The key characteristic of DeaM is it ability to create layers from the input image. Hence, during evaluation, the quality of each layer should be  explicitly checked for. Currently the image reconstruction quality is evaluated. Even if each layer is not perfect (say, two components that should have been separated into two layers, got into same layer), the reconstruction might be good, and hence is not a proxy for the quality of each layer. \n3. There are only two qualitative results in the paper (figure 4), which is too less to make an informed judgement. \n4. Line 415 reads: \"DeaM excels in text reconstruction due to its accurate prediction of text details such as content, font, size, and color.\", which is an absolutely flawed assertion as the figure 5 referred to in this section has gibberish text (see how breakfast is transformed), and has spelling mistakes \"#ThouchFreeDelivery\", \"count -> cocount\" and so on...\n5. Need to showcase failure cases too.", "theoretical_claims": "None", "experimental_designs_or_analyses": "See above", "relation_to_broader_scientific_literature": "Properly placed.", "essential_references_not_discussed": "None", "other_strengths_and_weaknesses": "The writing in the paper should improve a lot, like the term CARD is not introduced in the writing. The intro reads like it is preempted abruptly.  The paper will benefit from a strong proof-reading.\n\nMinor comment:\n- The new problem setting that the paper proposes is named as \"Layer Decomposition (LD)\", which is too broad. Ideally, any image, 3d-scene or any such data can be decomposed into layers, and the paper is not proposing a generic method to decompose all of those. The paper is specific to single-page graphic design, and hence the problem setup can be better termed as \"Layer Decomposition of Graphic Designs (LDGD)\"", "other_comments_or_suggestions": "I genuinely feel that there is merit to the problem setup, but the loopholes in the methodology, and insufficient evaluation makes me gravitate towards rejecting the paper in its current state. Happy to be convinced otherwise.", "questions_for_authors": "Please see the other sections.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J64Al20fti", "reviewer": "Reviewer_NQZP", "summary": "This paper proposes a novel layer decomposition model (DeaM) to transform a given graphic design into a set of ordered transparent layers. The key challenges include predicting the correct layer ordering and resovling the mutual occlusion between overlapping layers. To this end, DeaM first predicts a layer-specific JSON metadata followed by a condition-aware RGB-A decoder that can recontruct high-quality transparent layers.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable feedback and insightful comments.\n\n**About RGBA autoencoder:** \n1. This idea is very straightforward and interesting. Initially, we also tried using RGBA autoencoder as the visual encoder but found that the model's output generated severe hallucination information. We speculate that the RGBA autoencoder, using only image reconstruction as the training objective, leads to its inability to understand some semantic information. Therefore, using RGBA autoencoder as the visual encoder for MLLM would make it challenging to perform visual understanding tasks (layer decomposition tasks require visual understanding because they need to output not only image encoding but also metadata about the layers). Additionally, we observed that some recent works like [UniTok](https://arxiv.org/abs/2502.20321) simultaneously train the visual encoder using both contrastive and reconstruction losses. Using such an encoder might aid in layer decomposition tasks, which is also a future research direction for us. \n2. Our understanding is that the discrete tokens in the RGBA autoencoder space actually correspond to certain visual concepts (colors, textures, etc.). Considering the model capacity of LLM, learning this mapping relationship should still be feasible.\n\n**Variable-Length Layer Encoding:**\nYour suggestion is very reasonable. We have considered this approach, but we believe there are some issues: \n1. Resolution and encoding length might not be directly correlated. Some layers, such as pure color backgrounds, may have very high resolution but convey very little information, and thus can be represented with fewer tokens. Therefore, determining the appropriate encoding length for an image is also a challenge.\n2. If variable lengths are introduced, the model often makes errors in predicting token lengths, which can lead to failures in decoding the image. \n\n\n**Regarding Figure 4:**\nThank you for your reminder, it helps us further clarify the details. This example is intriguing. We believe that if there are many decorative elements and their spatial relationships do not have an apparent hierarchical order, the model tends to predict them as being on the same layer. This situation is relatively uncommon. Our model does have some capability in decomposing multi-layer graphic designs, and we will display more results of layer decomposition below.\n\n\n**Qualitative Results Display:**\n1. The decomposition results for some designs with more than 5 layers are [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_more_than_5_layers). \n2. We tested the single-layer images generated by the T2I model (here is the ideogram) and found that it is quite difficult to decompose such data, which significantly differs from the training data. This may require specialized collection of such T2I data for customized training to be successful. The decomposition results are as [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_ideogram).", "claims_and_evidence": "The authors propose using an LLM to transform a given set of discrete image tokens (prefix) that represent an entire image into a set of tokens consisting of the position of each layer, the attributes of each layer, and the exact visual tokens of each layer, with two different lengths: either 144 tokens or 64 tokens. A non-trivial design aspect is that the authors only encode the entire image with a combination of CLIP and DINOV2 visual encoders, rather than also considering the fine-tuned RGBA autoencoder. According to my understanding, the potential benefits of using the RGBA autoencoder are twofold: first, most of the predicted discrete visual tokens are essentially copied from the conditional global visual tokens based on the RGBA autoencoder; second, it is confusing for me to understand how the LLM can excel in learning to predict the discrete tokens in the RGBA autoencoder space from the space based on CLIP and DINOV2, which are non-trivial and challenging for the model to learn.\n\nAnother major concern is that the authors should support variable lengths for each transparent layer, rather than simply choosing two fixed lengths, considering that different transparent layers have totally different resolutions.\n\nAccording to Figure 4, the layer decomposition effect is weak, as all of the decoration layers are arranged within a single layer in the first example. Thus, it may raise concerns that the proposed layer decomposition scheme is weak and cannot be extended to support any number of layer decompositions.", "methods_and_evaluation_criteria": "The proposed approach is concise but non-trivial, considering that the target visual token space in the image layer is extracted using a completely different RGB-A autoencoder, while the input visual tokens are extracted based on CLIP and DINOV2.\n\nAnother concern is that the authors should demonstrate the following key aspects:\n\n- Whether the proposed approach performs well when handling an increasing number of layers, such as more than 5 layers.\n- Whether the proposed approach can generalize to single-layer images generated with T2I models like FLUX, rather than to in-domain ones like the Crello dataset.", "theoretical_claims": "There are no theoretical claims in this paper.", "experimental_designs_or_analyses": "The major concerns about the experimental results are that the FID score is still very high, even with the proposed three techniques. In addition, the visual results in Figure 4 are too naive and fail to demonstrate the potential value of such weak layer decomposition performance.", "relation_to_broader_scientific_literature": "I do not see the potential of this work for the broader scientific literature.", "essential_references_not_discussed": "No.", "other_strengths_and_weaknesses": "As mentioned earlier, the major concerns include:\n\n- The authors propose using an LLM to transform a given set of discrete image tokens (prefix) that represent an entire image into a set of tokens consisting of the position of each layer, the attributes of each layer, and the exact visual tokens of each layer, with two different lengths: either 144 tokens or 64 tokens. A non-trivial design aspect is that the authors only encode the entire image with a combination of CLIP and DINOV2 visual encoders, rather than also considering the fine-tuned RGBA autoencoder. According to my understanding, the potential benefits of using the RGBA autoencoder are twofold: first, most of the predicted discrete visual tokens are essentially copied from the conditional global visual tokens based on the RGBA autoencoder; second, it is confusing for me to understand how the LLM can excel in learning to predict the discrete tokens in the RGBA autoencoder space from the space based on CLIP and DINOV2, which are non-trivial and challenging for the model to learn.\n\n- Another major concern is that the authors should support variable lengths for each transparent layer, rather than simply choosing two fixed lengths, considering that different transparent layers have totally different resolutions.\n\n- Whether the proposed approach performs well when handling an increasing number of layers, such as more than 5 layers.\n\n- Whether the proposed approach can generalize to single-layer images generated with T2I models like FLUX, rather than to in-domain ones like the Crello dataset.", "other_comments_or_suggestions": "No other comments.", "questions_for_authors": "No further questions.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "mLA48Px9wM", "reviewer": "Reviewer_EBbB", "summary": "This paper focuses on the graphic design layer decomposition task that converts graphic designs into ordered RGB-A layers and metadata. A large multimodal model, i.e., DeaM, is proposed with a two-stage process. The first stage produces layer-specific JSON metadata, and the second stage reconstructs pixel-perfect layers. The proposed model could support full decomposition and interactive decomposition.\n\n## update after rebuttal\nI appreciate the authors' clarifications of some details. But my main concerns are not fully addressed by the rebuttal. Thus, I would like to keep my score as weak reject.\n\nThe proposed problem is not new, as there are many layer decomposition and generation papers published before, especially in graphics and vision conferences. The current version does not have a thorough discussion about the related works. Integrating LLM is a straightforward solution. In addition, compared with the commonly used metrics for graphic designs, the current evaluation metrics are not thorough enough.\n\nThe authors claim in the rebuttal that [2] does not have open-source code. But it is easy to find the corresponding GitHub link via Google.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Existing Layered Graphic Design Generation and Decomposition Work:**\nAlthough layered design generation and decomposition have been studied previously, there are many differences compared to our proposed work. Our focus is on the task of layer decomposition in graphic design. We will include discussions of works [1-4] in our paper. [1] actually investigates image generation by decomposing objects into different region layers (e.g., a portrait being decomposed into hair, face, etc.), making the generation process more organized. [2] and [3] generate layout information and materials from the user's input text and then synthesize the posters. Our layer decomposition process is the opposite of the aforementioned works. Although [2] also implements layer decomposition, it is somewhat different from our approach, as it focuses mainly on parsing single objects and cannot parse text. It is better suited for scenarios where each layer has minimal color and texture variations. Since [2] does not have open-source code, we cannot compare with it. Therefore, we follow the reviewers' suggestions and compare our work with [LIVE](https://arxiv.org/abs/2206.04655) in the derendering field.\n\n\n**Experimental Evaluation:**\n1. We added an [evaluation](https://github.com/anonymous-icml25-0328/rebuttal/blob/main/quan_res_fid_layer/1.png) of layer quality. The Loc metric can reflect the accuracy of bounding boxes and layer order to some extent.\n2. In the ablation experiments, Enhancing Prediction Regularity and Condition-Aware RGB-A Encoder had no impact on the accuracy of the output boxes from the MLLM. We tested the model without the Conjoined Visual Encoder, and the $Loc_\\hat{\\alpha}$ is 0.6826.\n3. Since [2] does not have open-source code, we followed the reviewers' suggestions and compared our work with [LIVE](https://arxiv.org/abs/2206.04655) in the derendering field. The results are [here](https://github.com/anonymous-icml25-0328/rebuttal/blob/main/test_live/test_live.png). We observed that these methods convert images into SVG format. However, SVG struggles to represent complex details in images, and these methods currently can only decompose simple graphics and cannot parse text.\n\n\n**Regarding the writing:**  Thank you for your reminder. We will revise the task description and abbreviations.\n\n\n**CARD Training Loss:** CARD stands for Condition-Aware RGB-A Decoder. It is an improvement of VQ-GAN by adding a conditional branch, and its training loss is the same as that of VQ-GAN.\n\n[1] Sbai, Othman, Camille Couprie, and Mathieu Aubry. \"Vector image generation by learning parametric layer decomposition.\" arXiv preprint arXiv:1812.05484 (2018).\n\n[2] Du, Zheng-Jun, et al. \"Image vectorization and editing via linear gradient layer decomposition.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-13.\n\n[3] Peidong Jia, Chenxuan Li, Zeyu Liu, Yichao Shen, Xingru Chen, Yuhui Yuan, Yinglin Zheng, Dong Chen, Ji Li, Xiaodong Xie, et al. Cole: A hierarchical generation framework for graphic design. arXiv preprint arXiv:2311.16974, 2023.\n\n[4] Naoto Inoue, Kento Masui, Wataru Shimoda, and Kota Yamaguchi. Opencole: Towards reproducible automatic graphic design generation. arXiv preprint arXiv:2406.08232, 2024", "claims_and_evidence": "Layered design generation and decomposition have been studied before, and cannot be considered as a novel vision task.", "methods_and_evaluation_criteria": "The proposed method is a reasonable solution to graphic design decomposition and editing. The proposed dataset could be useful for future research in the community.", "theoretical_claims": "The task formulation in Sec. 3 is a little bit unclear. The proposed method not only decomposes the image into an ordered series of RGB-A mode layers but also with metadata.", "experimental_designs_or_analyses": "The experimental evaluation is not thorough.\n1. The evaluation metrics only contain FID and Loc. FID cannot fully reflect the image quality. Additional metrics are necessary to evaluate the image quality, accuracy of the bounding box, and layer order.\n2. For the ablation study, even Loc. is not used for evaluation.\n3. The current experiment only compares the proposed method with one baseline. Additional layer decomposition methods should be compared and discussed in the experiments.\n4. It would be better to add experiments to show the applications of the proposed layer decomposition method.", "relation_to_broader_scientific_literature": "The proposed method is a reasonable and useful solution for automatic graphic design understanding and manipulation.", "essential_references_not_discussed": "Existing layered graphic design generation and decomposition should be cited and discussed in the paper [1-4].\n\n[1] Sbai, Othman, Camille Couprie, and Mathieu Aubry. \"Vector image generation by learning parametric layer decomposition.\" arXiv preprint arXiv:1812.05484 (2018).\n\n[2] Du, Zheng-Jun, et al. \"Image vectorization and editing via linear gradient layer decomposition.\" ACM Transactions on Graphics (TOG) 42.4 (2023): 1-13.\n\n[3] Peidong Jia, Chenxuan Li, Zeyu Liu, Yichao Shen, Xingru Chen, Yuhui Yuan, Yinglin Zheng, Dong Chen, Ji Li, Xiaodong Xie, et al. Cole: A hierarchical generation framework for graphic design. arXiv preprint arXiv:2311.16974, 2023.\n\n[4] Naoto Inoue, Kento Masui, Wataru Shimoda, and Kota Yamaguchi. Opencole: Towards reproducible automatic graphic design generation. arXiv preprint arXiv:2406.08232, 2024", "other_strengths_and_weaknesses": "The proposed method is a reasonable solution to graphic design decomposition and editing. The proposed dataset could be useful for future research in the community.", "other_comments_or_suggestions": "The abbreviation of the proposed method should be consistent. L024 is DeaM, while Fig. 6 uses DeAM.", "questions_for_authors": "The statement that the training loss for CARD is the same as that for VQGAN(Esser et al., 2021) is unclear.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "niyloMkc1u", "reviewer": "Reviewer_ZHaU", "summary": "The paper proposes a particular method for layerwise decomposition of graphic designs into RGBA images which can be stacked and composed to reform the original image. The authors train a transparency aware RGBA VQ-GAN encoder-decoder. The encoder is used to encode the input graphic design images for a LLM based decomposition/derendering model. The LLM predicts the JSON encoded structure of each layer in the graphic design, directly outputting text content for text layers and the VQ-GAN tokens for image layers. The VQ-GAN decoder translates these image tokens back into pixel-space. Training of the LLM is primarily done on a proprietary dataset of 200k designs. But evaluation is done on the publicly available Crello dataset. Experiments show the model outperforming a simple (but creative!) baseline. A small number of qualitative results are provided and ablation studies show the importance of a number of designs decisions for the VQ-GAN.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**Method Evaluation:**\n1. Considering the current scarcity of publicly available poster datasets and time constraints, we have first added a test set based on our own dataset split. We have added [qualitative](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_ours_vis) and [quantitative](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/quan_res) experimental results. \n2. Regarding evaluation metrics: Your suggestion is reasonable, we have added new similarity scoring metric in [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/quan_res_new_metric). We recruited 10 volunteers  for a human study to compare the evaluation effectiveness of CLIP and DINOv2. The choices between CLIP and DINOv2 were quite balanced.\n3. Regarding the addition of hierarchical captions: We were inspired by [COLE](https://arxiv.org/abs/2311.16974), which mentioned that layer-wise captions can enhance the model's perception of design elements. Hence, we intuitively added hierarchical captions.\n\n\n**Qualitative Results:**\nWe have added more qualitative experimental results, including both [successful](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_crello_succ) and [unsuccessful](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_crello_fail) cases.  We also tested the single-layer images generated by the T2I model (here is the ideogram) and found that it is quite difficult to decompose such data, which significantly differs from the training data. This may require specialized collection of such T2I data for customized training to be successful. The decomposition results are [here](https://github.com/anonymous-icml25-0328/rebuttal/tree/main/test_ideogram).\n\n**Discussion of Related Work:**\nThank you for your reminder. We will add these discussions. Existing design derendering methods such as LIVE and StarVector, although somewhat similar to our layer decomposition approach, still have some differences. We observed that these methods convert images into SVG format. However, SVG struggles to represent complex details in images, and these methods currently can only decompose simple graphics and cannot parse text. We have displayed some test results of LIVE in [here](https://github.com/anonymous-icml25-0328/rebuttal/blob/main/test_live/test_live.png). The tested images are from the test set of our dataset.\n\n\n**Others:**\n1. Equation 1 is correct. Minimizing the IoU loss is equivalent to maximizing the IoU, as IoU loss = 1 - IoU.\n\n2. Thank you for pointing this out. We will revise it. Table 2 should be written as $Loc_\\hat{\\alpha}$.\n\n3. In theory, images can be represented using SVG or HTML. However, these representations require relatively long character strings (compared to our image encoding length), and they struggle to represent complex images and detailed information.\n\n4. Thank you for your reminder. This part was not clearly stated. ImageNet and LAION were used as the training datasets for VQ-GAN. We used these datasets because posters contain many natural images, and we aimed to improve the encoding quality. The training for MLLM uses a dataset of 200k posters and the instruction data extended from it (see section 6.2).", "claims_and_evidence": "See other sections, the claims are primarily empirically justified.", "methods_and_evaluation_criteria": "- A single evaluation dataset Crello is used, perhaps additional dataset could have been added to the evaluation.\n- The metrics used for evaluation are rather limited. The FID score is a useful metric but may be only somewhat relevant for this work given that the underlying network embeddings were trained on ImageNet natural images which are quite far from the structured graphic designs presented in this work. In particular, I would have liked to have seen more visual similarity metrics. As the authors themselves point out CLIP and DINO embeddings are both standard, but they are also standard for evaluation. It would have been nice to have had CLIP similarity score metrics and DINO embedding cosine similarity between the original and reconstructed image. Even better would have been a small scale human study in which the humans were asked to rate which reconstruction (baseline or DeaM) was more faithful to the original.\n- Regarding the method: it was unclear to me why it was necessary to caption every layer in the LLM training dataset. IIUC the captions are not necessary at inference time and are solely used to instruction tune the model. But it instruction tuning is the use case for the captions, then why are they required at the layer level? Presumably it would suffice to have design level instructions/captions as the authors themselves present in Figure 6.", "theoretical_claims": "N/A", "experimental_designs_or_analyses": "The experimental design seems fine to me, however as discussed above the breath of evaluation datasets and metrics is rather limited and as discussed below it may have been possible to include some other derendering baselines.", "relation_to_broader_scientific_literature": "See below: Essential References Not Discussed.", "essential_references_not_discussed": "The paper is fundamentally about derendering, there is quite a bit of existing work on derendering, both of graphic designs and more generally e.g. charts, tikz code. etc. that should have been discussed. In addition existing design derendering methods could have been compared to and discussed e.g. LIVE [https://arxiv.org/abs/2206.04655] and StarVector [https://arxiv.org/abs/2312.11556].", "other_strengths_and_weaknesses": "I would like to use this section to state that despite the issues mentioned above, I really liked the paper. It is well written, on an interesting and underexplored topic. If the authors address several of my comments in the rebuttal phase, which seems very feasible I would be happy to raise my score and advocate more strongly for the paper.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "- Is eq. 1 correct? Should it be argmax rather than argmin, given that presumably we want to maximize the IoU over all possible permutations?\n- Related, where is $Loc_{\\alpha}$ in Table 2 defined? I assume it is the metric from eq. 1 but that merely defines $\\hat{\\alpha}$ and not $Loc_{\\alpha}$.\n- Did the authors consider using an off the shelf layered graphic design representation e.g. SVG, HTML, etc. rather than the custom RGBA based representation used in the paper?\n- It was not 100% clear to me how ImageNet or LAION were used for training, presumably this was only for the VQ-GAN as these datasets do not have the layered structure required for the LLM decomposition training. Could you confirm my understanding is correct and only the 200k posters dataset was used in the LLM training.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper introduces Layer Decomposition (LD), a new vision task that converts graphic designs into structured, ordered RGB-A layers with metadata for easier editing and reuse. The paper proposes DeaM, a multimodal model that tackles challenges like layer ordering and occlusion through a two-stage process combining visual encoding, metadata generation, and precise layer reconstruction.

The paper was reviewed by four experts with final ratings of one Accept, two Weak Accepts and one Weak Reject. All reviewers appreciated the setting but also had concerns around the evaluations (limited results, need for more ablation studies). The authors' response largely addressed the concerns of the reviewers. After the rebuttal phase, Rev#6rxP upgraded their score to Weak Accept, explicitly stating that the problem setup is novel, and the approach is logical although the evaluation could have been stronger. While some concerns remained around the need for more qualitative results, the additional results in the rebuttal and the interestingness of the setting itself tilted the reviewers' ratings towards acceptance.

Given the mixed reviews, the AC went through the paper before making a decision. Considering that the positives outweigh the negatives, the paper is recommended for acceptance. The authors are recommended to integrate the rebuttal responses in the final version.
Decision: Accept (poster)