PUMA: Empowering Unified MLLM with Multi-granular Visual Generation

## Abstract

Abstract Recent advancements in multimodal foundation models have yielded significant progress in vision-language understanding. Initial attempts have also explored the potential of multimodal large language models (MLLMs) for visual content generation. However, existing works have insufficiently addressed the varying granularity demands of different image generation tasks within a unified MLLM paradigm — from the diversity required in text-to-image generation to the precise controllability needed in image manipulation. In this work, we propose PUMA , em P owering U nified MLLM with M ulti-gr A nular visual generation. PUMA unifies multi-granular visual features as both inputs and outputs of MLLMs, elegantly addressing the different granularity requirements of various image generation tasks within a unified MLLM framework. Following multimodal pretraining and task-specific instruction tuning, PUMA demonstrates proficiency in a wide range of multimodal tasks. This work represents a significant step towards a truly unified MLLM capable of adapting to the granularity demands of various visual tasks. The code and model will be released in https://github.com/rongyaofang/PUMA .

## 1 Introduction

Unifying multimodal understanding and generation capabilities within a single model is a critical milestone toward artificial general intelligence (AGI). Towards this goal, recent advancements (Liu et al., ; Zhu et al., ) in multimodal large language models (MLLMs) have made significant progress in integrating visual reasoning and understanding with natural language interfaces. However, developing a unified framework that excels at both comprehending and generating multimodal content remains a significant challenge in the field of artificial intelligence.

Recent studies (Sun et al., ; Ge et al., ) have explored MLLM’s potential for visual generation, beyond the previously well-explored visual understanding and reasoning with MLLMs. These approaches enable MLLMs to process image-text inputs and produce either textual outputs or semantic-level visual tokens. In the case of image generation, these visual tokens are subsequently transformed into pixel-space images using diffusion-based decoders. Such unified frameworks empower MLLMs to perform a wide spectrum of tasks within a single framework, ranging from detailed visual analysis to creative image synthesis.

However, existing MLLM-based methods (Sun et al., ; ) face a common challenge in the trade-off between diversity for text-to-image generation and high controllability for tasks such as image editing. Previous methods mostly rely on single-granular features extracted from a visual encoder and neglect the varying granularity requirements of different tasks. On the one hand, generating diverse images reflecting the real world from text descriptions requires features that encode coarse semantic concepts. Such features are fed as conditions into the diffusion-based image decoder, allowing the diffusion model to generate diverse images that semantically align with the text prompt. On the other hand, tasks demanding precise control over output images, such as image editing and inpainting, require the LLMs to predict fine-grained features that encode rich, detailed visual information for the image decoder. This dichotomy presents a significant challenge for current MLLM-based methods, which typically generate single-granular feature representations for all tasks. As a result, models optimized for diverse image generation often lack the fine-grained controllability necessary for detailed downstream tasks such as editing, while those focused on precise controllability produce less varied outputs for the task of text-to-image generation. Although recent work like SEED-X (Ge et al., ) attempts to bypass this issue by leveraging condition images directly input to the diffusion-based decoder for fine-grained control, a unified solution to the multi-granularity problem remains underexplored.

Towards the multi-granular feature demands of various tasks, we propose a novel paradigm emPowering Unified MLLM with Multi-grAnular visual generation (PUMA). PUMA facilitates seamless integration of image generation and understanding processes, while simultaneously handling multiple feature granularities — from coarse-grained abstractions to fine-grained details — within a single framework. By leveraging multi-scale features, our approach empowers MLLMs to excel in diverse image generation and controllable downstream tasks, within a unified framework.

Our method comprises three key modules: 1) An image encoder that extracts multi-granular representations, which serve as the foundation for visual generation and understanding; 2) An autoregressive MLLM that processes and progressively generates multi-scale image features; and 3) A set of dedicated diffusion-based image decoders that decode images from MLLM-generated features at multiple granularities. To optimize this framework, we employ a two-stage training strategy: first fine-tuning the set of pre-trained diffusion models as our image decoders, where each model reconstructs or generates images conditioned on the corresponding feature granularities from the encoder; then training the autoregressive MLLM with regression loss supervised by the multi-scale encoder features to process and generate multi-granular image features. PUMA leverages large-scale pre-training followed by task-specific instruction tuning on a collection of linguistic-visual datasets, enabling our model to handle various tasks including image understanding, text-to-image generation, editing, inpainting, colorization, and conditional generation.

In summary, we introduce a novel multi-granularity paradigm for MLLMs that addresses the limitations of existing single-scale methods. By simultaneously processing and generating features at multiple granularities, our approach enables a unified framework to handle a wide range of tasks, from diverse image generation to precise editing and highly controllable generation. This unified framework represents a significant advancement towards more versatile and capable MLLMs, contributing to the broader goal of achieving AGI in multimodal domains.

## 2 Related Work

### 2.1 Multimodal Understanding

The rapid advancement of large language models (LLMs) has catalyzed significant progress in multimodal large language models (MLLMs) for multimodal understanding tasks Dai et al. (); Li et al. (); Zhang et al. (); Chen et al. (); Lin et al. (); Zhang et al. (); Li et al. (). Pioneering works such as LLaVA (Liu et al., ) and MiniGPT-4 (Zhu et al., ) have demonstrated remarkable performance across diverse image understanding tasks, including visual question answering (VQA), visual reasoning, optical character recognition (OCR), and object grounding. These approaches typically employ visual encoders, such as the CLIP encoder (Radford et al., ), to extract continuous image features, which are then projected into the LLM’s embedding space for subsequent tasks. While successfully unifying various image understanding tasks within a single model, these methods mostly adhere to a multimodal-input, text-output paradigm. Consequently, they excel at text-based responses to visual inputs but cannot generate multimodal outputs beyond text, limiting their applicability in tasks requiring visual content generation.

### 2.2 Unified Understanding and Generation for MLLMs

Recent research has focused on equipping MLLMs with multimodal output capabilities (Wu et al., ; Tang et al., ; Ye et al., ; Zhu et al., ). GILL (Koh et al., ) pioneered the integration of image generation abilities into MLLMs. Subsequently, SEED-LLaMA (Ge et al., ) and Emu (Sun et al., ) further advanced image generation and understanding capabilities within MLLMs, while DreamLLM (Dong et al., ) proposed an end-to-end training approach for enhanced performance.

More recent works, such as SEED-X (Ge et al., ) and Emu2 (Sun et al., ), have scaled up MLLMs for unified generation, adopting continuous feature-based methods. These approaches utilize pre-trained vision encoders to extract continuous semantic features, which MLLMs then autoregressively regress. Specialized diffusion model-based decoders transform these MLLM-generated features into pixel-space images. However, the single-scale image feature generation pipeline employed by these methods struggles to address tasks with varying granularity demands, making it challenging to balance diverse image generation with fine-grained control for manipulation tasks. SEED-X attempts to address the multi-granularity issue by introducing conditional image input to the diffusion-based decoder for fine-grained control. However, this approach limits its applicability to image editing tasks encountered during decoder training. Consequently, a unified solution to the multi-granularity problem remains underexplored. In contrast, our work proposes a novel multi-granularity paradigm that addresses these limitations by simultaneously handling multiple levels of feature granularity within a single, unified framework.

Alternative approaches have also been investigated. Chameleon (Team, ) explored using discrete image tokens to bridge image understanding and generation, but the vector quantization process leads to information loss, hindering high-performance image understanding. TransFusion (Zhou et al., ) and show-o (Xie et al., ) proposed transforming the MLLM backbone itself into a denoiser in a diffusion-based or demasking-based approach. However, these methods require numerous denoising steps for each image generation, resulting in substantial computational costs given the scale of current MLLM backbones. VAR (Tian et al., ) is another track of generation framework that implements hierarchical autoregressive with discrete tokens for image generation, but it only discusses image generation and cannot unify multimodal tasks.

Figure: Figure 2: Upper: PUMA’s unified multi-granular autoregressive pipeline for processing and generating text and multi-granular visual features. Lower: Illustration of PUMA’s versatility across various tasks: 1) diverse text-to-image generation, 2) image editing, 3) conditional image generation, and 4) image understanding, showcasing different input-output configurations.
Refer to caption: x2.png

## 3 Method

Existing approaches typically optimize for either fine or coarse-grained features, resulting in a trade-off between precise control and generation diversity. To overcome this limitation, we propose PUMA, a unified multi-granular MLLM paradigm. Our approach simultaneously processes multiple levels of feature granularity within a unified MLLM framework, facilitating seamless transitions across a wide spectrum of multimodal tasks.

Our framework consists of three key components: an image encoder (Sec. 3.1), a set of image decoders conditioned on different granular features (Sec. 3.2), and a multi-granular autoregressive MLLM (Sec. 3.3). These components work synergistically to extract, process, and generate multi-scale image features, adapting to various task-specific granularity requirements. To optimize our MLLM, we employ a two-stage process of pretraining and instruction tuning (Sec. 3.4), enabling it to perform a wide range of tasks including image understanding, generation, editing, and conditional image generation.

### 3.1 Image Encoding and Multi-granular Feature Extraction

Our unified multi-granularity paradigm leverages a semantic image encoder to extract multi-scale features, forming the foundation for diverse visual task processing. We employ a CLIP (Radford et al., ) semantic image encoder to process input images $x$ and generate the initial set of high-resolution features $f_{0}\in\mathbb{R}^{H\times W\times C}$, with $H$ and $W$ representing the spatial dimensions of the highest resolution feature grid, and $C$ denoting the channel dimension. In our setting, the feature size is $H=W=16$, thus the highest resolution feature $f_{0}$ has $256$ visual tokens.

To obtain multi-granular representations, we derive lower resolution features through successive applications of 2D average pooling with kernel size $2$ and stride $2$:

$$ $f_{i}=\text{AvgPool}(f_{i-1}),\quad i=1,2,...,N$ (1) $$

where $N$ is the number of additional granular levels. This process generates a series of feature grids at progressively coarser resolutions, ranging from fine-grained features preserving detailed spatial information and local textures, through mid-level features capturing object parts and regional structures, to features representing coarse-grained semantic concepts. These features are denoted as $f_{0}$, $f_{1}$, $f_{2}$, $f_{3}$, and $f_{4}$, which have $256$, $64$, $16$, $4$, and $1$ visual tokens respectively.

### 3.2 Multi-Granular Visual Decoding

Image features at different granularities encode varying levels of information. We employ diffusion-based models as decoders due to their flexible capability to handle multi-scale features. When processing coarse-grained semantic features, the decoders can effectively synthesize missing fine-grained information with their learned image priors and generate diverse, semantics-aligned images. On the other hand, when handling fine-grained features, they accurately reconstruct precise image details. This versatility in generating or reconstructing images across different granularities makes diffusion-based models suitable for our multi-granularity approach.

We develop a set of dedicate diffusion-based image decoders ${D_{0},D_{1},...,D_{N}}$ corresponding to the feature scales ${f_{0},f_{1},...,f_{N}}$. These decoders enable the visual decoding of images at various levels of granularity. We formulate the image decoding process for each granularity level $i$ as $\hat{x}_{i}=D_{i}(f_{i},z)$, where $\hat{x}_{i}$ is the decoded image, $f_{i}$ is the feature map at granularity level $i$, and $z$ is a random noise vector for the diffusion process.

We leverage the pre-trained SDXL models (Podell et al., ) as our decoding framework and fine-tune these pre-trained models to generate or reconstruct images conditioned on different granular features. By modifying the conditional input mechanism through cross-attention in SDXL to accept our multi-granular features $f_{i}$, we harness the models’ inherent ability to decode coherent images.

Figure: Figure 3: Multi-granular visual decoding from fine-grained to coarse-grained granularity.
Refer to caption: x3.png

Figure: Figure 4: Training phase of multi-granular visual decoding.
Refer to caption: x4.png

Fig. 4 shows the training process of different granular image decoding, during which the image encoder is frozen to preserve semantic property. Fig. 3 illustrates the visual decoding capabilities of multi-granular decoders. The visualizations demonstrate the fidelity of decoded images across different granularities, with finer-grained features yielding reconstructions closer to the original input, and coarser-grained features leading to image generation guided by the semantics of the input image. This validates the effectiveness of our approach in preserving and utilizing multi-granular visual information.

This multi-granular decoding framework, in conjunction with our hierarchical feature extraction, establishes a foundation for the subsequent stages of our MLLM architecture, paving the way for diverse visual tasks in later training phases.

### 3.3 Progressive Multi-Granular Image Modeling in Autoregressive MLLM

Driven by the goal of utilizing a unified framework capable of adapting to a wide range of visual-linguistic tasks with varying granularity requirements, we design an autoregressive MLLM to process and generate both text tokens and multi-granular image features.

Our autoregressive MLLM, denoted as $M$, processes text and multi-granular image features progressively, as illustrated in Fig. 2. The model processes features token by token, predicting each token sequentially within each granularity level, and progressing from the coarsest level $N$ to the finest level 0 0. This approach allows the model to refine its predictions as more detailed information becomes available.

We structure the input sequence as a concatenation of text tokens and flattened image feature tokens from multiple granularity levels. This progressive approach enables the model to capture dependencies across different scales, from coarse global structures to fine local details.

The MLLM is trained using an autoregressive next token prediction objective, combining both text and image losses:

$$ $\mathcal{L}=-\sum_{i}\log P(t_{i}|t_{<i},F_{<i})+\sum_{i=0}^{N}\alpha_{i}\sum_ {j=1}^{k_{i}}|f_{i,j}-\hat{f}_{i,j}|^{2}\vspace{-8pt}$ (2) $$

The first term represents the cross-entropy loss for text token prediction, where $t_{i}$ are text tokens. The second term is the regression loss for image feature prediction, where $f_{i,j}$ and $\hat{f}_{i,j}$ are the ground truth and predicted feature tokens, respectively, at the $i$-th granularity level. $k_{i}$ is the number of visual tokens at the $i$-th granularity level. The coefficient $\alpha_{i}$ allows for adjusting the importance of each granularity level during training.

### 3.4 Multimodal Pretraining and Instruct Tuning

To demonstrate the effectiveness of our unified multi-granularity paradigm, we implement a comprehensive two-stage training pipeline for PUMA: multimodal pretraining followed by task-specific instruct tuning. This approach allows our model to first acquire broad multimodal capabilities before specializing in targeted visual-linguistic tasks during the subsequent instruct tuning stage.

Multimodal Pretraining: Our multimodal pretraining leverages a diverse set of large-scale datasets: Laion-2B (Schuhmann et al., ), Laion-Aesthetics (Burger, ), GRIT (Peng et al., ), The Pile (Gao et al., ), OCR-VQA-200K (Mishra et al., ), and LLaVAR (Zhang et al., ). This combination of datasets provides a rich mixture of image-text pairs, textual data, and specialized visual question-answering samples. To enhance the model’s bidirectional understanding of image-text relationships, we employ a dynamic training strategy that randomly alternates between text-to-image and image-to-text tasks for each image-text pair.

Instruct Tuning: Following pretraining, we conduct targeted instruct tuning to adapt our model to specific visual-linguistic tasks. To evaluate PUMA’s performance across different task types, we fine-tune four dedicated models for the four types of tasks, each initialized from the pretraining checkpoint.

High-quality Text-to-Image Generation: We utilize Laion-Aesthetics (Burger, ) and JourneyDB (Sun et al., ) to focus on generating aesthetically pleasing and diverse images.

Precise Image Manipulation: Training on the SEED-Edit (Ge et al., ) dataset enables accurate and controlled image editing.

Conditional Image Generation: The subset of
MultiGen-20M dataset (Qin et al., ) including canny-to-image, inpainting, and colorization is employed to equip the model with the ability to generate images under specific conditions and constraints.

Image Understanding: Fine-tuning on the subset of LLaVA-OneVision (Li et al., ) and Cambrain (Tong et al., ) to enhance the model’s image comprehension capabilities. Data about math/reasoning and cross-duplicated data in the two datasets are removed.

## 4 Experiments

We present our experimental results as follows: Sec. 4.1 details our experimental setup. In Sec. 4.2, we evaluate the effectiveness of our multi-granularity feature encoding and diffusion-based multi-granularity image decoders. We then demonstrate PUMA’s versatility across various tasks: diverse text-to-image generation (Sec. 4.3), image editing (Sec. 4.4), conditional image generation (Sec. 4.5), and vision-language understanding (Sec. 4.6).

### 4.1 Setup

Our unified multi-granular MLLM employs LLaMA-3 8B (Touvron et al., ) as the language model backbone and CLIP-Large ($224\times 224$ input) (Radford et al., ) as the image encoder. The image decoders are initialized from pretrained SDXL models (Podell et al., ). For more details on the experimental setup, please refer to the Appendix.

Figure: Figure 5: Fine-grained image reconstruction of SEED-LLaMA (Ge et al., ), SEED-X (Ge et al., ), Emu2 (Sun et al., ) and PUMA ($f_{0}$ scale). High quality image reconstruction is the foundation of precise image manipulation tasks.
Refer to caption: x5.png

**Table 1: Image decoding evaluation using image encoder and decoder on the ImageNet validation set. PSNRr and LPIPSr measure the difference between reconstructed and ground truth images. PSNRd and LPIPSd measure the difference between two separate reconstructions of the same image, reflecting decoding diversity.**
| Model | Encoder foundation | Token num. | PSNRr$\uparrow$ | LPIPSr$\downarrow$ | PSNRd$\downarrow$ | LPIPSd$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- |
| SEED-LLaMA () | BLIP-2 ViT (0.3B) | 32 | 9.73 | 0.6756 | 10.45 | 0.6189 |
| SEED-X () | Qwen-VL Encoder (4B) | 64 | 10.86 | 0.5152 | 11.60 | 0.4292 |
| Emu2 () | EVA02-CLIP-E-plus (4B) | 64 | 15.72 | 0.2532 | 16.07 | 0.2101 |
| PUMA ($f_{4}$ scale) | CLIP-Large (0.3B) | 1 | 10.76 | 0.6481 | 12.82 | 0.5751 |
| PUMA ($f_{3}$ scale) | CLIP-Large (0.3B) | 4 | 11.04 | 0.5971 | 12.61 | 0.5329 |
| PUMA ($f_{2}$ scale) | CLIP-Large (0.3B) | 16 | 12.35 | 0.4992 | 13.50 | 0.4354 |
| PUMA ($f_{1}$ scale) | CLIP-Large (0.3B) | 64 | 13.26 | 0.4325 | 14.12 | 0.3631 |
| PUMA ($f_{0}$ scale) | CLIP-Large (0.3B) | 256 | 18.16 | 0.2215 | 19.36 | 0.1559 |

### 4.2 Multi-granular Visual Decoding

We evaluate the multi-granular visual decoding capabilities of our model using multi-scale features from the encoder (Sec. 3.1) and dedicated visual decoders (Sec. 3.2). Our aim is twofold: to achieve precise reconstruction using fine-grained feature scales (such as $f_{0}$ and $f_{1}$), and to implement high diversity semantics-guided image generation using coarse-grained features (such as $f_{4}$ and $f_{3}$). It is worth mentioning that in this subsection we validate the multi-granularity encoder and decoders (Fig. 4), while the MLLM (Sec. 3.3) is not leveraged for the experiments in this subsection.

#### 4.2.1 Fine-grained Image Reconstruction

Fine-grained image reconstruction is crucial for preserving image details, yet it has posed significant challenges for models like SEED-LLaMA (Ge et al., ), SEED-X (Ge et al., ), and Emu2 (Sun et al., ). While SEED-LLaMA and SEED-X struggle with detailed reconstruction, limiting their precise image manipulation capabilities without additional techniques such as conditional image input (as used in SEED-X), Emu2 attempts to improve reconstruction by scaling up its image encoder to 4 billion parameters. Our approach achieves superior reconstruction quality with a more efficient architecture. We employ the CLIP-Large encoder (0.3 billion parameters), which is over 10 times smaller than Emu2’s, and implement fine-grained level image embedding with 256 tokens. As demonstrated in Tab. 1, our method using $f_{0}$ scale features achieves $18.16$ PSNRr and $0.2215$ LPIPSr (Zhang et al., ) on the ImageNet validation set reconstruction. These results outperform Emu2’s reconstruction performance and significantly surpass SEED-LLaMA and SEED-X (without conditional input). Fig. 5 visually illustrates our method’s superior reconstruction quality.

#### 4.2.2 Semantics-guided Generation

While fine-grained reconstruction is crucial for precise image manipulation, tasks like text-to-image generation benefit from a balance of semantic fidelity and output diversity. Our approach leverages coarse-grained features (such as $f_{4}$) to implement semantics-guided image generation that preserves diversity in outputs. To quantify this semantics-guided diversity, we decode twice to obtain two images from the same image input using different random seeds and measure their differences, denoted as PSNRd and LPIPSd. Tab. 1 presents the diversity results for various visual decoding models and feature scales. Notably, our $f_{3}$ and $f_{4}$ scale decoders produce more diverse samples compared to the decoders in SEED-X and Emu2, while still preserving the core semantics of the input, as illustrated in Fig. 5. This demonstrates our approach’s effectiveness in balancing semantic accuracy with generative diversity, a crucial factor in tasks like text-to-image generation.

Figure: Figure 6: Diversity visualization of text-to-image generation results from PUMA feature scales $f_{4}$ (1 visual token), $f_{3}$ (4 visual tokens), and Emu2 (Sun et al., ). The generated features are input to corresponding diffusion-based decoders with different random seeds.
Refer to caption: x6.png

Figure: Figure 7: Visualization of text-to-image generation results from PUMA feature scales $f_{4}$ (1 visual token) and $f_{3}$ (4 visual tokens).
Refer to caption: x7.png

**Table 2: Diverse text-to-image generation evaluation on MSCOCO 30K validation set. CLIP-I and CLIP-T measure the similarity between generated images and ground truth images or prompts. LPIPSd quantifies the difference between two images generated from the same prompt, reflecting generation diversity. 5-scale Max denotes selecting the image with the highest score among the 5 outputs and computes the average maximum value.**
| Model | Token num. | CLIP-I$\uparrow$ | CLIP-T$\uparrow$ | LPIPSd$\uparrow$ |
| --- | --- | --- | --- | --- |
| SD-v1.5 () | - | 0.667 | 0.302 | 0.692 |
| DALL-E 2 () | - | - | 0.314 | - |
| SDXL () | - | 0.674 | 0.310 | 0.600 |
| DALL-E 3 () | - | - | 0.320 | - |
| SEED-LLaMA () | 32 | 0.682 | - | 0.652 |
| Emu () | 64 | 0.656 | 0.286 | 0.700 |
| Emu2 () | 64 | 0.686 | 0.297 | 0.329 |
| SEED-X () | 64 | 0.729 | 0.314 | 0.493 |
| PUMA ($f_{4}$ scale) | 1 | 0.699 | 0.295 | 0.613 |
| PUMA ($f_{3}$ scale) | 4 | 0.703 | 0.300 | 0.558 |
| PUMA (5-scale Max) | - | 0.736 | 0.317 | - |

**Table 3: Image editing evaluation on Emu-edit test benchmark (Sheynin et al., ). 5-scale Max denotes selecting the image with the highest score among the 5 outputs and computes the average maximum value.**
| Model | CLIP-I$\uparrow$ | CLIP-T$\uparrow$ | DINO$\uparrow$ |
| --- | --- | --- | --- |
| InstructPix2Pix () | 0.834 | 0.219 | 0.762 |
| MagicBrush () | 0.838 | 0.222 | 0.776 |
| EMU-Edit () | 0.859 | 0.231 | 0.819 |
| OmniGen () | 0.836 | 0.233 | 0.804 |
| PUMA ($f_{1}$ scale) | 0.802 | 0.258 | 0.679 |
| PUMA ($f_{0}$ scale) | 0.840 | 0.264 | 0.784 |
| PUMA (5-scale Max) | 0.846 | 0.270 | 0.785 |

### 4.3 Diverse Text-to-image Generation

Our method can generate diverse outputs by utilizing the coarse-grained feature ($f_{4}$ and $f_{3}$ scales). This capability enables our model to produce diverse images that correspond to text conditions. Fig. 6 demonstrates that when generating images with a fixed text prompt utilizing feature scales $f_{4}$ and $f_{3}$, our model achieves high generation diversity. It also shows that $f_{4}$ scale outputs exhibit higher diversity, while $f_{3}$ scale results demonstrate better consistency. In contrast, the generation results of Emu2 (Sun et al., ) show low diversity. For qualitative evaluation, Fig. 7 presents visualizations of our model’s text-to-image generation with various prompts. For quantitative results, we evaluate our model on the MSCOCO 30K validation dataset (Lin et al., ) and present the CLIP-I, CLIP-T, and LPIPSd in Tab. 2, which the former two metrics measures the consistency while LPIPSd measures generation diversity. Compared with recent works, our model demonstrates superior performance in generation quality, diversity, and prompt relevance.

### 4.4 Image Editing

Figure: Figure 8: Left: Visualizations of PUMA’s image editing result. Image editing utilizes $f_{0}$ scale feature to preserve the fine-grained detail of input image. Right: Visualization of PUMA’s conditional generation results. ❶: canny-to-image generation; ❷: image inpainting; ❸: image colorization.
Refer to caption: x8.png

To assess PUMA’s image editing capabilities, we evaluated it on the Emu-Edit test benchmark (Sheynin et al., ). Tab. 3 presents the results using CLIP-I, CLIP-T, and DINO (Caron et al., ) scores. CLIP-I and DINO scores measure the model’s ability to preserve elements from the source image, while CLIP-T reflects the consistency between the output image and the target caption. Our results demonstrate that PUMA exhibits strong preservation ability, second only to the current state-of-the-art model, EMU-Edit. Notably, PUMA achieves significantly better CLIP-T scores, even surpassing the state-of-the-art model. This indicates superior alignment between edited images and target captions. For qualitative evaluation, Fig. 8 provides visualizations of the editing results, illustrating PUMA’s effectiveness in image manipulation tasks.

Figure: Figure 9: Comparison of $f_{0}$ and $f_{1}$ feature scales for tasks requiring precise controllability.
Refer to caption: x9.png

**Table 4: Evaluation on multimodal understanding benchmarks. PUMA utilizes CLIP-Large encoder with $224\times 224$ input. Und. and Gen. denote “understanding” and “generation”, respectively.**
| Type | Model | # Params | MMB$\uparrow$ | MME$\uparrow$ | GQA$\uparrow$ | VQAv2(test)$\uparrow$ | POPE$\uparrow$ | Vizwiz$\uparrow$ |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Und. Only | LLaVA-v1.5 () | 7B | 64.3 | 1510.7 | 62.0 | 78.5 | 85.9 | 50.0 |
| InstructBLIP () | 13B | - | 1212.8 | 49.5 | - | 78.9 | 33.4 |  |
| Qwen-VL-Chat () | 7B | - | 1487.5 | 57.5 | 78.2 | - | 38.9 |  |
| mPLUG-Owl2 () | 7B | 64.5 | 1450.2 | 56.1 | 79.4 | 85.8 | 54.5 |  |
| Und. and Gen. | Emu () | 13B | - | - | - | 57.2 | - | - |
| NExT-GPT () | 7B | 58.0 | - | - | 66.7 | - | 48.4 |  |
| SEED-X () | 17B | 75.4 | 1457.0 | 47.9 | - | 84.2 | - |  |
| Chameleon () | 34B | - | - | - | 66.0 | - | - |  |
| Emu2-Chat () | 40B | - | - | 65.1 | 84.9 | - | 54.9 |  |
| PUMA (Ours) | 8B | 68.9 | 1490.3 | 60.6 | 76.2 | 85.2 | 47.9 |  |

### 4.5 Conditional Image Generation

We select a subset of canny-to-image, inpainting, and colorization tasks from the multigen-20M dataset to train PUMA’s conditional image generation ability. Fig. 8 demonstrates the conditional generation results for these tasks. The $f_{0}$ feature scale results provide the highest preservation of image details, particularly for tasks like inpainting and colorization, while the $f_{1}$ scale offers better overall visual fidelity with limited generation diversity.

### 4.6 Image Understanding

We evaluate PUMA’s image understanding performance on several MLLM benchmarks, including MMB (Liu et al., ), MME (Fu et al., ), GQA (Hudson & Manning, ), VQAv2 (Antol et al., ), POPE (Li et al., ), and Vizwiz (Gurari et al., ). Tab. 4 presents the results of this evaluation. Despite PUMA’s relatively few 8B parameters and the use of an image encoder with $224\times 224$ resolution input, it demonstrates competitive and often superior image understanding performance compared to other unified understanding and generation models. Notably, PUMA’s performance on some metrics even surpasses that of understanding-only baselines. This performance can be attributed to PUMA’s use of multi-granular continuous visual tokens as input to the MLLM. A detailed ablation study examining the impact of different scale features as input on image understanding tasks is provided in the Appendix, offering further insights into the effectiveness of PUMA’s multi-granular approach.

### 4.7 Ablation

We conduct an ablation study to examine the impact of feature scale selection on tasks requiring fine-grained controllability. Fig. 9 compares the outputs of $f_{0}$ and $f_{1}$ feature scales for image editing and colorization tasks. The results demonstrate that $f_{1}$ scale features are insufficient for preserving crucial image details, while $f_{0}$ scale features maintain the necessary fine-grained information for precise manipulation tasks. More ablation studies are in the Appendix.

## 5 Conclusion

In this paper, we introduce PUMA, a novel unified multi-granular MLLM that unifies various granular tasks in visual generation and understanding. By leveraging multi-granular representations, PUMA effectively addresses the challenge of balancing diversity and controllability in image generation tasks. Our approach demonstrates superior performance across a spectrum of visual tasks, including diverse text-to-image generation, image editing, inpainting, colorization, conditional generation, and understanding. PUMA’s ability to adapt to varying granularity requirements within a single framework represents a significant advancement in MLLM capabilities. This work opens up new possibilities for more versatile and powerful multimodal AI systems, contributing to the broader goal of achieving artificial general intelligence in multimodal domains.

## Appendix A PUMA Training

### A.1 Visual Decoding Training

#### A.1.1 Dataset Details

For training the image decoding process, we leverage three large-scale datasets: Laion-2B (Schuhmann et al., ), Laion-Aesthetics (Burger, ), and JourneyDB (Sun et al., ). To ensure high-quality generation capabilities, we apply a resolution-based filtering criterion, selecting only images with resolutions of $512\times 512$ pixels or larger. We only use center crop as the data augmentation method.

#### A.1.2 Training Settings

We train five dedicated image decoders for the $f_{0}$, $f_{1}$, $f_{2}$, $f_{3}$, and $f_{4}$ scale features respectively. The image encoder is the frozen CLIP-L image encoder (Radford et al., ). Each image decoder is initialized from the SDXL model. The VAE (Kingma, ) remains frozen throughout the training process. The corresponding image features are input to the diffusion model through the cross-attention mechanism, replacing the original text embedding input. We train the decoders using AdamW optimizer (Loshchilov, ) with a maximum learning rate of 8e-5, using linear learning rate decay and a gradient clipping value of 1.0. The training batch size is 1,024. The training steps for the five features are $40,000$, $30,000$, $20,000$, $15,000$, and $10,000$ respectively, with features containing more visual tokens using longer training steps. We use noise off value of 0.1 and random drop of 10% of the input image to blank image for classifier-free guidance.

### A.2 MLLM Training

#### A.2.1 Training Objective

PUMA employs a unified framework with supervision on both text tokens and image features. For text tokens, we use cross-entropy classification loss, while for image features, we adopt MSE regression loss. To balance the contribution of text and image outputs, we apply a loss ratio of 0.02 for text and 1.0 for image features. Within the image feature regression loss, we use different ratios for the progressively generated 5 scales of image features ($f_{4}$, $f_{3}$, $f_{2}$, $f_{1}$, and $f_{0}$), with ratios of 1024.0, 512.0, 64.0, 8.0, and 1.0 respectively. This scaling compensates for the varying number of tokens at each feature scale, with larger ratios for scales with fewer tokens. The training loss objective remains consistent across both the pretraining and instruction tuning phases.

#### A.2.2 Pretraining Dataset Details

During PUMA’s pretraining phase, we utilize a diverse set of datasets including Laion-2B (Schuhmann et al., ), Laion-Aesthetics (Burger, ), GRIT (Peng et al., ), The Pile (Gao et al., ), OCR-VQA-200K (Mishra et al., ), and LLaVAR (Zhang et al., ). For the image-text pair data in Laion-2B, Laion-Aesthetics, and GRIT, we randomly assign 50% of the samples to text-to-image training and 50% to image-to-text training, fostering both image generation and understanding capabilities. We employ center crop as the primary image augmentation technique. To train on the GRIT dataset for object grounding, we append 224 additional position tokens to the MLLM’s codebook, representing object positions with bounding box coordinates [x_min, y_min, x_max, y_max]. We construct the training sequences by appending the tokens 
 and 
 to denote the beginning and end of each sequence. At the beginning and end of each image feature sequence, we include the special tokens [IMG] and [/IMG] to indicate the visual position.

#### A.2.3 Pretraining Settings

We conduct pretraining for 100K steps using the AdamW optimizer with a batch size of 2048. The maximum learning rates are set to 1e-4 for the projector and 3e-5 for the LLaMA backbone. We employ a 2,000-step warm-up period, cosine learning rate decay, and gradient clipping at 5.0 during pretraining. To optimize memory usage and computational efficiency, training is accelerated using DeepSpeed ZeRO Stage 3. The entire pretraining process is carried out on 256 NVIDIA V100 GPUs over a period of 10 days.

#### A.2.4 Instruct Tuning Settings

High-quality Text-to-Image Generation: We utilize Laion-Aesthetics (Burger, ) and JourneyDB (Sun et al., ) with a data ratio 1:1 to instruct tune the text-to-image generation model based on the previous pretraining checkpoint. We use training batch size 2048 and train for 20,000 steps with the max learning rate 1e-5, warm up 1,000 steps, and cosine learning rate decay. Random crop with fixed aspect ratio is adopted as the image augmentation.

Precise Image Manipulation: We train the image manipulation task with SEED-Edit Ge et al. (). It contains seven different operations: background alteration, comprehensive image changes, style alteration, object removal, object addition, localized modifications, and color/texture alterations. We train with batch size 1024 and train for 10,000 steps. The max learning rate is 1e-5, warm-up is 500 steps, and cosine learning rate decay is adopted. We apply random crop with fixed aspect ratio on the accordingly input image and output image. The sequence of the image manipulation sample is like “
[IMG]embedding of origin image[/IMG]instruct editing prompt[IMG]embedding of edited image[/IMG]
”.

Conditional Image Generation: We train on the subset of MultiGen-20M dataset (Qin et al., ) including canny-to-image, image inpainting, and colorization. We use the training batch size $1,024$ and train for $20,000$ steps. The max learning rate is 1e-5, warm-up is 500 steps, and cosine learning rate decay is adopted. We apply center crop as the image augmentation. The sequence of the conditional image generation is like “
[IMG]embedding of origin image[/IMG]instruct conditional generation prompt[IMG]embedding of edited image[/IMG]
”. The “instruct conditional generation prompt” contains the caption of the target image and with a 50% probability contain the task instruction like “Please convert the canny image to a natural image”.

Image Understanding: We train image understanding task on the subset of LLaVA-OneVision (Li et al., ) and Cambrain (Tong et al., ). Data about math/reasoning and cross-duplicated data in the two datasets are removed. We train with the batch size 512 and train all data for 1 epoch. The max learning rate is 1e-5 with the warm-up 500 steps. Cosine learning rate decay is adopted. We apply resizing as the image augmentation. Supervision is only applied to the output text tokens. We use the system message “A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions.”

## Appendix B Evaluation Details

### B.1 Image Reconstruction Evaluation

To evaluate the reconstruction performance of different scales of features and our baselines, we use the ImageNet validation set, comprising 50,000 images. Each image is resized to a rectangular shape before being input into each image encoder. We assess reconstruction precision by computing PSNRr and LPIPSr, which measure the difference between the reconstructed image and the original image.

Given the inherent randomness in the decoders, we measure reconstruction diversity by reconstructing each original image twice using different random seeds. We then calculate PSNRd and LPIPSd to quantify the difference between these two reconstructed images. Higher diversity is beneficial for downstream tasks such as text-to-image generation.

For PSNR and LPIPS evaluations, we use a resolution of $256\times 256$ to align with the evaluation settings in previous works. For LPIPS evaluation specifically, we employ AlexNet as the feature extractor.

### B.2 Text-to-image Generation Evaluation

We evaluate text-to-image generation on the COCO 30K validation set (Lin et al., ). We use CLIP-I and CLIP-T scores to measure the consistency between the generated image and the ground truth image and caption, respectively. CLIP-Base-32 serves as the feature extractor for these metrics. To assess generation diversity, we calculate LPIPSd between two images generated using the same input prompt but different random seeds. The LPIPSd measurement details are consistent with those described in Sec. B.1.

### B.3 Image Editing Evaluation

We evaluate image editing performance on the Emu-Edit benchmark Sheynin et al. (). To assess editing quality, we adopt CLIP-I, CLIP-T, and DINO scores. CLIP-I and DINO Caron et al. () scores measure the model’s ability to preserve elements from the source image, while CLIP-T reflects the consistency between the output image and the target caption. For the DINO score, we employ DINO-Small-16 as the feature extractor.

### B.4 Image Understanding Evaluation

For image understanding tasks, we use the same evaluation setting as LLaVA-v1.5 (Liu et al., ). During evaluation, we use the system message “A chat between a curious user and an artificial intelligence assistant. The assistant gives helpful, detailed, and polite answers to the user’s questions.”

**Table 5: Ablation of different visual token input on image understanding. The experiments are conducted on LLaVA-v1.5 setting with CLIP-Large-224 visual encoder.**
| Visual token type | Token number | MMB$\uparrow$ | MME$\uparrow$ | GQA$\uparrow$ | VQAv2(test)$\uparrow$ |
| --- | --- | --- | --- | --- | --- |
| $f_{4}$ | 1 | 56.8 | 1252.6 | 0.0 | 64.1 |
| $f_{3}$ | 4 | 58.3 | 1285.5 | 0.0 | 67.0 |
| $f_{2}$ | 16 | 61.5 | 1403.0 | 46.6 | 71.1 |
| $f_{1}$ | 64 | 63.6 | 1400.8 | 58.4 | 74.4 |
| $f_{0}$ | 256 | 65.4 | 1464.9 | 58.8 | 76.9 |
| $f_{4}$-$f_{0}$ | 341 | 65.1 | 1445.5 | 61.0 | 76.9 |

## Appendix C Ablation of Different Scale Features as Input on Image Understanding Task

Given that PUMA adopts a unified multi-granular image feature as both input and output for the MLLM backbone, we conducted an ablation study to investigate the influence of different scales of image feature input on image understanding tasks. For a fair comparison, we adopted the standard LLaVA-1.5-7B pretraining and finetuning setting, only changing the image encoder to a 224-input CLIP-Large with different granularities of features.

Tab. 5 presents the results of this ablation study. The findings demonstrate that finer-grained features generally lead to better performance in image understanding tasks. Notably, utilizing all image features from $f_{4}$ to $f_{0}$ (the PUMA setting) achieves comparable performance to using all 256 visual tokens of the finest scale ($f_{0}$). These results validate that the unified visual input and output format of PUMA provides a robust foundation of visual features for image understanding tasks, effectively balancing performance across different granularities.

## Appendix D Selection of 5 Scale Features in Text-to-image Generation

PUMA generates images at 5 granularity levels, allowing users to select the output that best meets their requirements. In our evaluation of diverse text-to-image generation, we produce 5 image outputs for each input prompt, corresponding to the 5 feature scales. To assess performance, we select the image with the highest CLIP-I and CLIP-T scores among the 5 outputs and compute the average maximum value. Tab. 6 presents the CLIP-I and CLIP-T scores for each of the 5 feature scales.

The results demonstrate that different granularity levels excel in various aspects of image generation. Notably, the ability to select the best output from multiple scales (PUMA 5-scale Max) yields significantly improved CLIP-I and CLIP-T scores compared to any single scale, highlighting the advantage of PUMA’s multi-granular approach.

Figure: Figure 10: Visualization of PUMA text-to-image outputs across five scale features given the generation prompt.
Refer to caption: x10.png

**Table 6: CLIP-I and CLIP-T scores on MSCOCO 30K validation set with different feature scales.**
| Model | Token num. | CLIP-I$\uparrow$ | CLIP-T$\uparrow$ |
| --- | --- | --- | --- |
| PUMA ($f_{4}$ scale) | 1 | 0.699 | 0.295 |
| PUMA ($f_{3}$ scale) | 4 | 0.703 | 0.300 |
| PUMA ($f_{2}$ scale) | 16 | 0.703 | 0.301 |
| PUMA ($f_{1}$ scale) | 64 | 0.693 | 0.299 |
| PUMA ($f_{0}$ scale) | 256 | 0.621 | 0.280 |
| PUMA (5-scale Max) | - | 0.736 | 0.317 |

## Appendix E Qualitative Results of Text-to-image Generation on Five Scale Features

In the text-to-image generation task, PUMA produces five distinct images corresponding to the five feature scales, all derived from a single input generation prompt. Fig. 10 presents samples of outputs across these five scales for given generation prompts.

## Appendix F More Qualitative Results

We present more qualitative cases for image reconstruction, diverse text-to-image generation, editing, and conditional image generation, as shown in Figures 11, 12, 13, 14 and 15.

Figure: Figure 11: More visualizations on multi-granular visual decoding from fine-grained to coarse-grained granularity.
Refer to caption: x11.png

Figure: Figure 12: More visualizations on fine-grained image reconstruction with $f_{0}$ scale feature.
Refer to caption: x12.png

Figure: Figure 13: More visualizations on text-to-image generation utilizing $f_{4}$ and $f_{3}$ scales.
Refer to caption: x13.png

Figure: Figure 14: More visualizations on image editing.
Refer to caption: x14.png

Figure: Figure 15: More visualizations on conditional image generation.
Refer to caption: x15.png

## Peer Reviews

confidence: 5
contribution: 1
initial_rating: 3
presentation: 4
questions: * I don't have any questions. I think addressing my first point would require major new results which would be only appropriate for a new submission. Unless I missed something major I am not sure how would this be addressed in the rebuttal. Perhaps a basic follow up to improve the paper would also be evaluating the pre-training performance of the presented model.
* I think however the multi-scale image feature contribution to be the most interesting part but does not seem to quite fit in the premise of the paper of having a unified generation and understanding model but seems a rather orthogonal contribution. If this was to be the main contribution of the paper, then the paper would need to be re-framed in terms of this contribution and then more ablations would be needed or a way to automatically select the feature scale as right now the best results are obtained through PUMA ( max of 5 scales) oracle performance.
soundness: 4
strengths: * The idea of multiscale features in the image generation domain for targeting a diverse array of tasks is original and useful. Some features at some scale might benefit one task while other features might be beneficial for other tasks as shown in some of the experiments.
* The paper makes a compelling argument for the need of unifying generation and understanding and represents an attempt in this direction.
* The paper is well written and well organized in general.
summary: The paper proposes a model that combines generation and understanding tasks, referred to as PUMA. The proposed model extracts features at multiple scales from a CLIP model and trains an array of diffusion models conditioned on scale-specific features. A multimodal LLM is optimized then to generate either text tokens or image features jointly using standard next word prediction or L2 distances between features (Eq 2).  The proposed model is pre-trained on an array of tasks that involve text-to-image and image-to-text generation. Then four dedicated models are fine-tuned and evaluated on four downstream tasks (Sec 3.4).
weaknesses: * The main concerns is that the paper does not fulfill what seems to be its main promise: To have a single integrated model that can perform generative and understanding tasks. The models that are evaluated for each of the four tasks are different model checkpoints that are produced by targeting each downstream task specifically through different finetuning datasets. 
* I also have some concern with not including FID (Frechet Inception Distance) as is customary to evaluate generative tasks. While this metric is not perfect it seems the paper is currently bypassing this measure of quality and mostly focusing on diversity and adherence to prompt.
* There are a couple of comparisons that could have been included to other recent models that provide evaluation results for the same checkpoint across understanding and generation and/or provide publicly available checkpoints for evaluation such as GILL (Koh et al NeurIPS 2023) -- mentioned in the paper, Unified-IO 2 (Lu et al CVPR 2024), and GenLLava (Hernandez et al 2024). These models are not finetuned for each type of tasks, generation, editing or understanding but are finetuned jointly to perform well across types of tasks.

---

confidence: 3
contribution: 3
initial_rating: 5
presentation: 3
questions: Please see weaknesses
soundness: 3
strengths: Originality:
To the best of my knowledge, this paper seems to be the first to combine multi-granularity image and text modeling. This makes it interesting and could spark some directions in the future.

Quality:
The thorough evaluations a presented in the experimental section are of fair quality which helps the reviewers make a better assessment of the contributions and significance of the paper.

Clarity:
The presentation of some of the ablations could use some work. Please see weaknesses section

Significance:
The significance of this paper given the results in somewhat average, but maybe clarifications and improved results could make it more significant?
summary: This paper proposes a method for multimodal visual generation and understanding at different levels of granularity. Concretely, the method proposes the use of image encodings at different resolutions combined with text tokens in a single autoregressive model called PUMA. Modeling image encodings at different granularities allows for more diverse image generation depending on the granularity of features from which the images are generated. In addition to the autoregressive model that generates the image encodings, there are multiple diffusion decoder models for each level of granularity in the generated encodings. In their experiments, the authors show the flexibility of using the different levels of generation granularity for different generation tasks such as text-to-image generation, editing, pinpointing, colorization and conditional generation. In addition, the authors show that the model is able to compete with other methods in understanding tasks
weaknesses: * Understanding is negatively affected by multi-granularity modeling.
Looking at Table 5 in the appendix, it is not clear whether multi-granularity modeling adds to the understanding ability of the model. For half of the metrics the performance becomes worse, for one is the same, and only for a single metric the performance actually seems to improve. Do the authors have an intuition of why this is the case? Is spending part of the compute budget in the coarse level features the reason for this? Or is it the features themselves that confuse the model.


* Understanding ablations.
It is not clear how the ablations in appendix C were done. Did the authors train multiple models with encodings from only a single granularity? Or did the authors remove the encodings at other granularities at inference time to evaluate on how well the understanding performs when given a single granularity encodings?


* Reconstruction performance.
In the provided evaluations, the proposed method outperforms previous models in terms of reconstruction given image encodings at the finest granularity. Is this an effect of the multi-granularity learning? Or is it just an effect of the encoder / decoder combination. If it's just the encoder / decoder combination, does this result validate the proposed multi-granular idea of the paper?



* Use of multiple decoders.
Is using multiple decoders necessary? Did the authors try using a conditioning signal to tell the diffusion decoders what type of inputs are being fed for generation?

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please refer to weaknesses.
soundness: 2
strengths: 1. The idea of multi-granularity for image generation is interesting. 
2. The visualization of multi-granularity images is clear and the performance looks good. 
3. The paper is well written and easy to follow.
summary: This paper proposes a unified MLLM, PUMA, which uses multi-granularity visual features as input and output to handle the different granularity requirements of various image generation tasks.
weaknesses: 1. The novelty of this paper seems limited; although it claims to introduce multi-granular visual generation, many papers (e.g., SEED-X[1], Matryoshka[2]) already focus on multi-granularity visual features, both in MLLM and image generation. Furthermore, the implementation of multi-granularity merely involves using different pooled visual features and different diffusion decoders, which also introduces additional parameters and computation.
2. In Table 1, is PUMA using (1+4+16+64+256) visual tokens? This number is much larger than in previous works. When using the same number of visual tokens, PUMA appears to underperform compared to other methods.
3. In Figure 2, the input and output of the PUMA Multimodal Generative Model are visual features covering different levels of visual granularity. I wonder whether the model can perform single granularity image generation during the inference stage (or human-specified multi-granularity).
4. In Tables 3 and Table 4, the authors should indicate how many visual tokens each model uses.
 
 
[1] Ge, Yuying, et al. "Seed-x: Multimodal models with unified multi-granularity comprehension and generation." arXiv preprint arXiv:2404.14396 (2024).
[2] Cai, Mu, et al. "Matryoshka Multimodal Models." arXiv preprint arXiv:2405.17430 (2024).

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: The current feature pyramid is constructed based on spatially averaging features from previous finer levels.  This approach means information in f_4, f_5 is already represented in low level features f_0. I want to learn from the authors on their opinion towards a possibly more compact, top down representation where only information not present in f_5 are encoded in f_4 and recursively to lower levels.  

In terms of generation diversity, there are existing approaches in diffusion models that can increase the output diversity. No need for experiments, but would be great to learn authors' opinion on using these approaches to address diversity needs, e.g. https://arxiv.org/abs/2310.17347, https://arxiv.org/abs/2310.12583
soundness: 2
strengths: The paper clearly stated the problem, which is a common challenge for MLLM that needs to balance the different needs of multiple visual generation tasks. Solving such challenge will definitely help improve the overall capability of the MLLMs. 

The proposed solution is relatively simple (in a good way) and clean, and is in principle straightforward to implement. Experiments show that the proposed approach is also capable of solving the proposed problem, which helps both high diversity and high control needs. 

The paper is clearly written with details on both training and evaluations in appendix. Shouldn't be too difficult to re-implement by 3rd party researchers.
summary: This paper address the problem of achieving both strong control and strong diversity in image generation tasks powered by MLLM with diffusion model as image decoder.
To address this problem, the paper propose to represent image using multi granular embeddings, which conceptually is similar to an image pyramid, and train MLLM to predict embeddings at all scales in autoregressive manner. The predicted fine grained representation can be used in diffusion based image decoder to generate images following strong input condition, such as in image editing, while the gross grained one can be used to generate images with diversity.
weaknesses: The simple design will lead to a model that's slow in inference time: 
a),  the image feature sequence length is essentially doubled with a lot of redundant information. Leading to a larger context window in inference time, and 2x inference cost as the output sequence length will be doubled for visual generation tasks.
b) several images has to be generated, each from a different diffusion decoder for the corresponding feature granularity. In the end, only one of these outputs will be used as the final output.

The result final model size will be much larger as well since there will be several SDXL UNet checkpoints, each is around 2B parameters.

In experiments, the paper fine-tunes separate MLLMs for different visual-linguistic tasks. This setup is inconsistent with the problem formulation, where one MLLM has to balance the needs of several tasks.

The authors did not discuss how to select the feature scale for final output. This is vaguely discussed in Appendix D. It should be more clearly discussed as it is an important part of the entire generation flow.