Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis

## Abstract

Abstract Diffusion models, such as Stable Diffusion, have made significant strides in visual generation, yet their paradigm remains fundamentally different from autoregressive language models, complicating the development of unified language-vision models. Recent efforts like LlamaGen have attempted autoregressive image generation using discrete VQVAE tokens, but the large number of tokens involved renders this approach inefficient and slow. In this work, we present Meissonic, which elevates non-autoregressive masked image modeling (MIM) text-to-image to a level comparable with state-of-the-art diffusion models like SDXL. By incorporating a comprehensive suite of architectural innovations, advanced positional encoding strategies, and optimized sampling conditions, Meissonic substantially improves MIM’s performance and efficiency. Additionally, we leverage high-quality training data, integrate micro-conditions informed by human preference scores, and employ feature compression layers to further enhance image fidelity and resolution. Our model not only matches but often exceeds the performance of existing models like SDXL in generating high-quality, high-resolution images. Extensive experiments validate Meissonic’s capabilities, demonstrating its potential as a new standard in text-to-image synthesis. We release a model checkpoint capable of producing 1024 × 1024 1024 1024 1024\times 1024 resolution images.

## 1 Introduction

Large Language Models (LLMs) (Touvron et al., ; Bai et al., ; Su et al., ) have showcased amazing advancements in natural language processing (NLP) tasks by adhering to established scaling laws. The combination of next-token prediction pre-training and supervised fine-tuning, has unlocked the potential of LLMs across a diverse range of language tasks, suggesting that similar methodologies might be applicable to text-to-image synthesis (T2I).

Simultaneously, diffusion models, such as Stable Diffusion (Rombach et al., ; Podell et al., ; Desync, ; Art, ), have rapidly advanced to become the dominant paradigm in visual generation. However, their operation significantly diverges from autoregressive models, posing challenges in achieving a unified approach to both language and vision tasks. Some effects have explored bridging this gap through connectors between LLMs and diffusion models.

Recent developments like LlamaGen (Sun et al., ) have ventured into autoregressive image generation using discrete image tokens derived from VQVAE (Yu et al., ). Despite progress, the substantial number of image tokens compared to text tokens makes autoregressive generation inefficient.
For example, tokenizing one $1024\times 1024$ image using a $16\times$ downsampled VQVAE yields 4096 tokens, where a sequential generation process is prohibitively slow.

On the other hand, MaskGIT (Chang et al., ) introduced a more efficient, non-autoregressive alternative, where all image tokens are predicted simultaneously in a parallel, iterative refinement process.
Then, MUSE (Chang et al., ) extended this technique to higher resolutions, achieving $512\times 512$ resolution T2I generation.
These non-autoregressive methods offer around 99% reduction in decoding steps compared to autoregressive methods.
However, despite their efficiency, non-autoregressive transformers remain limited in performance compared to advancing diffusion or autoregressive models, particularly in high-quality, high-resolution text-to-image synthesis.

In this work, we address these challenges and introduce two key innovations to make masked image modeling (MIM) competitive with advanced diffusion models:

Enhanced Transformer Architecture:
Previous MIM methods (Chang et al., ; ) predominantly utilized naive transformer architectures, potentially limiting their capabilities. We discovered that a combination of multi-modal and single-modal transformer layers can significantly boost MIM training efficiency and performance.
Language and vision representations are inherently different.
The multi-modal transformer can effectively capture cross-modal interactions, extracting information from unpooled text representations and effectively bridging the gap between these distinct modalities. This allows the model to harness useful signals from noisy data.
Additionally, subsequent single-modal transformer layers refine the visual representation, improving performance and training stability.
Empirically, a $1:2$ ratio between these two types of transformer layers yields optimal performance.

Advanced Positional Encoding & Masking Rate as Sampling Condition:
We incorporate Rotary Position Embedding (RoPE) (Su et al., ) for encoding positional information in queries and keys, which helps maintain detail in high-resolution images. RoPE effectively addresses the issue of context disassociation in transformers as the number of tokens increases. Traditional absolute positional encoding methods lead to distortions and loss of detail at $512\times 512$ resolutions, whereas RoPE significantly mitigates these issues. Additionally, we introduce the masking rate as a dynamic sampling condition throughout the generation process. Previous MIM methods (Chang et al., ; ) have overlooked this aspect, resulting in suboptimal image details. This issue arises because the number of tokens predicted by the MIM model changes dramatically throughout the sampling loop. With the masking rate condition, the model can ascertain the current stage of the sampling period by leveraging conditional information from the masking rate. Note that merely relying on attention masks is insufficient to bridge this gap. We achieve effective conditional encoding by discretizing the continuous masking rate into 1000 levels. This approach enables the model to adapt to different stages of the sampling process, significantly improving image detail and overall quality.

Beyond these architectural improvements, to achieve comparable performance with SDXL for high-resolution generation, we adopt effects in three additional aspects:

High-Quality Training Data: The quality of training data is crucial. While LAION (Schuhmann et al., ) offers a diverse visual dataset, its captions can be subpar (Chen et al., ). We curated a high-quality internal dataset with accurate captions, which, combined with our training strategy, significantly improved the generative capabilities of the base model.

Micro-Conditioning: We identified that incorporating original image resolution, crop coordinates, and human preference score (Wu et al., ) as micro-conditions greatly enhances model stability during high-resolution aesthetic training.

Feature Compression Layers: To efficiently generate high-resolution images, we integrated feature compression layers, maintaining computational efficiency even at $1024\times 1024$ resolution.

Our contributions culminate in Meissonic, a next-generation T2I model based on masked discrete image token modeling. Unlike larger diffusion models such as SDXL (Podell et al., ) and DeepFloyd-XL (Liu et al., ), Meissonic, with just 1B parameters, offers comparable or superior $1024\times 1024$ high-resolution, aesthetically pleasing images while being able to run on consumer-grade GPUs with only 8GB VRAM without the need for any additional model optimizations.
Moreover, Meissonic effortlessly generates images with solid-color backgrounds, a feature that usually demands model fine-tuning or noise offset adjustments in diffusion models.

Advancement of Meissonic represents a significant stride towards high-resolution, efficient, and accessible T2I MIM models. We evaluate Meissonic using various qualitative and quantitative metrics, including HPS, MPS, GenEval benchmarks, and GPT4o assessments, demonstrating its superior performance and efficiency.

## 2 Method

### 2.1 Motivation

Recent breakthroughs in text-to-image synthesis have been largely propelled by diffusion models, such as Stable Diffusion XL, which have set de facto standards for image quality, detail, and conceptual fidelity.

However, these models operate significantly differently from autoregressive language models, posing challenges in achieving a unified approach for vision and language tasks. This divergence not only complicates the integration of these modalities but also highlights the need for innovative methods that can bridge the gap between them. In particular, non-autoregressive Masked Image Modeling (MIM) techniques, exemplified by MaskGIT and MUSE, have shown potential for efficient image generation. Yet, despite their promise, MIM approaches face two critical limitations:

(a) Resolution Constraint. Current MIM methods are limited to generating images at a maximum resolution of $512\times 512$ pixels. This limitation hinders their broader adoption and advancement, particularly as the text-to-image synthesis community increasingly adopts $1024\times 1024$ resolution as the standard.

(b) Performance Gap. Existing MIM techniques have not yet achieved the level of performance exhibited by leading diffusion models like SDXL. They notably underperform in key areas such as image quality, intricate detailing, and conceptual representation, which are critical for practical applications.

These challenges necessitate the exploration of new approaches. Our objective is to empower MIM to efficiently generate high-resolution images (e.g., $1024\times 1024$), while narrowing the gap with top-tier diffusion models, and ensuring computational efficiency suitable for consumer-grade hardware.

Through our work, Meissonic, we aim to push the boundaries of MIM methods and bring them to the forefront of text-to-image synthesis.

### 2.2 Model Architecture

The Meissonic model is architected to facilitate efficient high-performance text-to-image synthesis through an integrated framework comprising a CLIP text encoder (Radford et al., ), a vector-quantized (VQ) image encoder and decoder (Esser et al., ), and a multi-modal Transformer backbone. Figure [2](#S2.F2) illustrates the overall structure of the model.

Figure: Figure 2: The architecture of Meissonic. During the image generation process, discrete tokens are created randomly according to a predefined schedule. Meissonic then applies masking and performs predictions over several steps to reconstruct all tokens and decode the resulting image. In the case of image editing, the original image is converted into discrete tokens, which are masked according to a specified masking strategy. After a series of processing steps, the masked tokens are reconstructed and utilized to decode the target image. Text prompt and other conditions are incorporated to control the synthesis process. $R$ represents the masking rate condition, and $C$ represents the micro conditions. $Comp.$ and $Decomp.$ denotes feature compression layers and feaure decompression layers, respectively. More details about Multi-modal Transformer Block can be found in Appendix [E](#A5).
Refer to caption: /html/2410.08261/assets/x2.png

Vector-quantized Image Encoder and Decoder. We employ a VQ-VAE model (Esser et al., ) to convert raw image pixels into discrete semantic tokens. This model comprises an encoder, a decoder, and a quantization layer that maps input images into sequences of discrete tokens using a learned codebook. For an image of size $H\times W$, the encoded token size is $\frac{H}{f}\times\frac{W}{f}$, where $f$ represents the downsampling ratio. In our implementation, we utilize a downsampling ratio of $f=16$ and a codebook size of 8192, allowing a $1024\times 1024$ image to be encoded into a sequence of $64\times 64$ discrete tokens.

Flexible and Efficient Text Encoder. Instead of using large language model encoders, such as T5-XXL(^1^11Many works indicate that the T5 text encoder is the key factor in obtaining the ability to synthesize words, we still show the ability to synthesize letters in Figure [8](#A2.F8). We leave this a future improvement.) (Raffel et al., ) or LLaMa (Touvron et al., ), which are prevalent in previous works (Chen et al., ; Esser et al., ), we utilize a single text encoder from the state-of-the-art CLIP model with a latent dimension of 1024, and fine-tune for optimal T2I performance. While this decision may limit the model’s capacity to fully comprehend lengthy text prompts, our observations indicate that excluding large-scale text encoders like T5 does not diminish visual quality. Moreover, this approach significantly reduces GPU memory requirements and computational cost. Notably, offline extraction of T5 features would entail approximately 11 times more processing time and 6 times more storage than employing the CLIP text encoder, underscoring the efficiency of our design.

Multi-modal Transformer Backbone for Masked Image Modeling. Our transformer architecture builds upon the Multi-modal Transformer framework (Sauer et al., ), incorporating sampling parameters $r$ to encode sampling parameters and Rotary Position Embeddings (RoPE) (Su et al., ) for spatial information encoding. We introduce feature compression layers to efficiently handle high-resolution generation with numerous discrete tokens. These layers compress embedding features from $64\times 64$ to $32\times 32$ before processing through the transformer, and followed by feature decompression layers to $64\times 64$, thereby alleviating computational burdens. To enhance training stability and mitigate the NaN Loss issue, we follow the training strategy from LLaMa (Touvron et al., ), implementing gradient clipping and checkpoint reloading during distributed training and integrating QK-Norm layers into the architecture. We elaborate on the designs of our transformer in the subsequent section.

Diverse Micro Conditions. To augment generation performance, we incorporate additional conditions such as original image resolution, crop coordinates, and human preference score (Wu et al., ). These conditions are transformed into sinusoidal embeddings and concatenated as additional channels to the final pooled hidden states of the text encoder.

Masking Strategy. Following the approach established in Chang et al. (), we employ a variable masking ratio with cosine scheduling. Specifically, we randomly sample a masking ratio $r\in[0,1]$ from a truncated $arccos$ distribution characterized by the following density function:

$$ $p(r)=\frac{2}{\pi}(1-r^{2})^{-\frac{1}{2}}$ $$

In contrast to autoregressive models that learn conditional distributions $P(x_{i}\mid x_{<i})$ for fixed token orders, our approach utilizes random masking with variable ratios to enable the model to learn $P(x_{i}\mid x_{\Lambda})$ for arbitrary subsets of tokens $\Lambda$. This flexibility is pivotal for our parallel sampling strategy and facilitates various zero-shot image editing capabilities, which will be demonstrated in Section [3](#S3).

### 2.3 Multi-modal Transformer For Masked Image Modeling

Meissonic employs the Multi-modal Transformer as its foundational architecture and innovatively customizes the modules to address the distinctive challenges inherent in high-resolution masked image modeling. We introduce several specialized designs for MIM as follows:

- •
Rotary Position Embeddings. RoPE (Su et al., ) has demonstrated exceptional performance within in LLMs (Su et al., ; Touvron et al., ; Ding et al., ; Bai et al., ). Some studies (Lu et al., ; Lin et al., ; Zhuo et al., ) have attempted to extend 1D RoPE (Su et al., ) to 2D or 3D for image diffusion models. Our findings reveal that, due to the high-quality image tokenizer used for converting images into discrete tokens, the original 1D RoPE yields promising results. This 1D RoPE facilitates a seamless transition from the $256\times 256$ stage to the $512\times 512$ stage, simultaneously enhancing the generative performance of the model.
- •
Deeper Model with Single-modal Transformer. Although the Multi-modal Transformer block demonstrated commendable performance, our experiments reveal that reducing the number of multi-modal blocks to a single-modal block configuration offers a more stable and computationally efficient approach for training T2I models. Therefore, we opt to employ Multi-modal Transformer blocks in the initial stages of the network, transitioning to exclusively Single-modal Transformer blocks in the latter half. Our findings suggest an optimal block ratio of about 1:2.
- •
Micro Conditions with Human Preference Score. Our experiments reveal that incorporating three micro-conditions is pivotal for achieving a stable and reliable High-resolution MIM Model: original image resolution, crop coordinates, and human preference score. The original image resolution effectively aids the model in implicitly filtering out low-quality data and learning the properties of high-quality, high-resolution data, while crop coordinates enhance training stability, likely due to improved consistency between image conditions and semantic conditions during cropped patch coordination. In the final stage, we leverage the Human Preference Score (Wu et al., ) to effectively enhance image quality, using signals provided by the Human Preference Model to guide the model’s outputs in mimicking and approximating human preferences.
- •
Feature Compression Layers. Existing multi-stage approaches, such as MUSE (Chang et al., ) and DeepFloyd-XL (DeepFloyd, ), employ cascading multiple subnetworks to achieve higher-resolution image generation. We argue that such multi-stage training introduces unnecessary complexity and hampers the generation of high-fidelity, high-resolution images. Instead, we advocate integrating streamlined feature compression layers during the fine-tuning stage to facilitate efficient high-resolution generation process learning. This approach functions akin to a lightweight high-resolution adapter (Guo et al., ), a module extensively explored and integrated within Stable Diffusion. By incorporating 2D convolution-based feature compression layers into the transformer backbone, we compress the feature maps prior to the transformer layers and subsequently decompress them after the transformer layers, effectively addressing the challenges of efficiency and resolution transition.

### 2.4 Training Details

**Table 1: Comparison of training data and time for various models.**
| Model | Params<br>(B) | Training<br>Images (M) | 8$\times$A100 GPU<br>Days^a |
| --- | --- | --- | --- |
| Würstchen (Pernias et al., ) | 1.0 | $1420$ | $128.1$ |
| SD-1.5 (Rombach et al., ) | 0.9 | $4800$ | $781.2$ |
| SD-2.1 (Rombach et al., ) | 0.9 | $3900$ | $1041.6$ |
| Imagen (Saharia et al., ) | 3.0 | $860$ | $891.5$ |
| Dall-E 2 (Ramesh et al., ) | 6.5 | $650$ | $5208.3$ |
| GigaGAN (Kang et al., ) | 0.9 | $980$ | $597.8$ |
| SDXL (Podell et al., ) | 2.6 | unknown | unknown |
| Meissonic | 1.0 | 210 | 19^b |

Meissonic is constructed using a CLIP-ViT-H-14(^2^22We utilize “laion/CLIP-ViT-H-14-laion2B-s32B-b79K” from OpenCLIP as our initial weights.) text encoder (Ilharco et al., ), a pre-trained VQ image encoder and decoder (Patil et al., ), and a customized Transformer-based (Esser et al., ) backbone.
We employ classifier-free guidance (CFG) (Ho & Salimans, ) and cross-entropy loss to train Meissonic.
Training occurs across three resolution stages, leveraging both public datasets and our curated data.
First, we train Meissonic-256 with a batch size of 2,048 for 100,000 steps. Second, we continue training Meissonic-512 with a batch size of 512 for an additional 100,000 steps. Third, we continue training Meissonic with a batch size of 256 for 42,000 steps with a resolution of $1024\times 1024$. The performance results of Meissonic-512 and Meissonic are reported in Table [2](#S2.T2).
All experiments are carried out with a fixed learning rate of $1\times 10^{-4}$ except Stage 4.
Further details are elaborated in Section [2.5](#S2.SS5). All inferences in this paper are performed with CFG = 9 and 48 steps.

It’s crucial to highlight the resource efficiency of our training process. Our training is considerably more resource-efficient compared to Stable Diffusion (Podell et al., ). Meissonic is trained in approximately 48 H100 GPU days, demonstrating that a production-ready image synthesis foundation model can be developed with considerably reduced computational costs. Additional details on this comparison can be found in Table [1](#S2.T1).

**Table 2: HPS v2.0 benchmark. Scores are collected from [https://github.com/tgxs002/HPSv2](https://github.com/tgxs002/HPSv2). We highlight the best.**
|  | HPS v2.0 |  |  |  |  |
| --- | --- | --- | --- | --- | --- |
| Model | Animation | Concept-art | Painting | Photo | Averaged |
| GLIDE (Nichol et al., ) | $23.34$ | $23.08$ | $23.27$ | $24.50$ | $23.55$ |
| LAFITE (Zhou et al., ) | $24.63$ | $24.38$ | $24.43$ | $25.81$ | $24.81$ |
| VQ-Diffusion (Gu et al., ) | $24.97$ | $24.70$ | $25.01$ | $25.71$ | $25.10$ |
| Latent Diffusion (Rombach et al., ) | $25.73$ | $25.15$ | $25.25$ | $26.97$ | $25.78$ |
| DALL·E mini | $26.10$ | $25.56$ | $25.56$ | $26.12$ | $25.83$ |
| VQGAN + CLIP (Esser et al., ) | $26.44$ | $26.53$ | $26.47$ | $26.12$ | $26.39$ |
| CogView2 (Ding et al., ) | $26.50$ | $26.59$ | $26.33$ | $26.44$ | $26.47$ |
| Versatile Diffusion (Xu et al., ) | $26.59$ | $26.28$ | $26.43$ | $27.05$ | $26.59$ |
| DALL·E 2 (Ramesh et al., ) | $27.34$ | $26.54$ | $26.68$ | $27.24$ | $26.95$ |
| Stable Diffusion v1.4 (Rombach et al., ) | $27.26$ | $26.61$ | $26.66$ | $27.27$ | $26.95$ |
| Stable Diffusion v2.0 (Rombach et al., ) | $27.48$ | $26.89$ | $26.86$ | $27.46$ | $27.17$ |
| Epic Diffusion | $27.57$ | $26.96$ | $27.03$ | $27.49$ | $27.26$ |
| DeepFloyd-XL (DeepFloyd, ) | $27.64$ | $26.83$ | $26.86$ | $27.75$ | $27.27$ |
| Openjourney | $27.85$ | $27.18$ | $27.25$ | $27.53$ | $27.45$ |
| MajicMix Realistic | $27.88$ | $27.19$ | $27.22$ | $27.64$ | $27.48$ |
| ChilloutMix | $27.92$ | $27.29$ | $27.32$ | $27.61$ | $27.54$ |
| Deliberate (Desync, ) | $28.13$ | $27.46$ | $27.45$ | $27.62$ | $27.67$ |
| SDXL Base 0.9 (Podell et al., ) | $28.42$ | $27.63$ | $27.60$ | $27.29$ | $27.73$ |
| Realistic Vision (SG_161222, ) | $28.22$ | $27.53$ | $27.56$ | $27.75$ | $27.77$ |
| SDXL Refiner 0.9 (Podell et al., ) | $28.45$ | $27.66$ | $27.67$ | $27.46$ | $27.80$ |
| Dreamlike Photoreal 2.0 (Art, ) | $28.24$ | $27.60$ | $27.59$ | $27.99$ | $27.86$ |
| SDXL Base 1.0 (Podell et al., ) | $28.88$ | $27.88$ | $27.92$ | $28.31$ | $28.25$ |
| SDXL Refiner 1.0 (Podell et al., ) | $28.93$ | $27.89$ | $27.90$ | $28.38$ | $28.27$ |
| Meissonic-512 | $28.90$ | $28.15$ | $28.22$ | $28.04$ | $28.33$ |
| Meissonic | 29.57 | 28.58 | 28.72 | 28.45 | 28.83 |

### 2.5 Progressive and Efficient Training Stage Decomposition

Our approach systematically decomposes the training process into four carefully designed stages, allowing us to progressively build and refine the model’s generative capabilities. These stages, combined with precise enhancements to specific components, contribute to continual improvements in synthesis quality. Given that SDXL has not disclosed details regarding its training data, our experience is particularly valuable for guiding the community in constructing SDXL-level text-to-image models. We present images generated by Meissonic at each of the four training stages in Appendix [G](#A7) to support our claims.

Figure: Figure 3: Qualitative Comparisons with SD 1.5, SD 2.1, DeepFloyd-XL, Deliberate, and SDXL.
Refer to caption: /html/2410.08261/assets/x3.png

Stage 1: Understanding Fundamental Concepts from Extensive Data. Previous studies (Chen et al., ; Yu et al., ) indicate that raw captions from LAION are insufficient for training text-to-image models, often requiring the caption refinement provided by MLLMs such as LLaVA (Liu et al., ). However, this solution is computationally demanding and time-intensive. While some studies (Chen et al., ; Sehwag et al., ) utilize the extensively annotated SA-10M (Kirillov et al., ) dataset, our findings reveal that SA-10M does not comprehensively cover fundamental concepts, particularly regarding human faces. Thus, we adopt a balanced strategy that leverages the original high-quality LAION data for foundational concepts learning in the initial training phase, utilizing a reduced resolution to enhance efficiency. Specifically, we carefully curated the deduplicated LAION-2B dataset by filtering out images with aesthetic scores below 4.5, watermark probabilities exceeding 50%, and other criteria outlined in Kolors (). This meticulous selection resulted in approximately 200 million images, which were employed for training at a resolution of $256\times 256$ in this initial stage.

Stage 2: Aligning Text and Images with Long Prompts.
In the first stage, our approach does not rely on high-quality image-text paired data. Therefore, in the second stage, we focus on improving the model’s capability to interpret long, descriptive prompts. We filtered the initial LAION set more rigorously, retaining only images with aesthetic scores above 8, and other criteria outlined in Kolors (). Additionally, we incorporate 1.2 million synthetic image-text pairs with refined captions exceeding 50 words, primarily derived from publicly available high-quality synthetic datasets, complemented by additional high-quality images from our internal 6 million dataset. This aggregation results in around 10 million image-text pairs. Notably, we maintain the model architecture while increasing the training resolution to $512\times 512$, enabling the model to capture more intricate image details. We observed a significant boost in the model’s ability to capture abstract concepts and respond accurately to complex prompts, including diverse styles and fantasy characters.

Stage 3: Mastering Feature Compression for Higher-resolution Generation.
High-resolution generation remains an unexplored area within MIM (Chang et al., ; ; Patil et al., ).
Unlike methods such as MUSE (Chang et al., ) or DeepFloyd-XL (DeepFloyd, ), which rely on external super-resolution (SR) modules, we demonstrate that efficient $1024\times 1024$ generation is feasible through
feature compression for MIM. By introducing feature compression layers, we achieve a seamless transition from $512\times 512$ to $1024\times 1024$ generation with minimal computational cost. In this stage, we further refine the dataset by filtering based on resolution and aesthetic score, selecting approximately 100K high-quality, high-resolution image-text pairs from the LAION subset utilized in Stage 2. This, combined with the remaining high-quality data, results in approximately 6 million samples for training at $1024$ resolution.

Stage 4: Refining High-Resolution Aesthetic Image Generation.
In the final stage, we fine-tune the model using a small learning rate, without freezing the text encoder, and incorporate human preference score as a micro condition. This can significantly enhance the model’s performance in high-resolution image generation. This targeted adjustment significantly enhances the model’s performance in generating high-resolution images, while also improving diversity. The training data remains the same as in Stage 3.

**Table 3: GenEval benchmark. We highlight the best result.**
| Model | Overall | Objects | Counting | Colors | Position | Color Attribution |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Single | Two |  |  |  |  |  |  |
| DALL-E mini | 0.23 | 0.73 | 0.11 | 0.12 | 0.37 | 0.02 | 0.01 |
| SD v1.5 | 0.43 | 0.97 | 0.38 | 0.35 | 0.76 | 0.04 | 0.06 |
| SD v2.1 | 0.50 | 0.98 | 0.51 | 0.44 | 0.85 | 0.07 | 0.17 |
| DALL-E 2 | 0.52 | 0.94 | 0.66 | 0.49 | 0.77 | 0.10 | 0.19 |
| SD XL | 0.55 | 0.98 | 0.74 | 0.39 | 0.85 | 0.15 | 0.23 |
| Meissonic | 0.54 | 0.99 | 0.66 | 0.42 | 0.86 | 0.10 | 0.22 |

Table: Table 4: GPU Memory Cost for for Different Models and Batch Sizes.

## 3 Results

### 3.1 Quantative Comparison

**Table 6: MPS scores on RealUser-800 Prompts. We highlight the best result.**
| Model | MPS |
| --- | --- |
| VQ-Diffusion (Gu et al., ) | 9.70 |
| Latent Diffusion (Rombach et al., ) | 10.56 |
| DALL·E mini (Dayma et al., ) | 11.32 |
| VQGAN + CLIP (Esser et al., ) | 11.50 |
| CogView2 (Ding et al., ) | 12.39 |
| Versatile Diffusion (Xu et al., ) | 12.61 |
| Stable Diffusion v1.4 (Rombach et al., ) | 13.89 |
| Stable Diffusion v2.0 (Rombach et al., ) | 14.39 |
| DeepFloyd-XL (DeepFloyd, ) | 15.22 |
| SDXL Base 0.9 (Podell et al., ) | 16.37 |
| SDXL Refiner 0.9 (Podell et al., ) | 16.64 |
| SDXL Base 1.0 (Podell et al., ) | 16.46 |
| SDXL Refiner 1.0 (Podell et al., ) | 16.56 |
| Meissonic | 17.34 |

Classic evaluation metrics for image generation models, such as FID and CLIP Score, have limited relevance to visual aesthetics, as highlighted by Podell et al. (); Chen et al. (); Kolors (); Sehwag et al. (). Therefore, we report our model’s performances using Human Preference Score v2 (HPSv2) (Wu et al., ), GenEval (Ghosh et al., ), and Multi-Dimensional Human Preference Score (MPS)(^3^33 Given that the KolorsPrompts benchmark was unavailable, we curated a diverse prompt dataset consisting of 800 real user-generated prompts spanning various concepts and themes for the MPS evaluation.) (Zhang et al., ),
as illustrated in Table [2](#S2.T2),[3](#S2.T3),[6](#S3.T6).

In our pursuit of making Meissonic accessible to the broader community, we optimized our model to 1 billion parameters, ensuring that it runs efficiently on 8GB VRAM, making inference and fine-tuning both convenient. Figure [5](#S2.T5) provides a comparative analysis of GPU memory consumption(^4^44GPU memory usage was gauged using torch.cuda.memory_reserved(). While this method might yield higher values, all models are measured under identical settings to maintain fairness.) across different inference batch sizes against SDXL. Additionally, Figure [5](#S2.T5)
details the inference time per step(^5^55Inference time is assessed using an A100 GPU with fp16 models. Notably, the reported times contributions from the VAE and text encoder, meaning that multi-step inferences do not scale linearly.).

Figure: Figure 4: GPT4o Preference Evaluation of Meissonic against current open Text-to-image Models.
Refer to caption: /html/2410.08261/assets/x6.png

We also present qualitative comparisons of image quality and text-image alignment in Figure [3](#S2.F3), with additional comparisons provided in the Appendix [I](#A9).
Furthermore, Figure [5](#S3.F5) illustrates Meissonic’s proficiency in generating text-driven style art image.

To complement these analyses, we conduct human evaluation by K-Sort Arena (Li et al., ) with internal checkpoint, we also conduct GPT-4o to evaluate the performance between Meissonic and other models in Figure [4](#S3.F4).

All Figures and Tables demonstrate that Meissonic achieves competitive performance in human performance and text alignment compared to DALL-E 2 and SDXL, as well as showcasing its efficiency.

Figure: Figure 5: Evaluating the ability to generate diverse styles. The enlarged samples of (d) Meissonic are provided in Appendix [H](#A8). Prompt: A garden full of [Y] illustrated in [X] style.
Refer to caption: /html/2410.08261/assets/x7.png

### 3.2 Zero-shot Image-to-Image Editing

**Table 7: Results on the EMU-Edit (Sheynin et al., ) test set. We highlight the best result.**
| Model | CLIP-I$\uparrow$ | CLIP-T$\uparrow$ | DINO$\uparrow$ |
| --- | --- | --- | --- |
| InstructPix2Pix (Brooks et al., ) | 0.834 | 0.219 | 0.762 |
| MagicBrush (Zhang et al., ) | 0.838 | 0.222 | 0.776 |
| PnP (Tumanyan et al., ) | 0.521 | 0.089 | 0.153 |
| Null-Text Inv. (Mokady et al., ) | 0.761 | 0.236 | 0.678 |
| EMU-Edit (Sheynin et al., ) | 0.859 | 0.231 | 0.819 |
| Meissonic | 0.871 | 0.266 | 0.760 |

Figure: Figure 6: Examples of image editing with mask on internal Image Editing Dataset
Refer to caption: /html/2410.08261/assets/x8.png

For image editing tasks, we benchmark Meissonic against state-of-the-art models using the EMU-Edit dataset (Sheynin et al., ), which includes seven different operations: background alteration, comprehensive image changes, style alteration, object removal, object addition, localized modifications, and color/texture alterations. We present results in Table [7](#S3.T7). Additionally, examples from our internal image editing dataset(^6^66This dataset will be released in a separate work.), including mask-guided editing in Figure [6](#S3.F6) and mask-free editing in Figure [7](#S3.F7), further showcase Meissonic’s versatility. Remarkably, Meissonic achieved this performance without any training or fine-tuning on image editing-specific data or instruction dataset.

Figure: Figure 7: Examples of image inpainting, outpainting, and mask-free image editing on internal Image Editing Dataset
Refer to caption: /html/2410.08261/assets/x9.png

## 4 Conclusion and Impact

In this work, we have significantly advanced masked image modeling (MIM) for text-to-image (T2I) synthesis by introducing several key innovations: a transformer architecture blends multi-modal and single-modal layers, advanced positional encoding strategies, and an adaptive masking rate as the sampling condition. These innovations, coupled with high-quality curated training data, progressive and efficient training stage decomposition, micro-conditions, and feature compression layers, have culminated in Meissonic, a 1B parameter model that outperforms larger diffusion models in high-resolution, aesthetically pleasing image generation while remaining accessible on consumer-grade GPUs. Our evaluations demonstrate Meissonic’s superior performance and efficiency, marking a significant step towards accessible and efficient high-resolution non-autoregressive MIM T2I models.

Boarder Impact. Recently, offline text-to-image applications on mobile devices have emerged, such as Pixel Studio from Google Pixel 9 and Image Playground from Apple iPhone 16. These innovations reflect a growing trend toward enhancing user experience and privacy. As a pioneering resource-efficient foundation model, Meissonic represents a significant advancement in this field, delivering state-of-the-art image synthesis capabilities with a strong emphasis on user privacy and offline functionality. This development not only empowers users with creative tools but also ensures the security of sensitive data, marking a notable leap forward in mobile imaging technology.

Acknowledgement. We would like to express our gratitude to all those who
contributed their time, expertise, and insights during the development of Meissonic. Listed in no particular order: Jingjing Ren, Sixiang Chen from HKUST(GZ), Wenhao Chai from University of Washington, Donghao Zhou from CUHK, and other anonymous friends. We are profoundly grateful for their commitment and the unique perspectives they brought to this project.

## Appendix A Model Name Origin

The name “Meissonic” is derived from a combination of the renowned French painter Ernest Meissonier and the term “sonic”. Ernest Meissonier is celebrated for his meticulous attention to detail and his ability to capture dynamic moments in art. The addition of “sonic” evokes a sense of speed and modernity, highlighting the model’s capabilities in efficient image synthesis and transformation.

## Appendix B Related Work

Diffusion-based Image Generation.
Diffusion models have achieved remarkable advances in image generation, with notable contributions like Stable Diffusion (Rombach et al., ), and the more recent SDXL (Podell et al., ), often driven by large-scale datasets. These models move beyond pixel-level operations by working within compressed latent spaces, forming what we now recognize as latent diffusion models (Luo et al., ; Podell et al., ; Wu et al., ; Shi et al., ; Zhou et al., ; Yi et al., ; Wu et al., ). SDXL represents a significant leap in this domain, introducing micro-conditions and multi-aspect training to gain greater control over image generation, which has inspired a wide range of derivative models in the community, such as Deliberate (Desync, ) and RealVisXL (SG_161222, ).

The integration of transformer architectures has also become more prevalent, with models like DiT (Peebles & Xie, ) and U-ViT (Bao et al., ) demonstrating the potential of diffusion transformers in this field. SD3 (Esser et al., ), which combines diffusion transformers with flow matching at an impressive scale of 8B parameters, underscores the scalability and potential of the multimodal transformer-based diffusion backbone. Despite these advances, diffusion models still face challenges, particularly their reliance on acceleration techniques (Sauer et al., ; Luo et al., ; Yin et al., ) to speed up inference, making them cumbersome for real-time applications. Additionally, the quantization of diffusion transformers has proven less straightforward than with large language models (Li et al., ).
The research community continues to explore better paradigms for image generation.
Addressing these limitations, our work aims to contribute an efficient, high-quality alternative in the form of Meissonic.

Token-based Image Generation.
Token-based autoregressive transformers (Lee et al., ; Chen et al., ; Yu et al., ), first validated by VQ-GAN (Esser et al., ), have shown considerable promise for image generation. However, these methods are inherently computationally demanding, requiring the prediction of hundreds to thousands of tokens to form a single image. As a pioneering work, MaskGIT (Chang et al., ) challenged this paradigm by introducing a masked image modeling (MIM) approach, achieving competitive fidelity and diversity in class-conditional image generation. Building on this, MUSE (Chang et al., ) extended MIM to text-to-image synthesis, scaling up to 3B parameters and achieving remarkable performance.

MUSE demonstrates the viability of non-autoregressive token-based models, but it encountered limitations in generating high-resolution images, capping at $512\times 512$, and lagging behind SDXL (Podell et al., ) in terms of fidelity and text-image alignment.
Meissonic advances the performance of token-based models beyond what latent diffusion methods have achieved, effectively pushing the envelope in terms of both quality and resolution in the text-to-image synthesis landscape with the MIM method.

Figure: Figure 8: Zero-shot generation of stylized letters. Meissonic can synthesize individual letters to form the word “MEISSONIC”. Prompt: A post featuring a [COLOR] ‘[LETTER]’ painted on top.
Refer to caption: /html/2410.08261/assets/x10.png

Figure: Figure 9: Memes generated by Meissonic.
Refer to caption: /html/2410.08261/assets/x11.png

Figure: Figure 10: Cartoon Stickers generated by Meissonic.
Refer to caption: /html/2410.08261/assets/x12.png

## Appendix C Applications

We present the letter synthesis capability of Meissonic in Figure [8](#A2.F8).

We present the combination capability of complex concepts of Meissonic in Figure [1](#S0.F1).

We present meme generation in Figure [9](#A2.F9).

We present cartoon sticker generation in Figure [10](#A2.F10).

## Appendix D Ablation Study

Detailed roadmap to build Meissonic. We present ablation studies during training Meissonic-512 in Table. [11](#A4.F11). The HPS v2.1 (Wu et al., ) scores are calculated for verifying the effectiveness of each compoment. Our ablations are based on training stage 2, ensuring consistency with the training dataset scale, model scale, and other training configurations.

Figure: Figure 11: HPS v2.1 Score on internal 1000 prompts
Refer to caption: /html/2410.08261/assets/x13.png

Figure: Figure 12: Multi-modal Transformer For Meissonic.
Refer to caption: /html/2410.08261/assets/x14.png

Figure: Figure 13: Word cloud image of our RealUser-800 prompts benchmark.
Refer to caption: /html/2410.08261/assets/imgs/wordcloud.jpg

Figure: Figure 14: Images generated using the same prompt across Meissonic’s four training stages. The resolutions for stages 1 and 2 are $256^{2}$ and $512^{2}$, respectively, while stages 3 and 4 are $1024^{2}$. For clarity and comparison, all images are displayed in a consistent layout.
Refer to caption: /html/2410.08261/assets/x15.png

## Appendix E Multimodal Transformer Block for Meissonic

We present a detailed structure of our Multi-modal Transformer Block for Meissonic in Figure [12](#A4.F12). Specifically, $x$ denotes image embedding inputs, $c$ denotes text embedding inputs, and $y$ denotes conditions inputs.

## Appendix F Word Cloud of our RealUser800 Benchmark

We present a word cloud image that illustrates the diverse concepts, styles, and themes encompassed within our RealUser-800 prompts benchmark in Figure [13](#A4.F13).

## Appendix G Images generated during different training stages

We present images generated using the same prompt across Meissonic’s four training stages in Figure [14](#A4.F14).

## Appendix H Enlarged Examples from generating diverse styles

We present enlarged samples from Figure [5](#S3.F5) (d) Meissonic in Figure [15](#A8.F15).

Figure: Figure 15: Enlarged Examples from generating diverse styles with Meissonic. Prompt: A garden full of [Y] illustrated in [X] style.
Refer to caption: /html/2410.08261/assets/x16.png

## Appendix I More Examples of Qualitative Comparisons

We present more examples of qualitative comparisons in Figure [16](#A9.F16).

Figure: Figure 16: Qualitative Comparisons with SD 1.5, SD 2.1, DeepFloyd-XL, Deliberate, and SDXL.
Refer to caption: /html/2410.08261/assets/x17.png

## Appendix J More Images Produced by Meissonic

We present additional images generated by Meissonic using CC3M (Sharma et al., ) items, with detailed captions provided by VILA-1.5 (Lin et al., ) and Morph (Pan et al., ).
These images can be found in Figure [17](#A10.F17),[18](#A10.F18),[19](#A10.F19),[20](#A10.F20),[21](#A10.F21),[22](#A10.F22),[23](#A10.F23),[24](#A10.F24),[25](#A10.F25),[26](#A10.F26),[27](#A10.F27),[28](#A10.F28),[29](#A10.F29),[30](#A10.F30),[31](#A10.F31),[32](#A10.F32),[33](#A10.F33),[34](#A10.F34),[35](#A10.F35).

We present additional images generated by Meissonic using HPS (Wu et al., ) benchmark prompts.
These images can be found in Figure [36](#A10.F36),[37](#A10.F37),[38](#A10.F38),[39](#A10.F39),[40](#A10.F40),[41](#A10.F41).

Figure: Figure 17: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x18.png

Figure: Figure 18: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x19.png

Figure: Figure 19: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x20.png

Figure: Figure 20: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x21.png

Figure: Figure 21: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x22.png

Figure: Figure 22: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x23.png

Figure: Figure 23: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x24.png

Figure: Figure 24: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x25.png

Figure: Figure 25: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x26.png

Figure: Figure 26: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x27.png

Figure: Figure 27: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x28.png

Figure: Figure 28: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x29.png

Figure: Figure 29: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x30.png

Figure: Figure 30: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x31.png

Figure: Figure 31: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x32.png

Figure: Figure 32: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x33.png

Figure: Figure 33: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x34.png

Figure: Figure 34: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x35.png

Figure: Figure 35: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x36.png

Figure: Figure 36: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x37.png

Figure: Figure 37: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x38.png

Figure: Figure 38: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x39.png

Figure: Figure 39: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x40.png

Figure: Figure 40: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x41.png

Figure: Figure 41: High Quality Samples Produced by Meissonic.
Refer to caption: /html/2410.08261/assets/x42.png

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 4
questions: Since Meissonic uses a CLIP text encoder, which might limit its effectiveness in scenarios demanding high text comprehension (e.g. generating images with visual texts), could the authors provide examples comparing Meissonic's performance on complex prompts versus simpler prompts? Also, would there be noticeable performance degradation if the model were applied to open-domain data?
soundness: 3
strengths: - The integration of multi-modal transformers with single-modal refinements optimizes MIM for high-resolution synthesis and addresses the historical gap in resolution and quality compared to diffusion models.
- The model's efficiency is notable as it requires only 48 H100 GPU days which is far less than other high-resolution models like SDXL while achieving comparable performance.
summary: - The paper presents Meissonic, a high-resolution, non-autoregressive masked image modeling (MIM) approach for text to image generation. 
- It introduces architectural innovations to elevate MIM’s performance to rival diffusion models like SDXL. 
- Its key advancements include a multi-modal transformer architecture, Rotary Position Encoding (RoPE), feature compression layers, and micro-conditions based on human preference scores. 
- It enables high-fidelity image generation at 1024×1024 resolution on 8GB VRAM. 
- The model is trained on high-quality datasets and achieves state-of-the-art results on various benchmarks.
weaknesses: - Using CLIP text encoder makes Meissonic potentially less adept at handling complex text prompts compared to models using larger language models like T5 which could impact its effectiveness in scenarios demanding high text comprehension. e.g. generating images with visual texts.
- The reliance on high-quality datasets may limit the model's generalization in open-domain applications.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: These are questions that are minor enough that do not count in the final scores.

- Does this model architecture supports DreamBooth-type fine-tuning for practical application?
- The success of U-Net-based architecture (e.g., SD, SDXL) partially relies on the strong controllabiliy gained from ControlNet. However, such I do not want to require the authors to provide the experimental results or detailed idea, but I wonder if they are planning to develop in this direction.
- I believe it would be great to have Figure 11 in the main manuscript, since this shows justification of changes made during the development of the proposed model.
soundness: 3
strengths: - The presented method is lightweight, in terms of the number of parameters and the training time, lowering production cost of large scale image generative models.
- The model also have reduced inference time with lower GPU memory cost compared to SDXL.
- The training procedure is opened with transparency as shown in Figure 15.
- The model supports masked image editing and inpainting.

Overall, the proposed method in this manuscript can be regarded as a good industrial image generative model with production-ready quality.
summary: The paper scales up a masked image modeling (MIM) framework for professional-quality image generative model comparable to the diffusion-based SDXL. The key problems the authors have find in MIM framework is its low resolution constraint and persisting performance gap. The newly proposed model, Meissonic model incorporates efficient text encoder, mixed single-modal and multi-modal transformer backbone using rotary positional encoding, and curated training datasets. The final model performs comparable to SDXL in quality and exceeds efficiency in inference time and memory requirement.
weaknesses: The main concerns are in the presentation of the material. Although the model at this reported performance can contribute to the generative AI community, the current form of manuscript seems more like an industrial technical report than an academic article.

- Since the authors have claimed for supporting inpainting, outpainting, mask-free editing, and masked editing, there should be a comparison in both quantitative and qualitative way for each of the task in the manuscript. For now, the article only contains EMU-Edit scores with reduced number of columns--the original EMU-Edit contains five algorithmic ratings (CLIP_dir, CLIP_im, CLIP_out, L1, DINO) and two human ratings (Text Alignment, Image Faithfulness), measured in two different test sets (Emu Edit / MagicBrush). Also, the qualitative comparison is not shown in the paper, showing only the proposed method in Figure 6 and 7. Overall, this writing style makes the article more of an well-written industrial technical report than an academic paper. As a recommendation, there are many well-established detailed quantitative metrics such as masked image preservation and text alignment (e.g., PSNR/LPIPS/MSE as in BrushNet, Table 2), and overall generation quality (e.g., FID as in Table 2 of PQDiff), which can be applied to compare with, for example, SDXL.
- Maybe it is due to the intrinsic characteristics of MIM framework, but the inpainting in Figure 7 seems to change the non-masked regions of the image. This deformation should be measured in quantitative metrics such as in BrushNet.
- It would be better to have plots depicting how the generated quality changes by changing the number of iterations in the inference, or at least a recommendation of the number of inference steps of using the pre-trained model.
- I believe image demonstration from page 20 to the end is unnecessary and carry little information about the model. It is not bad to have a lot of demonstration images in a technical report, but there should be at least qualitative comparison between other models in order to have persuasive narrative. For example, generating random 5-16 images from the same text prompt without curation and show the diversity and the point of breaking (e.g., broken hands or limbs) can be visually compared with other large scale generative models such as SDXL-base, which is quoted multiple time throughout the manuscript.
- I would like to recommend adding details about training and inference algorithms. For example, the explanation of “micro controls” and architectural detail of “feature compression layers” is not described in appropriate level of details (what is chosen, how's the size of this controls, and how each component benefits the training, etc.). Moreover, even if the original idea is borrowed from previous papers, such information should be contained in the paper for completeness, e.g., how is 1D RoPE encoding used in this specific 2D MIM Transformer block?

Regarding the mentioned issues, I believe that the current version of the manuscript is not ready for publication.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. It is interesting that the diverse micro conditions (Line 229 - Line 232) are helpful for the training stability (Line 134). From a high level perspective, why would these conditions help with the stability?
2. It is interesting to see that you choose to use a "lightweight" text encoder, CLIP (Line 210), instead of using LLM encoders or a mixed way like in SD3. Is there a reason or a hypothesis behind this choice, not just a "trial-and-error" experience?
soundness: 3
strengths: ### Motivation
- The paper shows a good motivation - MIM shows the potential capability to unify the language and vision modalities, and the paper attempts to bridge the gap between MIM and other popular image generation fashions. 
- The paper shares several interesting observations on the current MIMs, including architecture design, positional encoding, and masking rate as a condition. It then starts from those insights and implement effective components to gain improvement. 

### Method
- The proposed pipeline is straight-forward and seems effective. 
- It is an interesting finding that embedding original image resolution, crop coordinates, etc would be helpful for the stability of the training, as mentioned in Line 229 - Line 232. 
- The decomposition of training stages (Section 2.5) makes sense to me. It could work as a new procedure for other or future model training. 

### Experiments/Results
- The authors conduct extensive experiments and evaluations to demonstrate the performance of Messonic. 
- Messonic shows superiority on HPS benchmark, GenEval benchmark, and MPS scores. It shows competitive results with SDXL, while has a 2x smaller model size and transparent training costs (Table 1).
- Messonic also shows an economic expense of VRAM (i.e., 8GB). This demonstrates a promising value for customer level usage. 
- Messonic shows a good performance in Zero-shot I2I task, as shown in Table 6 and Figure 5.

### Writing/Delivery
The paper is well-written. The technical parts are clear and easy to understand.
summary: The paper explores Masked Image Modeling (MIM), an non-autoregressive technique, and propose a new model named Messonic. Messonic poses a promising enhancement on MIM compared to its auto-regressive and diffusion-based counterparts. The keys to the improvement of Messonic are enhanced transformer architecture, advanced positional encoding and adding masking rate as a condition. The authors conduct extensive evaluations on Messonic to demonstrate its superiority.
weaknesses: I do not see a critical weakness from this paper. Some minor points:
- Messonic seems to be designed to match the auto-regressive manner in language models. The authors indicates it aims to unify language-vision models (Line 014). However, I do not see a strong evidence to show the current Messonic achieves this goal in the paper.
- The proposed approach is more driven from an "add-on" manner, where most of the components in Messonic are "borrowed" from other works. For example, multi-modal transformer backbone from the ADD paper, RoPE from Roformer, tokenization from VQVAE. The core contribution of the paper, therefore, seems to be a practice of choosing these components and empirical trials.

Meta-review: The work introduced Messonic that improved MIM with enhanced transformer architecture, advanced positional encoding and adding masking rate as a condition. The effectiveness of proposed approach is demonstrated well and efficiency brought by the improvement is obvious. It consistently received positive feedback from reviewers with three borderline accept. No critical concerns from reviewers but there are some concerns about presentation and the choice of text encoders. Authors made significant effort for the rebuttal which addressed most of concerns after discussions. AC appreciate challenging the traditional backbone and finding sort of breakthrough to improve the foundation models, especially for high resolution generation which is known as non-trivial. After checking all the comments and discussions, a decision of accept is made and authors are suggested to incorporating the useful comments and newly added experiments in the revised version.

Decision: Accept (Poster)