Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis
Authors: Jinbin Bai
Tian Ye
Wei Chow
Enxin Song
Qing-Guo Chen
Xiangtai Li
Zhen Dong
Lei Zhu
Shuicheng YAN
Venue: ICLR 2025 Poster (2025)
Keywords: Text-to-Image Synthesis
Masked Generative Transformers
Primary area: generative models
TL;DR: We present Meissonic, a next generation text-to-image foundation model with non-autogressive masked image modeling method.

Abstract: We present Meissonic, which elevates non-autoregressive text-to-image Masked Image Modeling (MIM) to a level comparable with state-of-the-art diffusion models like SDXL. By incorporating a comprehensive suite of architectural innovations, advanced positional encoding strategies, and optimized sampling conditions, Meissonic substantially improves MIM's performance and efficiency. Additionally, we leverage high-quality training data, integrate micro-conditions informed by human preference scores, and employ feature compression layers to further enhance image fidelity and resolution. Our model not only matches but often exceeds the performance of existing methods in generating high-quality, high-resolution images. Extensive experiments validate Meissonic’s capabilities, demonstrating its potential as a new standard in text-to-image synthesis.

Published as a conference paper at ICLR 2025
MEISSONIC: REVITALIZING MASKED GENERATIVE TRANSFORMERS FOR EFFICIENT HIGH-RESOLUTION
TEXT-TO-IMAGE SYNTHESIS
Jinbin Bai Tian Yea Wei Chow Enxin Song", Qing-Guo Chen,
Xiangtai Li?, Zhen Dong", Lei Zhu Shuicheng Yan
National University of Singapore 2kkworrk Al HHKUST(ZZ) "HKUST SUC Berkeley 6ZJU
Model:
cta.sfacccccooeeee https//ccee.o/eessssosllee. https: /tn/viiika/Meiisonio
Code:
ABSTRACT
We present Meissonic, which advances non-autoregreesive text-to-image Masked Image Modeling (MIM) to level comparable with state-of-the-art diffusion mod- els like SDXL. By incorporating a series of architectural innovations, advanced positional encoding strategies, and optimized sampling conditions, Meissonic sig- nificantly improves MIM's performance and efficiency. Additionally, we leverage high-quality training data, integrate micro-conditions informed by human pref- erence scores, and employ feature compression layers to further enhance image fidelity and resolution. Our model not only matches but often exceeds the per- formance of existing methods in generating high-quality, high-resolution images. Extensive experiments validate Meissonic's capabilities, demonstrating its poten- tial as a new standard in text-to-image synthesis.
INTRODUCTION
Diffusion models, such Stable Diffusson (Rombach al., 2022: Podell al., 2024; per, 2024; Art, 2023), have rapidly advanced to become the dominant paradigm in visual generation by replac- ing Generative Adversarial Network (GAN). Recent developments like LlamaGen (Sun al., 2024) have ventured into autoregressive image generation using discrete image tokens derived from VQ- VAE (Yu et al., 2022a). Despite progress, the substantial number of image tokens compared text tokens makes autoregressive generation inefficient, For example, tokenizing one 1024 1024 im- age using 16x downsampled VQVAE yields 4096 tokens, where sequential generation process prohibitively slow.
Masked generative transformers, a class of generative models, have achieved significant results in the fields of image generation, Specifically, MaskGIT (Chang et al., 2022) introduced more ef- ficient, non-autoregressive alternative, where all image tokens are predicted simultaneously in a parallel, iterative refinement process. Then, MUSE (Chang et al., 2023) extended this technique to higher resolutions, achieving 512 x 512 resolution T2I generation. These non- aon-autoreeressvve meth- ods offer around 99% reduction in decoding steps compared autoregressive methods. However, despite their efficiency, non-autoregressive transformers remain limited performance compared to advancing diffusion autoregressive models, particularly high-quality, high-resolution text-to- image synthesis.
In this work, address these challenges and introduce two key innovations make masked image modeling (MIM) competitive with advanced diffusion models:
Enhanced Transformer Architecture: Previous MIM methods (Chang al., 2023: 2022) pre- dominantly utilized naive transformer architectures, potentially limiting their capabilities. We dis-
"qqual contribution. ø: jinbin. bai bhaiuu @u. nus-edi 'Corresponding authors.

---

Published as a conference paper at ICLR 2025
Images produced by Meissonic exhibit exceptional image quality. More samples can be found in Appendix M. Notably, Meissonic can effortlessly produce images with solid-color back- grounds without requiring any additional modifications.
Figure 1:
covered that combination of multi-modal and single-modal transformer layers can significantly boost MIM training efficiency and performance. Language and vision representations are inherently different. The multi-modal transformer effectively capture cross-modal interactions, extracting information from unpooled text representations and effectively bridging the gap between these dis- tinct modalities. This allows the model harness useful signals from noisy data. Additionally, subsequent single-modal transformer layers refine the visual representation, improving performance and training stability.
Advanced Positional Encoding & Masking Rate as Sampling Condition: We incorporate Rotary Position Embedding (RoPE) (Su et al., 2024) for encoding positional information in queries and
2

---

Published as a conference paper at ICLR 2025
keys, which helps maintain detail in high- high-resolution images. RoPE effectively addresses the issue of context disassociation in transformers as the number of tokens increases. Additionally, we intro- duce the masking rate dynamic sampling condition throughout the generation process. Previous MIM methods Chang et al. (2023; 2022) have overlooked this aspect, resulting in suboptimal image details. This issue arises because the number of tokens predicted by the MIM model changes dra- matically throughout the sampling loop. With the masking rate condition, the model can ascertain the current stage of the sampling period by leveraging conditional information from the masking rate.
Beyond these architectural improvements, achieve comparable performance with SDXL for high- resolution generation, we adopt effects in three additional aspects:
The quality of training data is crucial. While LAION (Schuhmann et al., 2022) offers a diverse visual dataset, its captions can be subpar (Chen et al., 2024), We curated a high-quality internal dataset with accurate captions, which, combined with our training strategy, significantly improved the generative capabilities of the base model.
High-Quality Training Data:
We identified that incorporating original image resolution, crop coordinates, and human preference scores (Wu et al., 2023) as micro-conditions greatly enhances model stability during high-resolution aesthetic training.
Feature Compression Layers: To efficiently generate high-resolution images, we integrated feature compression layers, maintaining computational efficiency even 1024 x 1024 resolution.
Our contributions culminate in Meissonic, next-generation T2I model based on masked discrete image token modeling. Unlike larger diffusion models such as SDXL (Podell et al., 2024) and DeepFloyd- --- (Liu al., 2024a), Meissonic, with just IB parameters, offers comparable supe- rior 1024 x 1024 riihutessouttin, aesthetically pleasing images while being able to run on consumer- grade GPUs with only 8GB VRAM without the need for any additional model optimizations. More- over, Meissonic effortlessly generates images with solid-color backgrounds, a feature that usually demands model fine-tuning or noise offset adjustments in diffusion models.
Advancement of Meissonic represents a significant stride towards high-resolution, efficient, and ac- cessible T2I MIM models. We evaluate Meissonic using various qualitative and quantitative metries, including HPS, MPS, GenEval benchmarks, and GPT4o assessments, demonstrating its superior performance and efficiency.
METHOD
2.1
MOTIVATION
Recent breakthroughs in text-to-image synthesis have been largely propelled by diffusion models, such as Stable Diftusion XL, which have set de facto standards for image quality, detail, and con- ceptual fidelity.
Another approach, non-autoregressive Masked Image Modeling (MIM) techniques. exemplified by MaskGIT and MUSE, has shown potential for efficient image generation replace slow autore- gressive techniques like Llamagen. Yet, despite their promise, MIM approaches face two critical limitations:
(a) Resolution Constraint. Current MIM methods are limited to generating images at maximum resolution of 512 x 512 pixels. This limitation hinders their broader adoption and advancement, particularly as the text-to-image synthesis community increasingly adopts 1024 x 1024 resolution as the standard.
(b) Performance Gap. Existing MIM techniques have not yet achieved the level of performance exhibited by leading diffusion models like SDXL. They notably underperform in key areas such as image quality, intricate detailing, and conceptual representation, which critical for practical applications.
These challenges necessitate the exploration of new approaches. Our objective is to empower MIM to efficiently generate high-resolution images (e.g., 1024 1024), while narrowing the gap with top- tier diffusion models, and ensuring computational efficiency suitable for consumer-grade hardware.
3

---

Published as a conference paper at ICLR 2025
The Canadian astronaut
TartEn
Gross Entraey Loss
Ene
VQ Dea
Mutiimodal Fransformed
x T' steps
The architecture of Meissonic, During the image generation process, discrete tokens are created randomly according to a predefined schedule. Meissonic then applies masking and performs predictions over several steps to reconstruct all tokens and decode the resulting image. In the case of image editing, the original image is converted into discrete tokens, which are masked according to specified masking strategy. After series of processing steps, the masked tokens are reconstructed and utilized to decode the target image. Text prompts and other conditions are incorporated to control the synthesis process. Rt represents the masking rate condition, and C' indicates the micro conditions. More details about Multi-modas Transformer Block can be found in Appendix I.
Figure 2:
Through our work, Meissonic, we aim to push the boundaries of MIM methods and bring them to the forefront of text-to-image synthesis.
2.2 MODEL ARCHITECTURE
The Meissonic model is architected to facilitate efficient high-performance text-to-image synthesis through an integrated framework comprising CLIP text encoder (Radford et al., 2021), ), a vector- quantized (VQ) image encoder and decoder (Esser et al., 2021), and Multi-modal Transformer backbone. Figure illustrates the overall structure of the model.
Vector--uantizer Image Encoder and Decoder. We employ VQ-VAE model (Esser et al., 2021) to convert raw image pixels into discrete semantic tokens. This model comprises an encoder, a decoder, and quantization layer that maps input images into sequences of discrete tokens using learned codebook. For an image of size W, the encoded token size is H W. where f represents + x T.
the downsampling ratio. In our implementation, we utilize downsampling ratio of = 16 and codebook size of 8192, allowing a 1024 x 1024 image to be encoded into sequence 64 64 discrete tokens.
Flexible and Efficient Text Encoder. Instead of using large language model encoders, such T5- XXL (Raffel et al., 2020) or LLaMa (Touvron et al., 2023), which are prevalent in previous works (Chen et al., 2024; Esser et al., 2024), we utilize a single text encoder from the state-of-the-ar CLIP model with a latent dimension of 1024, and fine-tune for optimal T21 performance. While this decision may limit the model's capacity fully comprehend lengthy text prompts, our observations indicate that excluding large-scale text encoders like T5 does not diminish visual quality. Moreover, this approach significantly reduces GPU memory requirements and computational cost. Notably, offline extraction of T5 features would entail approximately 11 times more processing time and 6 times more storage than employing the CLIP text encoder, underscoring the efficiency of our design.
Multi-modal Transformer Backbone for Masked Image Modeling. Our transformer architecture builds upon the Multi-modal Transformer framework (Sauer et al., 2024), incorporating sampling parameters encode sampling parameters and Rotary Position Embeddings (RoPE) (Su et al., 2024) for spatial information encoding. We introduce feature compression layers to efficiently han- dle high-resolution generation with numerous discrete tokens. These layers compress embedding features from 64 x 64 to 32 x 32 before processing through the transformer, and followed by feature decompression layers
64, thereby alleviating computational burdens. To enhance training stability and mitigate the NaN Loss issue, we follow the training strategy from LLaMa Touvron et al.
Many works indicate that the T5 text encoder is the key factor in obtaining the ability to synthesize words, still show the ability to synthesize letters Figure 10. We leave this future improvement.
4

---

Published as a conference paper at ICLR 2025
(2023), implementing gradient clipping and checkpoint reloading during distributed training and in- tegrating QK-Norm layers into the architecture. We elaborate on the designs of our transformer in the subsequent section.
Diverse Micro Conditions. To augment generation performance, we incorporate additional con- ditions such original image resolution, crop coordinates, aesthetic score, and human preference score (Wu et al., 2023). These conditions are transformed into sinusoidal embeddings and concate- nated additional channels to the final pooled hidden states of the text encoder.
Masking Strategy. Following the approach established in Chang et al. (2023), we employ a variable masking ratio with cosine scheduling. Specifically, randomly sample a masking ratio r E [0, 1] from a truncated arccos distribution characterized by the following density function:
p(r) = (1-
In contrast to autoregressive models that learn conditional distributions P(x; xzee) for fixed token orders, our approach utilizes random masking with variable ratios to enable the model to learn P(r;; | TA) ) for arbitrary subsets of tokens A. This flexibility is pivotal for our parallel sampling strategy and facilitates various zero-shot image editing capabilities, which will be demonstrated in Section 3.
2.3 MULTI-MODAL TRANSFORMER FOR MASKED IMAGE MODELING
Meissonic employs the Multi-modal Transformer as its foundational architecture and innovatively customizes the modules address the distinctive challenges inherent in high-resolution masked image modeling. We introduce several specialized designs for MIM as follows:
Rotary Position Embeddings. RoPE (Su et al., 2024) has demonstrated exceptional perfor- mance within LLMs (Su al., 2024; Touvron et al., 2023; Ding et al., 2024; Bai al., 2023), Some studies (Lu et al., 2024; Lin et al., 2023; Zhuo et al., 2024) have attempted to extend ID RoPE (Su al., 2024) to 2D or 3D for image diffusion models. We apply rotary position embeddings to each query and key features in attention layers.
Deeper Model with Single-modal Transformer. Although the Multi-modal Transformer block demonstrated commendable performance, reducing the number of multi-modal blocks to single-modal block configuration offers a more stable and computationally ef- ficient approach for training T2I models. Therefore, we opt to employ Multi-modal Trans- former blocks in the initial stages of the network, transitioning to exclusively Single-modal Transformer blocks in the latter half, Our findings suggest an optimal block ratio of about 1:2.
Micro Conditions with Human Preference Score. Incorporating three micro-conditions is pivotal for achieving stable and reliable High-resolution MIM Model: original image resolution, crop coordinates, and human preference score. The original image resolution effectively aids the model in implicitly filtering out low-quality data and learning the prop- erties of high-quality, high-resolution data, while crop coordinates enhance training stabil- ity, likely due improved consistency between image conditions and semantic conditions during cropped patch coordination. In the final stage, we leverage the Human Preference Score (Wu et al., 2023) to effectively enhance image quality, using signals provided by the Human Preference Model to guide the model's outputs in mimicking and approximating human preferences.
Feature Compression Layers. Unlike multi-stage approaches like MUSE Chang et al. (2023) and DeepFloyd-XL DeepFloyd (2023), which employ cascading multiple subnet- works for higher-resolution image generation. We simplify the process by integrating streamlined feature compression layers during the fine-tuning stage to facilitate efficient high-resolution generation process learning. This approach functions akin to a lightweight high-resolution adapter Guo et al. (2024), a module extensively explored and integrated within Stable Diffusion. By incorporating 2D convolution- based feature compression lay- ers into the transformer backbone, layers and subsequently decompress them after the transformer layers, effectively address- ing the challenges of efficiency and resolution transition.
compress the feature maps prior to the transformer
5

---

Published as a conference paper at ICLR 2025
Table 2: HPS v2.0 benchmark. Scores collected from httpt: httmmtttt////ttthb..c/m////////
HP Hv2. We highlight the best.
HPS v2.0
Model
Animation
Concept-art
Painting
Photo
Averaged
DALL-E 2 (Ramesh et al., 2022)
27.34
26.54
08
27.24
Stable Diffusion (Rombach
2022)
27.26
26.61
26.
27.27
26.95
Stable Diffusion v2.0 (Rombach al., 2022)
27.48
26.89
26
27.46
27.17
SDXL Base 0.9 (Podell et al,, 2024)
28.42
27.63
27.60 60
27.29
27.73
Realistic Vision (rea, 2024)
28.22
27.53
27.56
27.75
27.77
SDXL Refiner 0.9 (Podell al., 2024)
28.45
27.66
27.80
SDXL Base 1.0 (Podell al,, 2024)
28.88
27.88
27.92
28.31
28.25 28
SDXL Refiner 1.0 (Podell et al. 2024)
28.93
27.89
27.90
28.38
28.27
Mcissonie-513
28.90
28.15
28.22
28.04
28.33
Mcissonic
29.57
28.58
28.72
28.45
28.83
2.4 TRAINING DETAILS
Meissonic is constructed using CLIP-ViT-H-14 text encoder (II- harco et al., 2021), a pre-trained VQ image encoder and decoder (Patil al., 2024), and customized Transformer-based (Esser et al., 2024) backbone. We employ classifier-free guidance (CFG) (Ho & Salimans, 2022) and cross-entropy loss to train Meissonic. Training occurs across three resolution stages, leveraging both public datasets and our curated data. First, we train Meissonic-256 with batch size of 2,048 for 100,000 steps. Second, we continue training Meissonic-512 with batch size of 512 for additional 100,000 steps. Third, we continue training Meissonic with a batch size of 256 for 42,000 steps with a resolution of 1024 1024. The performance results of Meissonic-512 and Meissonic experiments are carried out with a fixed learning rate of 1 x 10-4, Further details are elaborated in Sec. 2.5, All inferences in this paper are performed with CFG 9 and 48 steps.
Table I: Comparison of training data and time for various models.
Params (B)
Training Images (M)
8xA100 GPU
Model
Days"
Würstchen (Pernaa et 2024)
1.0
1420
128.1
SD-1.5 (Rombach al., 2022)
0.9
4800
781.2
SD-2.1 (Rombach et al., 2022)
0.9
3900
1011.6
Imagen (Saharia et al., 2022)
3.0
891.5
Dall-E (Ramesh et al., 2022)
6.5
650
5208.3
GigaGAN (Kang et al., 2023)
0.9
980
597.8
SDXL (Podell al., 2024)
2.6
unknown
unknown
Meissonic
1.0
210
19
Data collected from Sebwag (2024).
FP16 Tensor Core of A100 is 312 TFLOPS and H100 is 756.5 TELOPS.
GPU hours are adjusted from H100 days based on this rate.
reported in Table 2. All
It's crucial to highlight the resource efficiency of our training process. Our training is considerably more resource-efficien compared Stable Diffusion (Podell et al., 2024). Meissonic is trained in approximately 48 H100 GPU days, demonstrating that production-ready image synthesis founda- tion model can be developed with considerably reduced computational costs. Additional details on this comparison can be found in Table 1.
2.5 PROGRESSIVE AND EFFICIENT TRAINING STAGE DECOMPOSITION
Our approach systematically decomposes the training process into four carefully designed stages, allowing us to progressively build and refine the model's generative capabilities. These stages, combined with precise enhancements to specific components, contribute to continual improvements in synthesis quality. Given that SDXL. has not disclosed details regarding its training data, our experience is particularly valuable for guiding the community in constructing SDXL-level text-to- image models. We present images generated by Meissonic at each of the four training stages in Figure to support our claims. More examples can be found Appendix K.
Stage 1: Understanding Fundamental Concepts from Extensive Data. Previous studies (Chen et al., 2024; Yu et al., 2024) indicate that raw captions from LAION are insufficient for training text- to-image models, often requiring the caption refinement provided by MLLMs such as LLaVA (Liu
We utilize
from OpenCLIP initial weights.

---

Published as a conference paper at ICLR 2025
Stage
Stage
Stage
Stage
image of Pikachu wearing a birthday playing guitar
A black and white line drawing of rhinoceros head.
Figure 3: Images generated using the same prompt across Meissonic's four training stages. The resolutions for stages and 2 are 256² and 512?, respectively, while stages and are 1024?. For clarity and comparison, all images are displayed consistent layout.
EFLIEL
A graphic poster depicting the fiery end of the world with detailed botanical illustrations and artistic influences.
Pakemon that resembles phone booth gaining popularity Artstation and Unreal Engine.
Low poly John Travolta in Golden Eye
1.5
2.1
DeepFloyd-XL
Deliberate
SDXL
Meissonic
Figure 4: Qualitative Comparisons with SD 1.5, SD 2.1, DeepFloyd-XX XL, Deliberate, and SDXL.
al., 2024b). However, this solution is computationally demanding and time-intensive. While some studies (Chen et al., 2024; Sehwag et al., 2024) utilize the extensively annotated SA-10M (Kirillov et al., 2023) dataset, our findings reveal that SA-1OM does not comprehensively cover fundamental concepts, particularly regarding human faces. Thus, we carefully curated the deduplicated LAION- 2B dataset by filtering out images with aesthetic scores below 4.5, watermark probabilities exceeding 50%, and other criteria outlined in Kolors (2024). This meticulous selection resulted in approxi- mately 200 million images, which were employed for training at a resolution of 256 256 this initial stage.

---

Published as a conference paper at ICLR 2025
Table 3: GenEval benchmark, We highlight the best result.
Table 4: MPS scores on RealUser-800
Prompts.
We highlight the best result.
Modid
Overall
Oljjects
Counting
Positinn
Attribation
Model
MPS
Single
Two
Stable Diffusion v1.4 [Rombach et (2022)
13.89
DALL-R mini
0.23
0.71
0.11
0.12
0:37
0.00
B:1
Stable Diffusion v2.0 [Rombach et al. (2022)
14.39
SD
0.43
0.97
0.34
0.35
0.76
0.04
B106
SDXI, Base 1.0 PPdell et al. (2024)
16.46
SD
0.50
0.98
0.51
0.44
8.85
0.07
0.17
DALL-62
052
0.94
0.00
0.89
8.77
0.10
8.19
SDXL Refiner 1.0 [Podell (2024))
16.56
SO
0.55
0.98
6.74
0.99
0.85
0.15
Meissonic
17.34
0.00
0.10
Stage 2: Aligning Text and Images with Long Prompts. the second stage, we focus on im- proving the model's capability to interpret long, descriptive prompts. We filtered the initial LAION set more rigorously, retaining only images with aesthetic scores above 8, and other criteria out- lined in Kolors (2024). Additionally, we incorporate 1.2 million synthetic image-text pairs with refined captions exceeding 50 words, primarily derived from publicly available high-quality syn- thetic datasets, complemented by additional high-quality images from our internal 6 million dataset. This aggregation results in around 10 million image-text pairs. Notably, we maintain the model architecture while increasing the training resolution 512 x 512, enabling the model capture more intricate image details. We observed significant boost in the model's ability to capture ab- stract concepts and respond accurately to complex prompts, including diverse styles and fantasy characters.
Stage 3: Mastering Feature Compression for Higher-resolution Generation. High-resolution generation remains an unexplored area within MIM (Chang et al., 2023; 2022; Patil et al., 2024). Unlike methods such MUSE(Chang et al., 2023) or DeepFloyd-XL. (DeepFloyd, 2023), which rely external super-resolution (SR) modules, we demonstrate that efficient 1024 x 1024 generation feasible through feature compression for MIM. By introducing feature compression layers, we achieve scamless transition from 512 x 512 1024 x 1024 generation with minimal computational cost. In this stage, we further refine the dataset by filtering based on resolution and aesthetic score, selecting approximately 100K high-quality, high-resolution image-text pairs from the LAION subset utilized in Stage 2. This, combined with the remaining high-quality data, results in approximately 6 million samples for training 1024 resolution.
Stage 4: Refining High-Resolution Aesthetic Image Generation. the final stage, we fine- tune the model using small learning rate, without freezing the text encoder, and incorporate aesthetic score as a micro condition. This can significantly enhance the model's performance in high-resolution image generation. This targeted adjustment significantly enhances the model's per- formance in generating high-resolution images, while also improving diversity. The training data remains the same as in Stage
RESULTS
3.1 QUANTATIVE COMPARISON
Classic evaluation metrics for image generation models, such as FID and CLIP Score, have lim- ited relevance to visual aestheties, as highlighted by Podell al. (2024); Chen al. (2024); Kolors (2024); Sehwag et al. (2024). Therefore, we report our model's performances using Human Pref- erence Score v2 (HPSv2) (Wu et al., 2023), GenEval (Ghosh et al., 2024), and Human Preference Score (MPS)² (Zhang al., 2024b), as illustrated in Table 2,3,4.
In our pursuit of making Meissonic accessible to the broader community, we optimized our model to 1 billion parameters, ensuring that it runs effi- ciently on 8GB VRAM, making in- ference and fine-tuning both conve- nient. Figure 5 provides a compar- ative analysis of GPU memory con-
Table 5: Comparison of step (50 steps) inference time (s) for Different Models and Batch Sizes.
Batch Size
Model
2
4
SDXL Base
0.36 (5.388
0.75 (10.06)
1.41 (19.69)
2.79 (38.58)
Meissonie-256
0.09 (3.11)
(3.14)
0.11 (3.22)
0.16 (4.70)
Meissonic-512
0.13 (3.24)
0.17
0.28 (7.74)
0.51 (14,51)
Mcissonic-1024
0.24 (3.48)
0.35 (4.62)
0.62 (8.52)
1.17( 16.46)
GGiven that the KolorsPrompts benchmark was unavailable, we curated diverse prompt dataset consisting of 800 real user-generated prompts spanning various concepts and themes for the MPS evaluation.
8

---

Published as a conference paper at ICLR 2025
(0) SD
1D/90
Origana
Anaiaatid
Red
Apprcial
Sketch
Fost
Octerpue
[c)
Messionic
Figure 6: Evaluating the ability to generate diverse styles. Prompt: garden full of [Y] illustrated in [X] style.
sumption' 4 across different inference hatch sizes against SDXL. Additionally, Table 5 details the inference time per step'.
3.2 QUALITATIVE COMPARISON
Hilisanc (8-40)
We also present qualitative comparisons of im- age quality and text-image alignment in Figure 4, with additional comparisons provided in the Ap- pendix. Furthermore, Figure illustrates Meis- sonic's 's proficiency in generating text-driven style art image. To complement these analyses, we conduct human evaluation by K-Sort Arena (Li et al., 2024), and we conduct GPT-4o to evalu- the performance between Meissonic and other models in Figure 9.
memory cost (GB)
40
Helisonc 512 (946)
Palasanc-102 1024 7s1.
buas-rellie
30
24.0 GB
10
8.0 GB
4.0 GB
Inference Batch Size
Figure 5: GPU Memory Cost vs Inference Batch Size Different Models.
All Figures and Tables demonstrate that Meissonic achieves competitive performance in human performance and text alignment compared to DALL-E and SDXL, as well as showcasing its effi-
ciency.
3.3 ZERO-SHOP IMAGE-TO-IMAGE EDITING
For image editing tasks, we benchmarked Meis- sonic against state-of-the -frth---rt-- models using the EMU-Edit dataset (Sheynin et al., 2024), with results presented Table 6. Additionally, ex- amples from HumanEdit Bai al. (2024), in- cluding guided editing in Figure 7 and mask-free editing in Figure further showcase Meissonic's s versatility. Remarkably, Meissonic achieved this performance without any training or fine-tuning on image struction dataset. More comparisons for pero-shot image editing ability can be found in Appendix F.
CLIP-I)
IestrucPPidPPi
MaricBrush (Zhang 20244)
(Tummnynn et 2023)
0.521
(Molady 2023)
0.761
EMU-Edit (hhenii et 2024)
0,850
0.819
Meissonic
0.871
0.266
0.700
Table 6: Results the EMU-Fdit test
data in-
'GPU memory usage was gauged using torch. cuda. www.mmmmr......m...mm wemoryyrreeeeeee rew.rvedeeeeeeee (). While this method might
yield higher values, all models are measured under identical settings maintain fairness. "Ineerence time is assessed using A100 GPU with fp16 models. Notably, the reported times contributions from the VAE and encoder, meaning that multi-step inferences do scale linearly.

---

Published as a conference paper at ICLR 2025
styiihh dog maaring
watton prapp
Thene edtiiinable
wamon wear in on farts sort to
with stort
greae grost
strring
made:
Figure 7: Examples of image editing with mask on internal Image Editing Dataset
lepax
Uuipos
Aguur ruceo
Skt Meart
digllng
Smcll
Maabrtte
Small Tger
Figure Examples of image inpainting, out- painting, and mask-free image editing on our in- ternal Image Editing Dataset
Figure 9: GPT4o Preference Evaluation of Meis- sonic against current open Text-to-image Mod- els.
CONCLUSION AND IMPACT
In this work, we have significantly advanced masked image modeling (MIM) for text-to-imane (T2I) synthesis by introducing several key innovations: a transformer architecture blends multi-modal and single-modal layers, advanced positional encoding strategies, and adaptive masking rate as the sampling condition. These innovations, coupled with high-quality curated training data, progressive and efficient training stage decomposition, micro-conditions, and feature compression layers, have culminated in Meissonic, a 1B parameter model that outperforms larger diffusion models in high- resolution, aesthetically pleasing image generation while remaining accessible on consumer -agudde grade GPUs. Our evaluations demonstrate Meissonic's superior performance and efficiency, marking significant step towards accessible and efficient bigh-resolution non-autoregressive T2I MIM mod- els.
Broader Impact. Recently, offline text-to-image applications on mobile devices have emerged, such as Pixel Studio from Google Pixel 9 and Image Playground from Apple iPhone. These in- novations reflect a growing trend toward enhancing user experience and privacy. As a pioneering resource-efficient foundation model, Meissonic represents significant advancement in this field, delivering state-of-the-ar image synthesis capabilities with a strong emphasis on user privacy and offline functionality.
10

---

Published as a conference paper at ICLR 2025
Acknowledgements. This work supported in part by NUS Start-up Grant A-0010106-00-00, the Guangdong Science and Technology Department (No. 20242DZX2004), the Nansha Key Area Science and Technology Project (No. 20232D003) and the InnoHK funding launched by Innovation and Technology Commission, Hong Kong SAR.
We would like to express our gratitude all those who contributed their time, expertise, and insights during the development Meissonic. Listed in no particular order: Jingjing Ren, Sixiang Chen from HKUST(GZ), Wenhao Chai from University of Washington, Donghao Zhou from CUHK, and other anonymous friends. We are profoundly grateful for their commitment and the unique perspectives they brought to this project.
REFERENCES
Perfect deliberate. 2024.
Realvisxi v5.0.
2024.
Dreamlike Art. Dreamlike photoreal 2.0. https://mugghinaaa....///eeeeeee
dreaml seemmikkepheeee wwweeeeeeepppeeeeeeeee 2023.
Jinbin Bai, Wei Chow, Ling Yang, Xiangtai Li, Juncheng Li, Hanwang Zhang, and Shuicheng Yan. Humanedit: high-quality human-rewarded dataset for instruction-based image editing. arXiv preprint 4rXiv:2412.04280 2024.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 16609, 2023.
Fan Bao, Shen Nie, Kaiwen Xue, Yue Cao, Chongxuan Li, Hang Su, and Jun Zhu. All are worth words: A vit backbone for diffusion models. In Proceedings of the IEEE/CVF conference on computer vision and pallern recognition, pp. 22669-226777 2023.
Tim Brooks, Aleksander Holynski, and Alexei Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9392211402, 2023.
Huiwen Chang, Han Zhang. Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition (CVPR), pp. 11315-11325, 2022.
Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang. Kevin Murphy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image gen- eration via masked generative transformers. arXiv preprint arXiv:2301.000 00704, 2023.
Junsong Chen, Jincheng YU, Chongjian GE, Lewei Yao, Enze Xie, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, and Zhenguo Li. Pixart-S alphaa$: Fast training of diffusion transformer for photorealistic text-to-image synthesis. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Xi Chen, Nikhil Mishra, Mostafa Rohaninejad, and Pieter Abbeel. Pixelsnail: An improved au- toregressive generative model. International conference on machine learning (ICML), pp- 864-872. PMLR, 2018
IF DeepFloyd. Deepfloyd if, 2023. URL http::/huugiinggfce co/DeepFloyd, 2023.
Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang. and Mao Yang. Longrope: Extending Ilm context window beyond 2 million tokens. arXiv preprint arXiv:2402. 2024.
Patrick Esser, Robin Rombach, and Bjom Ommer. Taming transformers for high-resolution image synthesis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recogni- fion (CVPR), pp. 12873-12883 2021.

---

Published as a conference paper at ICLR 2025
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederie Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first International Conference on Machine Learning, 2024.
Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused frame- work for evaluating text-to-image alignment. Advances in Neural Information Processing Systems (NeurIPS), 36, 2024.
Lanqing Guo, Yingqing He, Haoxin Chen, Menghan Xia, Xiaodong Cun, Yufei Wang, Siyu Huang, Yong Zhang, Xintao Wang, Qifeng Chen, et al. Make cheap scaling: A self-cascade diffusion model for higher-resolution adaptation. arXiv preprint 104911
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. arXiv:2207.12598, 2022.
arXiv preprini
Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongscok Namkoong, John Miller, Hannanch Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
Minguk Kang. Jun-Jan Zhu, Richard Zhang, Jaesik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10124-10134 2023.
Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In Proceed- ings of the IEEE/CVF International Conference on Computer Vision, pp. 4015-4026, 2023.
Kolors. Kolors: Effective training of diffusion model for photorealistic text-to-image synthesis. arXiv preprint, 2024.
Doyup Lee, Chiheon Kim, Saehoon Kim, Minsu Cho, and Wook-Ship Han. Autoregressive image generation using residual quantization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11523-11532, 2022.
Xiuyu Li, Yijiang Liu, Long Lian, Huanrui Yang, Zhen Dong, Daniel Kang, Shanghang Zhang, and Kurt Keutzer. Q-diffusion: Quantizing diffusion models. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 17535-17545, 2023.
Zhikai Li, Xuewen Liu, Dongrong Fu, Jianquan Li, Qingyi Gu, Kurt Keutzer, and Zhen Dong. K-sort arena: Efficient and reliable benchmarking for generative models via k-wise human preferences. arXiv preprint ar 2408 14468, 2024.
Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
Bingchen Liu, Ehsan Akhgari, Alexander Visheratin, Aleks Kamko, Linmiao Xu, Shivam Shrirao, Joao Souza, Suhail Doshi, and Daiqing Li. Playground v3: Improving text-to-imas -mage alignment with deep-fusion large language models. arXiv preprint arXiv:2409.10695. 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024b.
Zeyu Lu, Zidong Wang, Di Huang, Chengyue Wu, Xihui Liu, Wanli Ouyang, and Lei Bai. Fit: Flexible vision transformer for diffusion model. arXiv preprint arXiv:2402.123376 2024.
Simian Luo, Yiqin Tan, Longbo Huang, Jian Li, and Hang Zhao. Latent consistency models: Synthe- sizing higl-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
Ron Mokady, Amir Hertz, Kfir Aberman, Yael Pritch, and Daniel Cohen-Or. Null-text inversion for editing real images using guided diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 6038-6047, 2023.
12

---

Published as a conference paper at ICLR 2025
Kaihang Pan, Siliang Tang, Juncheng Li, Zhaoyu Fan, Wei Chow, Shuicheng Yan, Tat-Seng Chua, Yueting Zhuang, and Hanwang Zhang. Auto-encoding morph-tokens for multimodal Ilm. arXiv preprint ar arXiv:2405.01926, 2024.
Suraj Patil, William Berman, Robin Rombach, and Patrick von Platen. amused: An open muse reproduction, 2024.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195 4205, 2023.
Pablo Pernias, Dominic Rampas, Mats Leon Richter, Christopher Pal, and Mare Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. In The Twelfth International Conference Learning Representations (ICL.), 2024.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müller, Joe Penna, and Robin Rombach. SDXL: Improving latent diffusion models for high-resolttion image synthesis. In The Twelfth International Conference on Learning Representations (ICLR), 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. 8748-8763. PMLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, (1(14)):167, 2020.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint 2XXi:22004001225 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems (NeurIPS), 35:36479-36444 -3444, 2022.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion dis- tillation. arXiv preprint arXiv:2311.17042 2023.
Axel Sauer, Frederic Boesel, Tim Dockhorn, Andreas Blattmann, Patrick Esser, and Robin Rom- bach. Fast high-resolution image synthesis with latent adversarial diffusion distillation. arXiv preprint arXiv:2403. 12015, 2024.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon. Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Tiitty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
Vikash Schwag, Xianghao Kong. Jingtao Li, Michael Spranger, and Lingjuan Lyu. Stretching each dollar: Diffusion training from scratch on a micro-budget. arXiv preprint 1xXiv:2407.15811, 2024.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In Proceedings ACL, 2018.
Shelly Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 8871- 1- 2024.
13

---

Published as a conference paper at ICLR 2025
Qingyu Shi, Lu Qi, Jianzong Wu, Jinbin Bai, Jingbo Wang. Yunhai Tong, Xiangtai Li, and Ming- Husan Yang. Relationbooth: Towards relation- aware customized object generation. arXiv preprint arXiv:2410.23228. 2024.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 0042:22006.06525, 2024.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint 2301::2302.1371,, 2023.
Narek Tumanyan, Michal Geyer, Shai Bagon, and Tali Dekel. Plug-and-play diffusion features for text-driven -ome-ag translation. In Proceedings of the IEEE/CVF Conference Com- puter Vision and Pattern Recognition, pp. 1921-1930, 2023.
Jianzong Wu, Xiangtai Li, Yanhong Zeng, Jiangning Zhang, Qianyu Zhou, Yining Li, Yunhai Tong, and Kai Chen. Motionbooth: Motion-aware customized text-to-video generation. Advances in Neural Information Processing Systems (NeurIPS), 2024a.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of text-to- image synthesis. arXiv preprint arXiv:2306. 09341, 2023.
Zike Wu, Pan Zhou, Xuanyu Yi, Xiaoding Yuan, and Hanwang Zhang. Consistent3d: Towards consistent high-fidelity text-to-3d generation with deterministic sampling prior. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9892-9902, 2024b.
Xuanyu Zike Wu, Qingshan Xu, Pan Zhou, Joo-Hwee Lim, and Hanwang Zhang. Diffusion time- step currieulum for one image 3d generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9948-9958, 2024.
Tianwei Yin, Michaël Gharbi, Taesung Park, Richard Zhang. Eli Shechtman, Fredo Durand, and William T Freeman. Improved distribution matching distillation for fast image synthesis. arXiv preprint ar arXiv:2405.14867 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang, Ruoming Pang. James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved VQ- GAN. In International Conference on Learning Representations (ICLR), 2022a.
Jiahui Yu, Yuanzhong Xu, Jing Yu Koh, Thang Luong, Gunjan Baid, Zirui Wang, Vijay Vasudevan, Alexander Ku, Yinfei Yang. Burcu Karagol Ayan, al. Scaling autoregressive models for content- rich text-to-image generation. arXiv preprint 27Xvv:2206.10789, 2(3):5, 2022b.
Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, and Jingjing Liu. Capsfusion: Rethinking image-text data at scale. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14022-14032, 2024.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guidee image editing. Advances in Neural Information Processing Systems, 36, 2024a.
Sixian Zhang, Bohan Wang, Junqiang Wu, Yan Li, Tingting Gao, Di Zhang, and Zhongyuan Wang. Learning multi-dimensionall human preference for text-to-image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8018-8022 8-887, 2024b.
Donghao Zhou, Jiancheng Huang, Jinbin Bai, Jiaze Wang. Hao Chen, Guangyong Chen, Xiaowei Hu, and Pheng-Ann Heng. Magictailor: Component-contue controllable personalization in -oxt-to-image diffusion models. arXiv preprint arXiv:2410. 13370, 2024.
14

---

Published as a conference paper at ICLR 2025
Le Zhuo, Ruoyi Du, Han Xiao, Yangguang Li, Dongyang Liu, Rongji Huang, Wenze Liu, Lirui Zhao, Fu- Yun Wang. Zhanyu Ma, et al. Lumina-nexs next: Making lumina- 12x stronger and faster with next-dit. arXiv preprint 24x.::2406.1888. 2024.
15

---

Published as a conference paper at ICLR 2025
A MODEL NAME ORIGIN
The name "Meissonic" is derived from a combination of the renowned French painter Ernest Meis- sonier and the term "sonic". Ernest Meissonier is celebrated for his meticulous attention to detail and his ability to capture dynamic moments in art. The addition of "sonic" evokes a sense of speed and modernity, highlighting the model's capabilities in efficient image synthesis and transformation.
B RELATED WORK
Diffusion-based Image Generation. Diffusion models have achieved remarkable advances in im- age generation, with notable contributions like Stable Diffusion (Rombach al., 2022), and the more recent SDXL (Podell et al., 2024), often driven by large-scale datasets. These models move beyond pixel-Ievel operations by working within compressed latent spaces, forming what we now recognize as latent diffusion models (Luo et al., 2023; Podell et al., 2024; Wu et al., 2024a; Shi al., 2024; Zhou et al., 2024; Yi et al., 2024; Wu et al., 2024b). SDXL represents significant leap in this domain, introducing micro-conditions and multi-aspect training to gain greater control over image generation, which has inspired wide range of derivative models in the community, such as Deliberate (per, 2024) and Real VisXL (rea, 2024).
The integration of transformer architectures has also become more prevalent, with models like DiT (Peebles & Xie, 2023) and U-VIT (Bao et al., 2023) demonstrating the potential of diffusion transformers in this field. SD3 (Esser et al., 2024), which combines diffusion transformers with impressive scale of 8B parameters, underscores the scalability and potential flow matching transformer- based diffusion backbone. Despite these advances, diffusion models of the multimodal still face challenges, particularly their reliance on acceleration techniques (Sauer et al., 2023; Luo et al., 2023; Yin et al., 2024) to speed up inference, making them cumbersome for real-time ap- plications. Additionally, the quantization of diffusion transformers has proven less straightforward than with large language models (Li et al., 2023). The research community continues to explore better paradigms for image generation. Addressing these limitations, our work aims to contribute an efficient, high-quality alternative in the form of Meissonic.
Token-based Image Generation. Token- autoregressive transformers (Lee et al., 2022; Chen et al., 2018; Yu et al., 2022b), first validated by VQ-GAN (Esser et al., 2021), have shown con- siderable promise for image generation. However, these methods are inherently computationally demanding, requiring the prediction of hundreds to thousands of tokens to form a single image. As a pioneering work, MaskGIT (Chang et al., 2022) challenged this paradigm by introducing a masked image modeling (MIM) approach, achieving competitive fidelity and diversity in class-conditions image generation. Building on this, MUSE (Chang et al., 2023) extended MIM to text-to-image synthesis, scaling up to 3B parameters and achieving remarkable performance.
MUSE demonstrates the viability of aon-eutoreeresssvve token- token-based models, but it encountered limi- tations in generating high-resolution images, capping at 512, and lagging behind SDXL (Podell et al., 2024) in terms of fidelity and text-image alignment. Meissonic advances the performance of tosen-based models beyond what latent diffusion methods have achieved, effectively pushing the envelope in terms of both quality and resolution in the text-to-image synthesis landscape with the MIM method.
MEISSÓNIC
Figure 10: Zero-shot generation of stylized letters. Meissonic can synthesize individual letters to form the word "MEISSONIC. Prompt: A post featuring a [COLOR] "[LETTER] painted on top.
16

---

Published as a conference paper at ICLR 2025
Figure 11: Memes generated by Meissonic.
Figure 12: Cartoon Stickers generated by Meissonic.
C APPLICATIONS
We present the letter synthesis capability of Meissonic Figure 10.
We present the combination capability of complex concepts of Meissonic in Figure 1.
We present meme generation in Figure 11.
We present cartoon sticker generation in Figure 12.
D PERFORMANCE COMPARISONS FOR COMPLEX VERSUS SIMPLE PROMPTS
We present performance comparisons for complex prompts versus simple prompts in Figure 13.
E PERFORMANCE COMPARISONS WITH DIFFERENT NUMBERS OF INFERENCE STEPS AND CLASSIFIER FREE GUIDANCE (CFG)
We present performance comparisons with different numbers of inference steps and Classifier Free Guidance (CFG) in Figure 14,15,16,17718,,,
17

---

Published as a conference paper at ICLR 2025
flowers of coffee on top it, accomponied by plate of buttery creissonts, folded linen napiún, and faint ray of surlight streaming through nearby window a cozy dinirg room.
white
A white table with vase of flowers and cup of coffee on top it.
Table flowers.
busy trein station with people hurrying ciong platforms, corrying kugagg, while sleek modkurn arnwing, headlights gloss haze,
A busy train station with people hurrying along the platforms.
Train station.
rising chimney, survounded by A wooden cabin covered in pine. traes, snow/lokes gray sky, and worm yellow glow from windows blanket of snow.
Snow cabin.
inxites you in.
city night skyscropers illuminated by neon lights, busy streets filled with billboard flashing colorful azvertisements, while a clear night sky reveals the faint twinkle distont
A vibrant city at night with skyscrapers illuminated by neon lights.
Night city.
people, and towering
Figure 13: Performance Comparisons for Complex versus Simple Prompts
18

---

Published as a conference paper at ICLR 2025
Steps 7
Staps: 12
40
Stops
60
Figure 14: Performance Comparisons with Different Numbers of Inference Steps and Classifier Free Guidance (CFG). Prompt: A statue of man with crown on his head.
19

---

Published as a conference paper at ICLR 2025
CFG::
1
Staps
Stops:
Stops: Staps
Figure 15: Performance Comparisons with Different Numbers of Inference Steps and Classifier Free Guidance (CFG). Prompt: Studio photo portrait of Lain Iwakura from Serial Experiments Lain wearing floral garlands over her traditional dress.
20

---

Published as a conference paper at ICLR 2025
CFG:2
CFG:
1
Stops:
Staps
Stops:
Stops:
Staps
Figure 16: Performance Comparisons with Different Numbers of Inference Steps and Classifier Free Guidance (CFG), Prompt: A girl gazes at a city from a mountain at night in colored manga illustration by Diego Facio.
21

---

Published as a conference paper at ICLR 2025
Staps:
Stope:
Stops 7
Stops:
Stops,
Staps: 12
Stops:
Stops
Stops:
Steps:
Stoppe
60
Figure 17: Performance Comparisons with Different Numbers of Inference Steps and Classifier Free
Guidance (CFG). Prompt: A tranquil lake surrounded by snow-capped mountains under clear sky.
22

---

Published as a conference paper at ICLR 2025
Staps:
Stope:
Stops 7
Stops:
Staps,
Stape: 12
Stops
Stoppe
60
Figure 18: Performance Comparisons with Different Numbers of Inference Steps and Classifier Free
Guidance (CFG). Prompt: A futuristic cityscape with hovering vehicles and towering structures.
23

---

Published as a conference paper at ICLR 2025
7
Staps: 12
Stops
60 Stupps
Figure 19: Performance Comparisons with Different Numbers of Inference Steps and Classifier Free
Guidance (CFG). Prompt: A massive starship docked in a glowing nebula.
24

---

Published as a conference paper at ICLR 2025
F MORE COMPARISONS FOR ZERO-SHOT IMAGE EDITING ABILITY
To ensure fair evaluations of zero-shot capabiliiiiss with SDI.5 and SDXL, utilize Null-Text Inversion (Mokady et al., 2023) for zero-shot editing with our method, taking into account that other methods have been extensively trained editing datasets. The configurations used for Null-Text Inversion, along with any undocumented parameters, align with those provided in the official code repository. The primary parameters are outlined follows:
cross.replace.step.
default deffaul 0.8
self.replace.steps = 0.5
blend = None
equilizer.paraaee
None
For consistency, we used the recommended 512 512 resolution for editing and ran tests using
torch. f1cat32, which is the official setting for Null-Text Inversion. On A6000 GPUs (48 GB), the execution of MagicBrush (Zhang et al., 2024a) took approximately 36 hours for SD1.5 and 60 hours for SDXL. The runtime for Emu-Edit was significantly longer. Given the extensive computa- tion, we randomly sampled 500 examples per benchmark for testing.
We present more comparisons for zero-shot image editing ability EMU-Edit in Table 7.
CLIP-If
CLIP-Tf
DINO
L11
CLIPdir
SD 1.5 + Null-Text Inv.
0.780
0.240
0.637
0.159
0.096
SDXL Null-Text Inv.
0.787
0.238
0.653
0.146
0.085
Meissonic-511 (Ours)
0.791
0.244
0.689
0.128
0.102
Table 7: EMU-Edit Results
We present more comparisons for zero-shot image editing ability on MagicBrush in Table 8.
CLIP-It
CLIP-T+
DINOT
L14
CLiPdirT
SD 1.5 + Null-Text Inv.
0.824
0.228
0.647
0.121
0.106
SDXL Null-Text Inv.
0.840
0.241
0.665
0.122
0.111
Meissonic-512 (Ours)
0.835
0.248
0.689
0.115
0.120
Table 8: MagicBrush Results
Our findings indicate that due to the inherent characteristics of MIM, Meissonic exhibits faster zero- shot editing capabilities. Performance was evaluated with batch size 1 and inference step 50 (compared to Null-Text Inv., which requires 500 backpropagation steps). Tests were conducted on an A6000 GPU with 48 GB VRAM.
Besides, we present inference time comparision in Table 9.
SD 1.5 Null-Text Inv.
SDXL + Null-Text Inv.
Meissonic-512 (Ours)
Time (s/10 pairs) GPU (GB)
1040 + 100
1850 + 120
108
13.4
26.8
5.9
Table 9; Inference Time Comparison
These results demonstrate the substantial potential for reduced processing time with Mcissonic.
We also present qualitative comparisons on zero-shot image editing ability in Figure 20.
G MORE COMPARISONS WITH SDXL FOR IMAGE GENERATION ABILITY
We present comparisons with SDXL for image generation ability in Figure 21,22,23.
25

---

Published as a conference paper at ICLR 2025
SD 1.5
SDXL
Meissonic-510
Edit Prompt: A wooden bear
Edit Prompt:
Mountains with a river flowing between them
Edit Prompt: A cat wearing hat
Figure 20: Qualitative comparisons on zero-shot image editing ability.
Meissonic
SDXL
Qualitative comparisons with SDXL for image generation ability. Prompt: A breath- taking photo of a serene mountain lake at sunrise, crystal-clear water reflecting the surrounding snow-capped peaks, with soft mist floating above the surface.
Figure 21:
H ABLATION STUDY
Detailed roadmap to build Meissonic. We present ablation studies during training Meissonic-511 in Table. 24, The HPS v2.1 (Wu et al., 2023) scores calculated for verifying the effectiveness of
26

---

Published as a conference paper at ICLR 2025
Meissonic
SDXL
Figure 22: Qualitative comparisons with SDXL for image generation ability. Prompt: A professional studio photograph fresh bouquet of wildflowers in glass vase, water droplets visible on the petals and leaves, placed on clean white background.
Meissonic
SDXL
Figure 23: Qualitative comparisons with SDXL for image generation ability. Prompr: A sharp photo of modern skyseraper during blue hour, its glass facade reflecting the city lights and the deep indigo sky in the background.
each compoment. Our ablations are based on training stage 2, ensuring consistency with the training dataset scale, model scale, and other training configurations.
27

---

Published as a conference paper at ICLR 2025
Transformer baseline
26.36
add mricro_conde
27.85
finetune text enc
28.19
finetune vqvae
28.14
more inference steps
28.74
add RoPE
30.17
reduce training data
29.32
add masking rate conds
30.77
Meissonic-512
30.94
28
30
32
Figure 24: HPS v2.1 Score on internal 1000 prompts
Linear
Linear
Qc
LayerNore
Modulation
Linear
RoPE
K
LayerNomn
Softmax Attention
Ve
Modulation
RePt
Qx
LayerNore orr
LayerNore
Linesr
Linear
Linear
Linear
Figure 25: Multi-modal Transformer For MIM.
I OUR MULTIMODAL TRANSFORMER BLOCK
We present a detailed structure Multi-modal Transformer Block for MIM in Figure 25. Specif- ically, x denotes image embedding inputs, denotes text embedding inputs, and y denotes conditions inputs.
WORD CLOUD OF OUR REALUSERS800 BENCHMARK
We present a word cloud image that illustrates the diverse concepts, styles, and themes encompassed within RealUser-800 prompts benchmark in Figure 26.
28

---

Published as a conference paper at ICLR 2025
bT
Small
K
Dainting
Til
painting
beaut 1ful
eoman
sel
CRED
Jent
Ba young
KiNdON
rec
V
De
image
close
depiction
riding
daga
nolding
FOGA
paintiing
phato
W
gh
bolue
characia
THE
gointed por orrgaa
goodess
standing
buldin
inline
gir
anmor
haiir
Cybgi RUNK
pose
Inside
CA agon
fron
created
Artstation
art
Lisp re
intr
Fantasy
Saraa gatrnd
DLG sitting
Cistonnn Macie
eyee
Placc dar
aare
baby
Looging
detailed
dog
cat
t
Ting
Frending
na
g
dig LEET
Fan
art POO
tee
backk
an
g ound
TNE
our Lots artist
Figure 26: Word cloud image of RealUser-800 prompts benchmark.
K IMAGES GENERATED DURING DIFFERENT TRAINING STAGES
We present images generated using the same prompt across Meissonic's four training stages in Fig-
ure 27.
L MORE EXAMPLES OF QUALITATIVE COMPARISONS
We present more examples of qualitative comparisons in Figure 28.
M MORE IMAGES PRODUCED BY MEISSONIC
We present additional images generated by Meissonic using CC3M (Sharma et al., 2018) items, with detailed captions provided by VILA-1.5 (Lin et al., 2023) and Morph (Pan al., 2024). These images can be found in Figure 29,30,31,33300 3,,3,,,,,,,,,3,3,,,,, 7,38,33,,, ,3,39, 2,,,,,,,,,,11,2,444,,,4,,, 40,44,,,,,,4444444444,,,,,,,,,
We present additional images generated by Meissonic using HPS (Wu al., 2023) benchmark prompts. These images can be found in Figure 48, 5,,,,,533,,,
N MORE IMAGES PRODUCED BY MEISSONIC AT DIVERSE RESOLUTIONS
We present additional images generated by Meissonic at diverse resolutions. These images can be found in Figure 54,55.
29

---

Published as a conference paper at ICLR 2025
Stage 1
Stage
Stage
Stage 4
An image of Pikachu wearing birthday hat and playing guitar
black and white line drawing of rhinaceros head.
leopard is sitting on tree branch in forest with its front paws resting on the trunk.
sculpture of Greek woman head with headband and a head of hair.
white sports car is driving down desert road.
Figure 27: Images generated using the same prompt across Meissonic's 's four training stages. The resolutions for stages and are 256² and 512a, respectively, while stages and are 1024 For clarity and comparison, all images are displayed in consistent layout.
30

---

Published as a conference paper at ICLR 2025
Spiderman Wolverine with detailed muscular features and full face, trending multiple art platforms, created with hyperdetailed Unreal Engine, and eptimized for high resolution viewing.
digital painting of Pokémon a concept style.
The image features monks resembling Rasputin from The Lorax, cinematic lighting shallow depth of field.
The image depicts God smashing mirrors, while detailed unicom-dragon present scene.
Architecture render with pleasing oesthetics.
Exploded view diagram of xenomorph.
Samurai in space.
SD 1.5
SD 2.1
DeepFloyd XL
Deliberate
SDXL 1.0
Meissonic
Figure 28: Qualitative Comparisons with SD 1.5, SD 2.1, DeepFloyd- -XL, Deliberate, and SDXL.
31

---

Published as a conference paper at ICLR 2025
The wizard chants a spell over the apple
Pumpkin head wearing black wizard hat
Two women in black dresses with feathers on their heads.
A bedroom with a canopy bed and a wooden floor
sled sits field with sunset the background.
A blue and white drawing of dragon.
Figure 29: High Quality Samples Produced by Meissonic.
32

---

Published as a conference paper at ICLR 2025
body of water with a cliff in the background
A collection of statues of Asian men and women.
man blonde hair and glasses looking at the camera.
A table with parrot on a map on
beautiful sunset with reflection of the Marina Bay Sands hotel.
Two showmen standing next snowmon with blonk sign.
Figure 30: High Quality Samples Produced by Meissonic.
33

---

Published as a conference paper at ICLR 2025
people walking the snow with sled.
A tiger is swimming a body of water.
man with crown and blue robe holding glass.
A seal is sitting the snow with its mouth open.
small bird is perched wooden post.
An old man with blue turban and blue shirt is standing in front of wooden wall.
Figure 31: High Quality Samples Produced by Meissonic.
34

---

Published as a conference paper at ICLR 2025
small cute white dog sitting on bed.
A woman is laying on couch and smiling.
neertop
cloudy sky over body water.
A man is holding large black white dog a black-white figure style.
white with silver rim and headlight.
A young girl is holding bouguet flowers.
Figure 32: High Quality Samples Produced by Meissonic.
35

---

Published as a conference paper at ICLR 2025
in black leather suit sits red chair.
A bed with red and white quilt
A dog with blue collar is looking at the comera.
A woman in white dress is looking at her phone.
A statue of a man in front of a building.
A seal is wearing Santa hat and is on a snowy hill with the words Happy New Year written below it.
Figure 33: High Quality Samples Produced by Meissonic.
36

---

Published as a conference paper at ICLR 2025
plush toy of a girl with red eyes and pink shirt.
A woman with a flower crown on her head.
doll wearing white dress and tan shawl.
A metal sculpture of deer with antlers.
penguin walks in the with red hat on.
A red jersey holding basketball.
Figure 34: High Quality Samples Produced by Meissonic.
37

---

Published as a conference paper at ICLR 2025
A cat is looking at a butterfly
A woman in a white wedding dress stands in courtyard.
Two firefighters standing front of a smoky background.
A model walks down a runway a black dress.
man wearing orange hat and scarf is screaming
A black and white drawing of dog's 's head in circle.
Figure 35: High Quality Samples Produced by Meissonic.
38

---

Published as a conference paper at ICLR 2025
statue of a woman surrounded by flowers
A chess board with row of chess pieces.
bronze statue of an owl with wings spread.
A woman a pink shirt is sitting on bed.
group of women ned uniforms pose for picture
A gold mask with gold strap is black surface.
Figure 36: High Quality Samples Produced by Meissonic.
39

---

Published as a conference paper at ICLR 2025
white goat with horns is standing the snow.
A map of Africa with blue background.
woman stands on a dock the fog.
A woman standing next picture of onother woman.
man wearing virtual reality headset.
A white table with of flowers and cup of coffee top of it.
Figure 37: High Quality Samples Produced by Meissonic.
40

---

Published as a conference paper at ICLR 2025
white and blue coffee mug with picture a man on it.
A statue of man with crown on his head.
woman in black wetsuit on bench gazing the sea on the beach.
Four bottles of maple syrup in different colors.
soccer player in blue and white uniform runs with the ball.
A in yellow wet suit is holding big black dog in the water.
Figure 38: High Quality Samples Produced by Meissonic.
41

---

Published as a conference paper at ICLR 2025
pillow with a picture of man on it.
An Indian woman is wearing white saree and standing in front of pink wall.
large ship the water with a foggy bockground.
A woman holding a baby.
An ancient Egyptian carved stone wall with three figures and hieroglyphics.
A snowy owl sitting in the
Figure 39: High Quality Samples Produced by Meissonic.
42

---

Published as a conference paper at ICLR 2025
A woman is sitting on beat and looking at boat in the water.
A womon drinking from cup with blurry background.
puffin sitting on rock looking off into distance.
A large body water with a rock in the middle and mountains the background.
lynx standing in the
Two actors posing for picture with wearing black and white face paint.
Figure 40: High Quality Samples Produced by Meissonic.
43

---

Published as a conference paper at ICLR 2025
statue of Jesus Christ is holding a feather in his hand in a purple style,
A dog is laying on the floor.
black boat tied a dock on calm lake.
A narrow stone pathway is enveloped by lush greenery a veil of mist.
white and black motorcycle with headlight on it.
A woman with short red hair is looking off into the distance.
Figure 41: High Quality Samples Produced by Meissonic.
44

---

Published as a conference paper at ICLR 2025
statue of a lion stands in front of a building.
A bathroom with modern design and classic design.
UA
frozen river with ice on surface.
Benjamin Franklin appears among pile of dollars
Pope Francis is talking black priests.
Cherry blossoms bloom under the Eiffel Tower.
Figure 42: High Quality Samples Produced by Meissonic.
45

---

Published as a conference paper at ICLR 2025
young girl holding bowling ball.
A pair of bride and groom figurines are pasitioned atop white, two- coke.
ship sailing the ocean with mountains in the background.
A woman a gold dress poses photo.
wearing headband hat and white dress is walking down runway.
A squirrel is holding a gift bag with mouse open in the snow.
Figure 43: High Quality Samples Produced by Meissonic.
46

---

Published as a conference paper at ICLR 2025
man with long hair and beard stands room, with portrait of himself positioned behind him.
A gorilla looking at the camera serious expression.
sunset over body of water with tree in small island.
A man with a hoodie on looking at the camera.
The collage consists of photos featuring the bride and groom. The bride occupies half of the collage. The groom appeors two photos, one white suit and the other in black suit.
A black and white photo of in field.
Figure 44: High Quality Samples Produced by Meissonic.
47

---

Published as a conference paper at ICLR 2025
1
woman with short hair and earrings smiling.
A man dressed Viking attire is seated among the crowd.
dog sitting the snow in front mountain.
Two gloden statues of lions standing in field.
is driving track.
A guitar is sitting wooden floor in front of purple wall.
Figure 45: High Quality Samples Produced by Meissonic.
48

---

Published as a conference paper at ICLR 2025
A man with shaved head and tattoo on his back.
A deer drawn a geometric style.
woman is standing on a staircase, back camera three chains hanging from the ceiling.
A baby sitting on white blanket holding white rose.
woman wearing and necklace is smiling.
surreal mental landscape, which elements of nature and house emerge from the back of womon's head.
Figure 46: High Quality Samples Produced by Meissonic.
49

---

Published as a conference paper at ICLR 2025
forest with trees and fog.
A cloudy sky over green field.
man wearing a large blue and gold feathered headdress.
image depicting a minimalist design featuring a pool situated front of white building with palm
lion's head shown in grayscale image.
Three origumi dogs, wh purple, while the others yellow.
Figure 47: High Quality Samples Produced by Meissonic.
50

---

Published as a conference paper at ICLR 2025
blind monk weoring an orange robe stares out the window of spaceship in a dramatic lighting as depicted in matte painting.
A racoon wearing suit smoking cigar in the style of James Gurney.
Classical romantic painting of Hatsune Miku with blue hair.
An astronaut floats amidst planets against cosmic backdrop in a highly detailed, refreshing digital painting by James Jean.
T
Close-up hyperrealistic painting portrait of nun fashion model loo looking up against black background, classicism and 8Os sci-fi Taanese book art influences.
A digital painting of hairless, inside- Inside-out cat with intricate details and horror theme.
Figure 48: High Quality Samples Produced by Meissonic.
51

---

Published as a conference paper at ICLR 2025
raccoon formal attire, carrying bag and cane, depicted in a Rembrandt-style painting.
A cinematic fashion portrait a Hindu goddess standing a beautiful garden.
Landrover crosses a forest path the rain a highly-detailed digital painting by artists Greg Rutko wski and Artgerm.
A portrait of Mario and Luigi from Mario Bros with detailed face a city bac background painted by Bouguereau.
Image Albert Einstein created Park Jun Seong.
A painting depicting character standing on roof under moonl night.
Figure 49: High Quality Samples Produced by Meissonic.
52

---

Published as a conference paper at ICLR 2025
Steve Buscemi portrays the Joker.
depicts stermtreopers hyp listic style, with
intrice he and hyper detiiled design, chare la by ambiant and volumetric lighting, ran inicent of Star Wars concept ort by George Lueas and kelph McQuaris, with o style milerr to GTA
Image depicting person's face composed entirely of fruits and vegetables.
A space man sat on beach chair the moon, pixel ort.
A cyberpunk-style Batman in dark city, depicted in extremely detailed piece of artwork by Chris Labrooy.
The image trippy cheeseburger with warm colors, depicted in highly detailed illustration and rendered in octone, created by the award winning studio 4.
Figure 50: High Quality Samples Produced by Meissonic.
53

---

Published as a conference paper at ICLR 2025
Salem black cat girl in anime style with simple background.
A cute anthropomorphic fox knight wearing a cape and crown pale blue armor.
Blond-haired girl depicted anime style.
A cute anime-style female cat girl with large eyes is pictured derwaterr with simple background.
girl peers the edge of mountain giant city the dark of night, depicted in manga illustration by Kentaro Miura and Hiromu Arakawa.
Illustration of maid with pretty face and eyes, shown full- body upper shot.
Figure 51: High Quality Samples Produced by Meissonic.
54

---

Published as a conference paper at ICLR 2025
A full-body shot of an anime maid with rich detail, featuring pretty face and eyes.
A minimalist tattoo inspired by the Studio Ghibli films
puppy driving a car a film still.
Little Prince talking the fox in an animation shot by Tim Burton's s art.
Anime portrait of Asian schoolgirl with her pet sugar glider.
Luke Skywalker with Muppets.
Figure 52: High Quality Samples Produced by Meissonic.
55

---

Published as a conference paper at ICLR 2025
Medium shot black and white manga pencil drawing with highly de datalled face Alita by Yukito Kishiro.
Studio photo portrait of Lain Iwakura from Serial Experiments Lain wearing floral garlands over her traditional dress.
Frontal portrait anime girl with pink hair wearing white t-shirt and smiling.
Anime oil painting of from Zero
Anime-style fighter pilot cockpit engaged in night air battle with explosions.
A girl gazes city from mountain at night in colored manga illustration by Diego Facio.
Figure 53: High Quality Samples Produced by Meissonic.
56

---

Published as a conference paper at ICLR 2025
960 1280, A dense jungle with sunlight filtering through the canopy.
1280 x 960, A large body of water with rock in the middle and mountains the background.
960 1280, A massive starship docked in glowing nebula.
1280 1024, white table with vase of flowers and a cup of coffee top of it.
960 1280, mystical temple hidden deep within a coud-covered mountain.
Figure 54: More Images Produced by Meissonic at Diverse Resolutions.
57

---

Published as a conference paper at ICLR 2025
960 x 1280, quiet meadow bathed in soft morning dew.
1280 x 960, A frozen lake surrounded by snow-covered trees under pale winter sun.
1024 x 1280, A stormy sea with crashing
waves and lightning illuminating the clouds.
1024 x 2048, A frozen lake surrounded by snow-covered trees under pale winter sun
Figure 55: More Images Produced by Meissonic at Diverse Resolutions.
58


Reviews:
[{"review_id": "1IoJd9L5LP", "reviewer": "Reviewer_TngQ", "summary": "- The paper presents Meissonic, a high-resolution, non-autoregressive masked image modeling (MIM) approach for text to image generation. \n- It introduces architectural innovations to elevate MIM’s performance to rival diffusion models like SDXL. \n- Its key advancements include a multi-modal transformer architecture, Rotary Position Encoding (RoPE), feature compression layers, and micro-conditions based on human preference scores. \n- It enables high-fidelity image generation at 1024×1024 resolution on 8GB VRAM. \n- The model is trained on high-quality datasets and achieves state-of-the-art results on various benchmarks.", "questions": "Since Meissonic uses a CLIP text encoder, which might limit its effectiveness in scenarios demanding high text comprehension (e.g. generating images with visual texts), could the authors provide examples comparing Meissonic's performance on complex prompts versus simpler prompts? Also, would there be noticeable performance degradation if the model were applied to open-domain data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The integration of multi-modal transformers with single-modal refinements optimizes MIM for high-resolution synthesis and addresses the historical gap in resolution and quality compared to diffusion models.\n- The model's efficiency is notable as it requires only 48 H100 GPU days which is far less than other high-resolution models like SDXL while achieving comparable performance.", "weaknesses": "- Using CLIP text encoder makes Meissonic potentially less adept at handling complex text prompts compared to models using larger language models like T5 which could impact its effectiveness in scenarios demanding high text comprehension. e.g. generating images with visual texts.\n- The reliance on high-quality datasets may limit the model's generalization in open-domain applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9FoyVYTx0l", "reviewer": "Reviewer_BVBS", "summary": "The paper scales up a masked image modeling (MIM) framework for professional-quality image generative model comparable to the diffusion-based SDXL. The key problems the authors have find in MIM framework is its low resolution constraint and persisting performance gap. The newly proposed model, Meissonic model incorporates efficient text encoder, mixed single-modal and multi-modal transformer backbone using rotary positional encoding, and curated training datasets. The final model performs comparable to SDXL in quality and exceeds efficiency in inference time and memory requirement.", "questions": "These are questions that are minor enough that do not count in the final scores.\n\n- Does this model architecture supports DreamBooth-type fine-tuning for practical application?\n- The success of U-Net-based architecture (e.g., SD, SDXL) partially relies on the strong controllabiliy gained from ControlNet. However, such I do not want to require the authors to provide the experimental results or detailed idea, but I wonder if they are planning to develop in this direction.\n- I believe it would be great to have Figure 11 in the main manuscript, since this shows justification of changes made during the development of the proposed model.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The presented method is lightweight, in terms of the number of parameters and the training time, lowering production cost of large scale image generative models.\n- The model also have reduced inference time with lower GPU memory cost compared to SDXL.\n- The training procedure is opened with transparency as shown in Figure 15.\n- The model supports masked image editing and inpainting.\n\nOverall, the proposed method in this manuscript can be regarded as a good industrial image generative model with production-ready quality.", "weaknesses": "The main concerns are in the presentation of the material. Although the model at this reported performance can contribute to the generative AI community, the current form of manuscript seems more like an industrial technical report than an academic article.\n\n- Since the authors have claimed for supporting inpainting, outpainting, mask-free editing, and masked editing, there should be a comparison in both quantitative and qualitative way for each of the task in the manuscript. For now, the article only contains EMU-Edit scores with reduced number of columns--the original EMU-Edit contains five algorithmic ratings (CLIP_dir, CLIP_im, CLIP_out, L1, DINO) and two human ratings (Text Alignment, Image Faithfulness), measured in two different test sets (Emu Edit / MagicBrush). Also, the qualitative comparison is not shown in the paper, showing only the proposed method in Figure 6 and 7. Overall, this writing style makes the article more of an well-written industrial technical report than an academic paper. As a recommendation, there are many well-established detailed quantitative metrics such as masked image preservation and text alignment (e.g., PSNR/LPIPS/MSE as in BrushNet, Table 2), and overall generation quality (e.g., FID as in Table 2 of PQDiff), which can be applied to compare with, for example, SDXL.\n- Maybe it is due to the intrinsic characteristics of MIM framework, but the inpainting in Figure 7 seems to change the non-masked regions of the image. This deformation should be measured in quantitative metrics such as in BrushNet.\n- It would be better to have plots depicting how the generated quality changes by changing the number of iterations in the inference, or at least a recommendation of the number of inference steps of using the pre-trained model.\n- I believe image demonstration from page 20 to the end is unnecessary and carry little information about the model. It is not bad to have a lot of demonstration images in a technical report, but there should be at least qualitative comparison between other models in order to have persuasive narrative. For example, generating random 5-16 images from the same text prompt without curation and show the diversity and the point of breaking (e.g., broken hands or limbs) can be visually compared with other large scale generative models such as SDXL-base, which is quoted multiple time throughout the manuscript.\n- I would like to recommend adding details about training and inference algorithms. For example, the explanation of “micro controls” and architectural detail of “feature compression layers” is not described in appropriate level of details (what is chosen, how's the size of this controls, and how each component benefits the training, etc.). Moreover, even if the original idea is borrowed from previous papers, such information should be contained in the paper for completeness, e.g., how is 1D RoPE encoding used in this specific 2D MIM Transformer block?\n\nRegarding the mentioned issues, I believe that the current version of the manuscript is not ready for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bK4vdnkXUg", "reviewer": "Reviewer_qRU5", "summary": "The paper explores Masked Image Modeling (MIM), an non-autoregressive technique, and propose a new model named Messonic. Messonic poses a promising enhancement on MIM compared to its auto-regressive and diffusion-based counterparts. The keys to the improvement of Messonic are enhanced transformer architecture, advanced positional encoding and adding masking rate as a condition. The authors conduct extensive evaluations on Messonic to demonstrate its superiority.", "questions": "1. It is interesting that the diverse micro conditions (Line 229 - Line 232) are helpful for the training stability (Line 134). From a high level perspective, why would these conditions help with the stability?\n2. It is interesting to see that you choose to use a \"lightweight\" text encoder, CLIP (Line 210), instead of using LLM encoders or a mixed way like in SD3. Is there a reason or a hypothesis behind this choice, not just a \"trial-and-error\" experience?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "### Motivation\n- The paper shows a good motivation - MIM shows the potential capability to unify the language and vision modalities, and the paper attempts to bridge the gap between MIM and other popular image generation fashions. \n- The paper shares several interesting observations on the current MIMs, including architecture design, positional encoding, and masking rate as a condition. It then starts from those insights and implement effective components to gain improvement. \n\n### Method\n- The proposed pipeline is straight-forward and seems effective. \n- It is an interesting finding that embedding original image resolution, crop coordinates, etc would be helpful for the stability of the training, as mentioned in Line 229 - Line 232. \n- The decomposition of training stages (Section 2.5) makes sense to me. It could work as a new procedure for other or future model training. \n\n### Experiments/Results\n- The authors conduct extensive experiments and evaluations to demonstrate the performance of Messonic. \n- Messonic shows superiority on HPS benchmark, GenEval benchmark, and MPS scores. It shows competitive results with SDXL, while has a 2x smaller model size and transparent training costs (Table 1).\n- Messonic also shows an economic expense of VRAM (i.e., 8GB). This demonstrates a promising value for customer level usage. \n- Messonic shows a good performance in Zero-shot I2I task, as shown in Table 6 and Figure 5.\n\n### Writing/Delivery\nThe paper is well-written. The technical parts are clear and easy to understand.", "weaknesses": "I do not see a critical weakness from this paper. Some minor points:\n- Messonic seems to be designed to match the auto-regressive manner in language models. The authors indicates it aims to unify language-vision models (Line 014). However, I do not see a strong evidence to show the current Messonic achieves this goal in the paper.\n- The proposed approach is more driven from an \"add-on\" manner, where most of the components in Messonic are \"borrowed\" from other works. For example, multi-modal transformer backbone from the ADD paper, RoPE from Roformer, tokenization from VQVAE. The core contribution of the paper, therefore, seems to be a practice of choosing these components and empirical trials.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)