Lumina-mGPT: Illuminate Flexible Photorealistic Text-to-Image Generation with Multimodal Generative Pretraining

## Abstract

Abstract We present Lumina-mGPT, a family of multimodal autoregressive models capable of various vision and language tasks, particularly excelling in generating flexible photorealistic images from text descriptions. Unlike existing autoregressive image generation approaches, Lumina-mGPT employs a pretrained decoder-only transformer as a unified framework for modeling multimodal token sequences. Our key insight is that a simple decoder-only transformer with m ultimodal G enerative P re T raining (mGPT), utilizing the next-token prediction objective on massive interleaved text-image sequences, can learn broad and general multimodal capabilities, thereby illuminating photorealistic text-to-image generation. Building on these pretrained models, we propose Flexible Progressive Supervised Finetuning (FP-SFT) on high-quality image-text pairs to fully unlock their potential for high-aesthetic image synthesis at any resolution while maintaining their general multimodal capabilities. Furthermore, we introduce Ominiponent Supervised Finetuning (Omni-SFT), transforming Lumina-mGPT into a foundation model that seamlessly achieves omnipotent task unification. The resulting model demonstrates versatile multimodal capabilities, including visual generation tasks like flexible text-to-image generation and controllable generation, visual recognition tasks like segmentation and depth estimation, and vision-language tasks like multi-turn visual question answering. Additionally, we analyze the differences and similarities between diffusion-based and autoregressive methods in a direct comparison. All code and checkpoints will be available on our GitHub repository: https://github.com/Alpha-VLLM/Lumina-mGPT .

## 1 Introduction

Figure: Figure 1: Selected photorealistic images of various resolutions generated by Lumina-mGPT. At the bottom, we include two examples to demonstrate the omnipotent task unification in Lumina-mGPT.
Refer to caption: x1.png

Seminal models, including DALL-E 3 (Betker et al., ), Stable Diffusion 3 (Esser et al., ), and SoRA (Brooks et al., ), have demonstrated superior performance in photorealistic image and video generation using diffusion-based generative modeling over continuous latent image features. In contrast, autoregressive (AR) generative models, which rely on “next-token prediction,” have revolutionized text generation with groundbreaking reasoning abilities, as exemplified by models like GPT-4 (Achiam et al., ) and Gemini (Team et al., ), over discrete token representation. However, AR-based generative modeling over vector-quantized image features still lags far behind diffusion-based counterparts in terms of photorealistic and controllable image generation.

Although previous autoregressive efforts, such as DALL-E (Ramesh et al., ), CogView (Ding et al., ), Parti (Yu et al., ), OFA (Wang et al., ), Unified-IO (Lu et al., ; ), LlamaGen (Sun et al., ), and Chameleon (Team, ), have extensively explored generative modeling over vector-quantized image features following the paradigm of large language models (LLMs), their results on text-to-image generation have either been unsatisfactory or produced high-quality samples limited to academic benchmarks like ImageNet (Deng et al., ). We summarize the limitations of previous approaches below:

Randomly-Initialized Transformer While transfer learning has revolutionized key fields such as visual recognition (He et al., ; Radford et al., ; Lu et al., ) and language generation (Raffel et al., ; Radford et al., ; ; Brown et al., ), popular autoregressive image generation approaches such as DALL-E, Parti, and LlamaGen all adopt a randomly-initialized causal transformer, which fails to utilize pretrained transferable representation and large-scale datasets. As a result, AR-based approaches often lead to poor image generation quality and slow convergence without leveraging proper large-scale pretraining.

Verbose Encoder-Decoder Architeture DALL-E and CogView initially propose using a decoder-only transformer for image generation with discrete representation, where a single transformer acts as both a text encoder and an image token decoder. However, subsequent approaches, such as Parti and LlamaGen, adopt a verbose encoder-decoder architecture that injects frozen T5 text features (Raffel et al., ) using cross-attention or prefix-filling approaches, motivated by the findings of Imagen (Saharia et al., ). Compared to the trend in scaling LLMs (Touvron et al., ; Bai et al., ; Bi et al., ), such encoder-decoder architecture is cumbersome due to the decoupling of text encoding and image token modeling. This design significantly complicates the autoregressive-decoding framework, limits the scalability of image generation, and hinders the generalization to additional modalities and tasks.

Limited Decoding Resolution and Flexibility Natural images exsit in various resolutions and aspect ratios. Advanced diffusion models (Chen et al., ; ; Esser et al., ; Team, ; Li et al., ; Zhuo et al., ; Lu et al., ) can successfully generate diverse photorealistic images at arbitrary resolution with skewed ratios. In contrast, current AR-based approaches (Yu et al., ; Team, ; Sun et al., ) rely on central-cropping a low-resolution 512 $\times$ 512 image and transforming the cropped low-resolution image into a fixed-length sequence of discrete tokens using a pretrained Vector-Quantized Variational Autoencoder (VQ-VAE) (Van Den Oord et al., ; Razavi et al., ; Esser et al., ). This approach simplifies autoregressive training but at the cost of deteriorated image quality and generation flexibility.

Poor Task Extensability Autoregressive modeling excels at unified generative modeling for various tasks and modalities using discrete tokens (Wang et al., ; Lu et al., ; ). However, previous AR-based image generation approaches (Ramesh et al., ; Ding et al., ; Yu et al., ; Sun et al., ) have been limited to text-to-image generation without exploring the unification with other tasks, such as dense labeling and controllable image generation. This lack of task extensibility significantly constrains the applicability of AR-based image generation models to a broader range of scenarios.

**Table 1: Overview of the design choices and capabilities of multimodal autoregressive approaches. Lumina-mGPT is the only model capable of both flexible photorealistic image generation and multimodal task unification, due to its decoder-only transformer design and multimodal generative pretraining.**
| Method | Model | Multimodal | Flexible Photorealistic | Task |
| --- | --- | --- | --- | --- |
| Architecture | Pretraining | Image Generation | Extensibility |  |
| DALL-E (Ramesh et al., ) | Decoder-only | × | × | × |
| Cogview (Ding et al., ) | Decoder-only | × | × | × |
| Parti (Yu et al., ) | Encoder-Decoder | × | × | × |
| LlamaGen (Sun et al., ) | Encoder-Decoder | × | × | × |
| OFA (Wang et al., ) | Encoder-Decoder | ✓ | × | ✓ |
| Unified-IO (Lu et al., ) | Encoder-Decoder | × | × | ✓ |
| Unified-IO 2 (Lu et al., ) | Encoder-Decoder | ✓ | × | ✓ |
| Chameleon (Team, ) | Decoder-only | ✓ | × | × |
| Lumina-mGPT | Decoder-only | ✓ | ✓ | ✓ |

To address the aforementioned challenges, we present Lumina-mGPT, a decoder-only transformer initiated with effective multimodal Generative PreTraining (mGPT) and then supervised-finetuned over flexible, high-quality, high-resolution discrete image tokens in a progressive manner. This framework illuminates flexible high-resolution photorealistic image generation and can be easily extended to solve various component tasks in a unified manner. We provide a detailed comparison of the architecture design choices and model capabilities of existing multimodal autoregressive approaches in Table 1. The key features of Lumina-mGPT are outlined below:

➀ Effective Multimodal Generative Pretraining Unlike commonly adopted approaches that utilize a randomly-initialized causal transformer to generate discrete image tokens in an autoregressive manner, Lumina-mGPT is initialized from an effective multimodal generative pretraining representation. mGPT is a multimodal autoregressive transformer trained on a large-scale image-text interleaved corpus using a simple “next-token prediction” loss. Our key finding is that leveraging the rich knowledge from large-scale multimodal pretraining, mGPT can learn broad and general multimodal representation. This serves as an excellent starting point, remarkably simplifying the task of text-to-image generation, accelerating training convergence on downstream tasks, and unifying various vision-language tasks. Due to the lack of training resources, this mGPT representation is directly adopted from the pretrained Chameleon 7B and 30B models released by Meta (Team, ).

➁ Simple Decoder-Only Architecture As mGPT is pretrained on multimodal tokens, it naturally learns vision and language representation within a single transformer. Consequently, with one decoder-only causal transformer, Lumina-mGPT can handle both text encoding and image token decoding simultaneously. By sharing this decoder-only framework with text-only LLMs, Lumina-mGPT can benefit from well-established theories and infrastructures in the LLM community, such as scaling properties (Chowdhery et al., ; Brown et al., ) and techniques to optimize the training and inference processes (Dao et al., ; Dao, ; Kwon et al., ). We train a family of Lumina-mGPTs with up to 30 billion parameters to provide a flexible trade-off between efficiency and performance and to explore the limit of its multimodal capabilities as model parameters scale.

➂ Flexible High-Resolution Decoding Benefiting from large-scale pretraining on multi-modal interleaved datasets and advanced architectural designs including query-key normalization (QK-Norm) (Henry et al., ; Lu et al., ; Zhuo et al., ; Team, ) and rotary positional embedding (RoPE) (Su et al., ), the mGPTs are ideal initializations for flexible photorealistic image generation. We thus propose Flexible Progressive Supervised Finetuning (FP-SFT) to fully exploit the potential of mGPT for flexible, photorealistic high-resolution text-to-image generation. This approach starts with low-resolution discrete tokens and progressively transitions to high-resolution discrete tokens. Combined with resolution-aware prompts, this weak-to-strong SFT strategy effectively and efficiently enhances the image generation capabilities of Lumina-mGPT.
To equip Lumina-GPT with the ability to understand and generate images at flexible resolution and aspect ratio, we also propose Unambiguous image Representation (Uni-Rep), which eliminates the ambiguity in the original 2D shape of the flattened 1D image tokens by introducing height/width indicators and <end-of-line> tokens.

➃ Omnipotent Task Unification One advantage of AR-based approaches over diffusion-based generative modeling is their promising potential for unified sequence modeling across various modalities and tasks within a single framework (Chen et al., ; Wang et al., ; Lu et al., ; ). In this paper, we explore this potential through preliminary experiments. Specifically, we introduce Omnipotent Supervised Finetuning (Omni-SFT), which treats various tasks–such as multi-turn dialog (Achiam et al., ; Meta, ; Bai et al., ; Bi et al., ), visual multi-turn understanding (Liu et al., ; Chen et al., ), dense labeling (Carion et al., ; Kirillov et al., ; Yang et al., ), flexible image generation (Zhuo et al., ; Ren et al., ; Du et al., ), and spatial-conditional image generation (Zhang et al., ; Ye et al., )–as a unified discrete modeling task. We then perform Omni-SFT over mGPT to seamlessly achieve omnipotent task unification via a natural language interface.

We summarize our contributions as follows: (1) We discover that autoregressive transformers with multimodal generative pertaining, rather than being trained from scratch, can serve as an effective initialization for photoelastic text-to-image generation and the unification of vision and language tasks. (2) Building on mGPTs, we introduce two novel finetuning strategies, FP-SFT and Omni-SFT, to fully unleash their potential in a weak-to-strong paradigm using only 10M high-quality text-image data. (3) The resulting model, Lumina-mGPT, not only demonstrates impressive photorealistic text-to-image generation at any resolution, bridging the gap between autoregressive and diffusion-based methods, but also seamlessly supports various downstream tasks using a language interface.

## 2 Related Work

##### Multimodal Large Language Models

Recent large language models (LLMs) (Chowdhery et al., ; Touvron et al., ; Brown et al., ; Achiam et al., ; Team et al., ) demonstrate strong instruction-following and reasoning capabilities, coupled with extensive world knowledge. To extend LLMs’ expertise from the text domain to multimodal domains such as images and videos, prior works (Liu et al., ; Lin et al., ; Maaz et al., ; Lin et al., ) have aligned pretrained encoders from various modalities with LLMs by curating multimodal instruction tuning datasets. Although these multimodal large language models (MLLMs) have exhibited powerful visual understanding capabilities, their multimodal functionality is primarily limited to perceiving the visual world, exemplified by tasks such as visual question answering and image captioning, rather than generating visual outputs.
Another line of research Koh et al. (); Dong et al. (); Sun et al. (); has proposed augmenting MLLMs with the ability to generate images, videos, and audio from text instructions. These approaches introduce additional visual tokens for generation and align these generative tokens as conditional information with a pretrained generator, such as Stable Diffusion (Esser et al., ; Podell et al., ) for text-to-image generation. Consequently, the generation capabilities heavily rely on the external expert generator rather than MLLMs themselves, resulting in inconsistent and inferior generation results. To combine the strength of both approaches, our model aims to learn both understanding and generation of images using an MLLM with native multimodal capabilities, drawing inspiration from Chameleon (Team, ), a mixed-modal early-fusion foundation model.

##### Text-to-Image Generation

The task of text-to-image generation seeks to synthesize photorealistic and diverse images based on textual descriptions. Nowadays, diffusion models, whether in pixel space (Saharia et al., ) or in latent space (Podell et al., ; Esser et al., ; Pernias et al., ), have become the de-facto approaches due to their superior performance, particularly in producing extremely high-aesthetic images. Among these models, the recent trend of scaling diffusion transformers (DiTs) (Chen et al., ; ; Gao et al., ; Esser et al., ; Li et al., ; Team, ) suggests a unified architecture for both text and image modeling. However, existing DiTs still leverage separate language models, such as CLIP (Radford et al., ) or T5 (Raffel et al., ), as text encoders. This modality gap between text and image representations not only leads to inaccurate generation but also hinders the development of a unified multimodal foundational generative model.
Compared to the dominance of diffusion models, the progress of autoregressive image generation has received less attention in recent years. Early works (Ramesh et al., ; Ding et al., ) proposed a two-stage generation approach: first, training a VQ-VAE Van Den Oord et al. (); Esser et al. () for image tokenization and de-tokenization, and then using an autoregressive transformer to model discrete image token sequences, akin to language modeling. Parti (Yu et al., ) scaled up the autoregressive transformer to 20 billion parameters, demonstrating promising high-fidelity image generation results. LlamaGen (Sun et al., ) further improved the image tokenizer and integrated advanced techniques in LLMs, bridging the performance gap with diffusion counterparts. Unlike Parti and LlamaGen, Lumina-mGPT proposes multimodal generative pertaining on unified text-image sequences, followed by supervised finetuning on high-quality text-to-image pairs, achieving flexible high-aesthetic image generation with autoregressive models.

## 3 Methodolgy

Lumina-mGPT is a decoder-only transformer initialized with multimodal Generative PreTraining (mGPT) and finetuned over high-quality multimodal tokens derived from various tasks. Based on the robust mGPT representation and our proposed supervised finetuning strategies with unambiguous image representation, Lumina-mGPT achieves superior performance in photorealistic image generation and omnipotent task unification with high flexibility in image resolution and aspect ratio. In this section, we first introduce mGPT, followed by the training and inference details of Lumina-mGPT.

### 3.1 Revisiting mGPT with Chameleon

mGPT represents the family of models utilizing a decoder-only transformer architecture, pretrained on extensive multimodal token sequences. These models exhibit exceptional native multimodal understanding, generation, and reasoning capabilities, offering the potential for universal modeling across various modalities and tasks. We use the recent open-source model, Chameleon (Team, ), as an example to illustrate the design choices and implementing details of mGPT.

##### Multimodal Tokenization

To unify text and images into a multimodal token sequence, it is essential first to tokenize both text and image into discrete space. Especially for images, the choice of tokenizer is crucial as it determines the upper limit of generation quality. Specifically, Chameleon trains a new byte pair encoding tokenizer for text. For images, it adopts the quantization-based tokenization method following prior works (Van Den Oord et al., ; Razavi et al., ; Esser et al., ; Yu et al., ; Ramesh et al., ), converting continuous image patches into discrete tokens from a fixed codebook while reducing spatial dimensions. The quantized image tokens are then flattened into a 1D sequence and concatenated with text tokens in various ways to form a multimodal token sequence for unified modeling.

##### Decoder-Only Transformer

Unlike Unified-IO and Parti using an encoder-decoder architecture with pretrained encoders, mGPT trains a decoder-only autoregressive transformer from scratch after transforming text and image inputs into a unified sequence of discrete tokens $x=(x_{1},x_{2},...,x_{T})$, leading to a simpler and more unified approach for multimodal generative modeling. mGPT adopts a standard dense transformer architecture for scalability and generalizability, with minor adaptations like RoPE (Su et al., ) and SwiGLU (Shazeer, ) activation function, following the LLaMA series (Touvron et al., ).

However, this standard transformer architecture exhibits training instability when scaling up the model size and context length of multimodal token sequences, as observed in both Chameleon (Team, ) and Lumina-Next (Zhuo et al., ). It has been found that this instability is caused by the uncontrollable growth of network activations in transformer blocks. Therefore, Pre-Norm, Post-Norm, and QK-Norm are added to each transformer block to preserve the magnitude of intermediate activation and further stabilize the training process.

##### Training Objective

During training, mGPT models the conditional probability $p(x_{t}|x_{1},...,x_{t-1})$ of multimodal sequences using the standard next-token prediction objective.
Additionally, Chameleon applies z-loss (Chowdhery et al., ) to stabilize the training of 7B and 30B models. Initially, we underestimated the importance of z-loss in our attempts because it is absent in most works relevant to the training of (multimodal) LLMs. However, we found that without this term, the magnitude of logits surges, causing the loss to diverge. On the other hand, with z-loss, we observe that for the 7B and 30B models, the optimal temperature for inference-time image generation is much lower than that for the 1B model, as the magnitude of logits is significantly reduced in the larger models.

##### Limitations of Chameleon

Although mGPTs like Chameleon demonstrate potential for joint image and text understanding within one decoder-only transformer, their image generation ability remains inferior to state-of-the-art diffusion-based frameworks (Esser et al., ; Chen et al., ; Li et al., ; Team, ; Zhuo et al., ) in both quality and resolution flexibility. Moreover, it is worth noting that the image generation ability is even absent in the open-source version of Chameleon. Additionally, the capabilities of Chameleon are confined to vision-language and text-only tasks, excluding a broader range of vision-centric tasks. These include classic visual recognition tasks such as segmentation and depth prediction, as well as creative visual generation tasks like controllable generation and image editing. Lumina-mGPT is built upon Chameleon to unlock its full potential for flexible photorealistic image generation and to become a versatile vision generalist.

### 3.2 Lumina-mGPT

Building on pretrained Multimodal Generative Pretraining (mGPT) representation, Lumina-mGPT enables flexible photoelastic image generation as well as vision and language tasks unification. To fully unlock the potential of mGPT, we propose Unambiguous Image Representation (Uni-Rep), Flexible Progressive Supervised Finetuning (FP-SFT) and Omnipotent Supervised Finetuning (Omni-SFT). Uni-Rep eliminates the ambiguity in the original 2D shape of the flattened 1D image tokens, building the basis for image understanding and generation at flexible resolution and aspect ratio. FP-SFT progressively finetunes mGPT to generate higher-resolution discrete image tokens in a simple-to-hard manner, while incorporating multi-task finetuning to prevent the catastrophic forgetting of text knowledge. After the FP-SFT process, Lumina-mGPT becomes capable of generating photorealistic images at flexible resolutions. Building on the image generation ability from the FP-SFT stage, we continue to finetune Lumina-mGPT over omnipotent task datasets by tokenizing images and annotations from dense labeling, spatial-conditional image generation, and multi-turn editing datasets into discrete tokens. All finetuning stages adopt the next-token prediction objective to jointly model multimodal tokens and integrate z-loss, similar to the pretraining stage.

#### 3.2.1 Effective Initlization

Large-scale pre-training and scalable model architecture have been widely verified as the golden path to advanced intelligence. As mGPTs, like Chameleon (Team, ), are pretrained on large-scale image-text interleaved datasets and have developed effective and generalizable representations for both image and text, they can better serve as the starting pointing for flexible photorealistic image generation and beyond than random initialization or language-only models. Furthermore, the LLaMA (Touvron et al., ) architecture, which incorporates features like query-key normalization (Henry et al., ) and rotary position encoding (Su et al., ), has demonstrated strength and scalability through extensive validation (Touvron et al., ; Meta, ; Gao et al., ). By initializing from the Chameleon mGPT, which adheres to the LLaMA architecture, we can leverage these architectural advantages. Consequently, initializing from mGPTs allowed us to efficiently train high-performing Lumina-mGPT models, with parameters ranging from 7B to 30B, using just 10M high-quality image-text data points.

We emphasize that pre-training a decoder-only multimodal Generative PreTraining (mGPT) model has been extensively explored and applied across various applications (Team, ; Wang et al., ; Lu et al., ). Unlike these existing approaches, which aim to unify different modalities and tasks, the primary contribution of Lumina-mGPT lies in demonstrating the benefits of starting with an appropriate mGPT representation rather than using large language models (LLMs) or random initialization, especially in text-to-image generation. We believe this finding can inspire future advancements in image generation and vision generalist models.

Figure: Figure 2: Illustration of Resolution-Aware Prompt (bottom left) and Unambiguous Image Representation (bottom right). These designs are used in all supervised finetuning stages to eliminate the ambiguity in image representation, enabling flexible resolution image modeling.
Refer to caption: x2.png

#### 3.2.2 Supervised Finetuning for Lumina-mGPT

##### Unambiguous Image Representation

Existing methods, such as Chameleon (Team, ) and LlamaGen (Sun et al., ), represent images as 1D flattened sequences of 2D discrete image codes. While this approach is adequate for fixed image resolution, it becomes ambiguous when the image resolution is supported to be variable, as with Lumina-mGPT. For instance, images with resolutions of 512 $\times$ 512, 256 $\times$ 1024, and 1024 $\times$ 256 can all be encoded into the same number of tokens, making it impossible to infer the original shape without examining the token contents. This ambiguity poses significant challenges for both image perception and generation.

To address this problem, we propose Unambiguous image Representation (Uni-Rep), which augments the image representations by adding extra height/width indicator tokens immediately after the <start-of-image> token and inserting <end-of-line> tokens after image tokens belonging to the same row. As shown in Figure 2, this modification ensures that the original shape of the images can be accurately parsed from the 1D representation without additional context or delving into the contents of the image tokens. This enhancement provides the foundation for Lumina-mGPT’s ability to perform image-related tasks at any resolution and aspect ratio.

Note that while either the height/width indicators or the <end-of-line> tokens can independently achieve disambiguation, we still use both simultaneously because they have distinct benefits. When generating images, the height/width indicators, generated before any image tokens, pre-determine the shape of the image, aiding Lumina-mGPT in composing the image contents. On the other hand, the <end-of-line> tokens can serve as anchors, offering the 1D tokens sequence with additional explicit spatial information.

##### Flexible Progressive Supervised Finetuning (FP-SFT)

The FP-SFT process equips the pretrained mGPT with the capability to generate high-resolution images with flexible aspect ratios in a progressive manner. The process is divided into three stages, where
the product of width and height approximates $512^{2}$, $768^{2}$, and $1024^{2}$, respectively. In each stage, a set of candidate resolutions with similar areas but different height-width ratios are prepared, and each image is matched to the most suitable resolution. In the low-resolution stage, shorter sequence lengths and the resulting high training throughput allow the model to quickly traverse a large amount of data, learning the general composition of images and a broad spectrum of visual concepts. Conversely, in the high-resolution stage, the model is expected to focus on learning high-frequency fine-grained details unique to high-resolution images. Benefiting from the strong foundation built during the high-throughput pretraining and low-resolution finetuning stages, the low-throughput high-resolution finetuning stage is data-efficient, thereby enhancing the overall efficiency of the FP-SFT process.

A meticulously curated dataset of high-resolution photorealistic image-text pairs is used for FP-SFT. Moreover, the pure-text data from OpenHermess (Teknium, ) and the image-to-text data from Mini-Gemini (Li et al., ) are also incorporated during training to prevent catastrophic forgetting. To provide users with a natural way to specify the desired resolution of generated images, we developed the resolution-aware prompt (Figure 2). For each image and its corresponding description, the prompt is structured as follows:

Generate an image of {width}x{height} according to the following prompt:
\n
{description}

##### Omnipotent Supervised Finetuning (Omni-SFT)

While flexible photorealistic image generation is the primary target of Lumina-mGPT, we find that the resulting model after FP-SFT can be efficiently transferred to a wide spectrum of image understanding and generation tasks. We thus present Omni-SFT, a preliminary exploration toward boosting Lumina-mGPT to a visual generalist. Training data for Omni-SFT consists of the following:

1. Single- and multi-turn language-guided image-editing from MagicBrush (Zhang et al., ) and SEED (Ge et al., ) (only involving the real-world and multi-turn subsets).

2. Dense prediction tasks, including surface norm estimation from NYUv2 Silberman et al. () and ScanNet Dai et al. (), depth estimation from Kitti v2 Cabon et al. () and Sintel Butler et al. (), pose estimation from MSCOCO Lin et al. (), semantic segmentation data annotated with OneFormer (Jain et al., ) on image from Laion (Schuhmann et al., ), and grounding data from RefCOCO (Kazemzadeh et al., ).

3. In-house spatial-conditional image generation data, with conditions including surface norm, depth, pose, and segmentation.

4. A small fraction of data sampled from those used in the previous FP-SFT process.

As demonstrated in Section 4.2, after Omni-SFT, Lumina-mGPT exhibits a general capability for completing a wide range of tasks other than text-to-image generation, indicating the potential for building a multimodal generalist along this direction.

##### Training Setup

Though multiple tasks are involved in the SFT process, a unified next-token-prediction loss is used for all of the tasks. As Lumina-mGPT is designed as a chat model, all data are organized into single or multi-turn dialogs, with the loss applied only to the response parts.
For all experiments, the AdamW (Loshchilov & Hutter, ) optimizer with weight decay = $0.1$ and betas = $(0.9,0.95)$ is used, and the learning rate is set to 2e-5. For stabilizing training, z-loss is applied with weight 1e-5 for both the 7B and the 30B model, and for the 7B model dropout with probability 0.05 is additionally applied. To accommodate the large model volume, PyTorch FSDP (Zhao et al., ) is employed with gradient checkpointing. To increase training throughput, all data are pre-tokenized before training and are clustered according to the number of tokens, ensuring that each global batch is composed of data with similar lengths.

#### 3.2.3 Inference for Lumina-mGPT

##### Classfier-Free Guidance

Classifier-Free Guidance (CFG) (Ho & Salimans, ; Sanchez et al., ) is originally proposed to enhance the quality and text alignment of generated samples in text-to-image diffusion models. We incorporate this technique into autoregressive models during inference. When generating an image token, the CFG-processed logits $l_{\text{cfg}}$ are formulated as $l_{\text{cfg}}=l+s(l-l^{\prime})$, where $l$ represents the original logits conditioned on the complete context; $l^{\prime}$ represents the context-independent logits, which are conditioned solely on the tokens following the <start-of-image> token of the currently generating image, and are independent of any prior context; $s$ denotes the guidance scale of Classifier-Free Guidance. To make CFG work, during training, the context before <start-of-image> is randomly dropped by a probability of 10%. In practice, KV cache can be used for accelerating the computation of both $l$ and $l^{\prime}$. As shown in Section4.4, CFG has a significant impact on the generation performance of Lumina-mGPT.

##### Different Decoding Hyperparameters for Image and Text

During inference, the sampling strategy of autoregressive models involves numerous hyperparameters that significantly influence the sampling results. We find that the optimal decoding hyperparameters differ greatly between text decoding and discrete image code decoding. For example, the top-k=5 setting performs well in generating text. However, when generating images, the value of top-k should be much larger (e.g. 2000) to avoid repetitive and meaningless patterns. Therefore, we implement a status-aware control mechanism for inference. Specifically, a set of default hyperparameters is used for text decoding; once a <start-of-image> token is generated, the hyperparameters switch to those optimized for image generation. After the <end-of-image> token is generated, the parameters revert to the initial settings.

Figure: Figure 3: Qualitative comparison with LlamaGen and Parti. Lumina-mGPT can generate more photorealistic images with high aesthetics and fine-grained details.
Refer to caption: x3.png

## 4 Experiments

### 4.1 Fundamental Photorealistic Text-to-Image Generation

We first demonstrate the fundamental text-to-image generation capabilities of Lumina-mGPT with FP-SFT. As shown in Figure 1, Lumina-mGPT can generate photorealistic images in a variety of resolutions, achieving the first native 1K autoregressive generation without model cascaden (Yu et al., ; Saharia et al., ; Chang et al., ; Pernias et al., ), a common technique in text-to-mage generation. These generated images exhibit strong semantic coherence with intricate visual details, despite being finetuned on limited computational resources and text-image pairs.

##### Compaison with SoTA AR-based Approaches

We compare the text-to-image synthesis ability of Lumina-mGPT with LlamaGen (Sun et al., ) and Parti (Yu et al., ). LlamaGen beats state-of-the-art diffusion models on ImageNet FID score. Compared with LlamaGen, Lumina-mGPT can achieve better visual quality as shown in Figure 3 in text-to-image generation. Note that Lumina-mGPT only requires 10M image-text pairs while LlamaGen is trained over 50M low-quality image-text pairs accompanied by 10M in-house aesthetic image-text pairs. Compared to Parti, an AR text-to-image model with 20 billion parameters, Lumina-mGPT also demonstrates better visual quality and aesthetics. However, due to significant differences in computational costs and training datasets, Lumina-mGPT demonstrates inferior text instruction following ability compared to Parti. In addition, neither LlamaGen nor Parti supports the end-to-end generation of 1K resolution images with arbitrary aspect ratios, as achieved by Lumina-mGPT. LlamaGen only supports a fixed resolution of 512 $\times$ 512, while Parti generates 1024 $\times$ 1024 images using an additional super-resolution upsampler.

##### On the Effectiveness of FP-SFT

To further validate the effectiveness of FP-SFT, we visualize the images generated at different finetuning stages in Figure 4. With increasing image resolution, we observe a progressive decrease in visual artifacts introduced by VQ-VAE and the emergence of diverse fine-grained visual details. From these illustrations, we can conclude that our FP-SFT can unleash the potential of generating high-quality images from mGPT in a progressive manner.

Figure: Figure 4: Samples with zoom-ins generated by Lumina-mGPT in different resolution finetuning stages. The visual details continuously improve along with the progressively increasing resolution.
Refer to caption: x4.png

### 4.2 Omnipotent Task Unification with Lumina-mGPT

By performing Omni-SFT over FP-SFT, Lumina-mGPT can perform a multitude of capabilities, which can be categorized into text-only multi-turn dialog, visual multi-turn dialog, multi-turn image editing, dense labeling, and spatial-conditional image synthesis. To intuitively illustrate these capabilities, we qualitatively visualize how various types of downstream tasks can be seamlessly integrated into Lumina-mGPT in Figure 12 to 16.

First, Lumina-mGPT supports general text-only tasks in LLMs, such as solving math problems, coding, and commonsense reasoning, thanks to the extensive pertaining in Chameleon and our multi-task finetuning to mitigate catastrophe forgetting. For instance, Lumina-mGPT correctly answers ‘‘Which is bigger? 9.9 or 9.11’’, as shown in Figure 12, a question that has confused almost all existing LLMs including GPT-4 (Achiam et al., ) and Gemini (Team et al., ).

As demonstrated in Figure 13, Lumina-mGPT is also capable of handling various vision-language tasks including image captioning, visual question answering, and general multi-turn multi-image dialog.
As a visual generalist, it is important to incorporate classic visual recognition tasks. Using natural language as a unified interface, Lumina-mGPT can perform multiple computer vision tasks including image segmentation, pose estimation, depth estimation, surface normal estimation, and referring object detection. See Figure 14 and 15 for examples.

Beyond photorealistic image generation, it is also vital to add additional control to the text-to-image generation model. Demonstrated in Figure 15 and 16, Lumina-mGPT supports versatile spatial control, including depth maps, segmentation maps, normal maps, and human poses, to guide the generation of target images. Given the above examples, though preliminary, they showcase that Lumina-mGPT can effectively follow diverse instructions, indicating its promising potential as a unification of various challenging tasks in one framework.

### 4.3 Comparison with Diffusion-based Appraoches

For a long period of time, diffusion models have dominated the field of text-to-image generation compared to autoregressive models. Although LlamaGen claims to beat diffusion models, their results are limited to the ImageNet benchmark and there has been no direct comparison between these two architectures so far. In this section, we aim to provide a detailed comparison of autoregressive and diffusion-based methods trained on these same text-image datasets, focusing on image quality, diversity, text-rendering, and multilingual capabilities. Specifically, we adopt Lumina-mGPT and Lumina-Next-SFT (Zhuo et al., ) as representatives of autoregressive and diffusion-based methods, respectively. A direct visual comparison between Lumina-Next-SFT and Lumina-mGPT reveals both the similarities and differences between autoregressive and diffusion-based generative modeling approaches.

Figure: Figure 5: Comparison between Lumina-Next-SFT and Lumina-mGPT using different random seeds. Images generated by Lumina-mGPT exhibit comparable aesthetics with greater diversity.
Refer to caption: x5.png

Figure: Figure 6: Text rendering comparison between Lumina-Next-SFT and Lumina-mGPT. From left to right, the correct texts to be rendered on the image are: “Bloom”, “Hope”, “Brew”, and “Joy”.
Refer to caption: x6.png

Figure: Figure 7: Emoji and multilingual instruction understanding ability comparison between Lumina-Next-SFT and Lumina-mGPT. Lumina-mGPT struggle to understand emojis and multilingual prompts.
Refer to caption: x7.png

On the Similarity between Diffusion- and AR-based Generation
Given the same set of text prompts, both diffusion- and AR-based approaches generate photorealistic images with similar aesthetic style and fine-grained details, illustrated in Figure 5. This reveals the fact that both architectures can achieve satisfactory text-to-image generation performance when provided with the same training data, training budget, and comparable model sizes. The AR-based methods display remarkable visual aesthetics on par with their diffusion counterparts, challenging the notion that diffusion models are more effective and promising architecture in generative modeling. This finding also aligns with the platonic representation hypothesis (Huh et al., ) that neural networks are converging to learn a shared representation space despite being trained with different architectures and objectives. Therefore, this hypothesis highlights the importance of collecting more high-quality data and optimizing training infrastructure as directions for data and model scaling, to boost the overall model performance that is agnostic to any specific architecture.

On the Differences between Diffusion- and AR-based Generation
Despite the visual similarities, there are notable differences between diffusion- and AR-based methods. As shown in Figure 5, Lumina-mGPT exhibits more diversity using different random seeds, while Lumina-Next-SFT generates similar images with identical layouts and textures. This can be partly attributed to the use of high temperature and top-k values in Lumina-mGPT. However, excessive diversity also causes our model to be less stable and more prone to producing visual artifacts, which is discussed in Section 4.7.

We also compare the text rendering and multilingual understanding capabilities between Lumina-mGPT and Lumina-Next-SFT. As illustrated in Figure 6, Lumina-mGPT exhibits significantly better text synthesizing results, while Lumina-Next-SFT struggles to generate any complete character. We argue that this underscores the importance of mGPT, where the model learns a seamless multimodal representation between text and images using massive interleaved data during the pertaining stage. However, when it comes to multilingual understanding, Lumina-mGPT performs worse than Lumina-Next-SFT in terms of emoji and Chinese prompts shown in Figure 7. The reason is that although Lumina-mGPT learns better text-image alignment, the lack of multilingual text corpus used in pertaining limits its performance. In contrast, the text encoder used in Lumina-Next-SFT showcases significantly stronger multilingual capabilities than Chameleon. Hence, we hope that by comprehensively enhancing the capabilities of the base mGPT model, such as adding more multilingual data, Lumina-mGPT can benefit in all downstream tasks.

In addition to text-to-image generation, Lumina-mGPT supports various vision and language tasks within a unified framework. However, the design of diffusion models limits their compatibility and performance across multiple modalities and tasks. They often require specific architecture designs and additional training for each unseen task (Ke et al., ; Xu et al., ). In contrast, Lumina-mGPT treats input from all modalities as multimodal token sequences and leverages natural language as the interface to unify diverse tasks with next-token prediction.

### 4.4 Decoding Configuration Matters

Figure: Figure 8: Samples generated by Lumina-mGPT using different decoding configs. Increasing temperature, top-k, and guidance scale all lead to improved visual details.
Refer to caption: x8.png

In autoregressive models, various configuration parameters during Lumina-mGPT’s decoding stage significantly affect sample quality (Holtzman et al., ; Radford et al., ; ). Hyperparameters such as temperature (T), top-k, and classifier-free guidance scale (CFG) have not been extensively investigated in the visual domain. In this section, we explore how these hyperparameters influence the generated image in terms of quality, texture, and style.

Influence of Temperature
To evaluate the effect of these decoding parameters, we first set a standard decoding configuration: T=1.0, Top-k=2000, CFG=4.0, which serves as a good-to-use setting. From this baseline, we gradually shift T from 0.7 to 1.0 to generate corresponding images at different temperatures. The results are shown in Figure8. It is evident that when setting the temperature low, visual details diminish and objects tend to be over-smoothed. Conversely, when setting the temperature high, the generated images contain rich visual content but are prone to contain more artifacts.

Influence of Top-k
Based on the standard decoding setting, we vary the top-k value, from 50 to 8192, where 8192 is equal to the size of the VQ-VAE codebook used usedin Lumina-mGPT. The results, visualized in Figure8, indicate a similar trend with increasing temperature. When top-k is low, the image content and texture are relatively simple, exhibiting the over-smoothed problem as well. When top-k is set high, the image detail and texture are diverse, making it more aesthetically appealing, while increasing the potential of artifacts.

Influence of Classifier-Free Guidance
CFG is extremely important in text-to-image diffusion models (Ho & Salimans, ; Sanchez et al., ). To verify its effectiveness in autoregressive generation, we change the CFG value from 1.0 to 8.0. As shown in Figure8, with the CFG increasing, the quality of generated images improves, proving the effectiveness of the classifier-free guidance in this context.

### 4.5 Lumina-mGPT as VQ Codes Refiner

Figure: Figure 9: Reconstruction quality comparison between different methods. Lumina-mGPT can be instructed to refine the encoded VQ codes, leading to better reconstructed images.
Refer to caption: x9.png

It is well-known that both VQ-VAE (Van Den Oord et al., ; Razavi et al., ; Esser et al., ) and VAE (Kingma & Welling, ) compress images into latent representations at the cost of information loss, which introduces quality degradation at reconstruction, especially for high-frequency details such as edges, hair, and text. This evitable loss upper bounds the quality of image generation for all latent space generative models (Esser et al., ; Yu et al., ; Sun et al., ). However, we surprisingly found that Lumina-mGPT can improve the discrete image tokens encoded by VQ-VAE using a simple ‘‘No edit. 
’’ instruction, then Lumina-mGPT can generate a sequence of refined image tokens that can be decoded into a better image.

We compare the reconstruction quality of VQ-VAE used in Chameleon, Chameleon’s VQ-VAE using VQ codes refined by Lumina-mGPT, and the SDXL VAE (Podell et al., ) in Figure 9. The SDXL VAE showcases significantly better reconstruction performance compared to Chameleon’s VQ-VAE. This advantage also explains why diffusion-based methods can beat autoregressive models in text-to-image generation. However, after applying the aforementioned simple trick to refine the VQ codes using Lumina-mGPT, the quality of reconstructed images exhibits remarkable gains in both fine-grained visual details and text rendering, on par with the SDXL VAE. We hypothesize that Lumina-mGPT learns the underlying distribution of real image tokens during large-scale pertaining and high-quality FP-SFT. The Omni-SFT further triggers the emergence of such zero-shot capability after training on various tasks and instructions. We believe this zero-shot capability to bridge the representation gap between the encoder and decoder in VQ-VAE suggests promising directions to improve the current paradigm of two-stage latent generative modeling. For example, we can leverage Lumina-mGPT as a teacher to refine the VQ codes for student training or to design inference-time techniques by manipulating the token logits similarly to classifier-free guidance.

### 4.6 Attention Visualization

To better understand the sampling behavior of Lumina-mGPT, we visualize the average attention logits of the last image token during text-to-image generation, as shown in Figure 10. The results reveal that the attention score decreases for distant tokens, indicatin that the model pays more attention to local tokens compared to distant ones. This behavior aligns with the long-term decay property designed in RoPE. Besides, we observe a similar pattern known as “attention sinks” in LLMs (Xiao et al., ), where a large proportion of attention score is allocated to a small number of tokens. After looking into details, these sink tokens are identified to be the indicators, including the <start-of-image> and <end-of-line> tokens. Notably, the attention scores of text tokens are significantly smaller than those of the <start-of-image> token. This suggests that most of the semantic information in text tokens may have been encapsulated into the <start-of-image> token. These intriguing findings indicate that Lumina-mGPT aggregate more information from these sink tokens, demonstrating the effectiveness of these indicators in our proposed Uni-Rep and opening up the potential for accelerating the sampling process with these sink tokens.

Figure: Figure 10: Visualization of averaged attention logits from the last image token. We observe “attention sinks” (Xiao et al., ), where added indicator tokens allocate a large proportion of attention score.
Refer to caption: x10.png

Figure: Figure 11: Failure cases of current Lumina-mGPT. Due to inadequate training and limited data size, Lumina-mGPT sometimes struggles to understand input conditions and produce visual artifacts.
Refer to caption: x11.png

### 4.7 Limitations

##### Failure Cases

Despite Lumina-mGPT can generate photorealistic images, it sometimes produces images with noticeable visual artifacts. For example, Lumina-mGPT may generate people and animals with unreasonable limbs, as shown in the first row of Figure 11. Besides, compared to SoTA text-to-image
generation approaches including SD3 (Esser et al., ), Kolors (Team, ), and HunyuanDiT (Li et al., ), all pretrained over 1B image-text pairs, Lumina-mGPT’s prompt-following ability is inferior due to the limited training resources and data size, which are many times smaller than these SoTA methods.
Regarding dense labeling and controllable generation, Lumina-mGPT currently showcases preliminary results with a limited training budget. Hence, the second row in Figure 11 provides such an example where Lumina-mGPT produces inaccurate predictions or semantically inconsistent images, failing to understand the given image conditions.
Therefore, we expect by scaling data size with more computational resources, Lumina-mGPT can effectively address the above failure cases such as inadequate instruction-following ability and visual artifacts.

##### Generation Speed

Autoregressive models require numerous network evaluations during inference due to the nature of next-token prediction, similar to the iterative denoising process in diffusion models. This becomes worse when generating high-resolution images, which often require minutes to generate a full sequence of image tokens, significantly slower than current diffusion models with advanced samplers. However, there have been plenty of techniques to optimize the inference speed designed for autoregressive models, such as vLLM (Kwon et al., ) and FlashAttention (Dao et al., ; Dao, ). We believe that by integrating these approaches in the future, Lumina-mGPT can achieve a remarkable speed up during inference.

##### VQ-VAE Reconstruction Quality

VQ-VAE is employed as the image tokenizer to convert continuous images into discrete token representations. Meanwhile, it also introduces information bottlenecks by compressing the spatial dimensions of images. As a result, the reconstruction quality of VQ-VAE largely determines the upper limit of generation quality. We discover that the VQ-VAE proposed in Chameleon sometimes struggles to reconstruct high-frequency details, especially when text and human faces are present in images. Incorporating further improvements on VQ-VAE, such as FSQ (Mentzer et al., ), may also enhance the generation quality of Lumina-mGPT.

## 5 Concolusion

In this work, we introduce Lumina-mGPT, a decoder-only transformer that can produce diverse, photorealistic images at any resolution from text prompts. Instead of random initialization, Lumina-mGPT features initializing from an autoregressive transformer with multimodal Generative PreTraining (mGPT). Leveraging the general multimodal representation learned from massive interleaved data, we design two finetuning strategies named FP-SFT and Omni-SFT to unleash the potential of mGPT on text-to-image generation and omnipotent task unification, respectively. We demonstrate Lumina-mGPT’s broad multimodal capabilities across a wide range of tasks, highlighting its high-quality text-to-image generation in side-by-side comparison with diffusion models.

Figure: Figure 12: Lumina-mGPT as a generalist for various multimodal tasks.
Refer to caption: x12.png

Figure: Figure 13: Lumina-mGPT as a generalist for various multimodal tasks.
Refer to caption: x13.png

Figure: Figure 14: Lumina-mGPT as a generalist for various multimodal tasks.
Refer to caption: x14.png

Figure: Figure 15: Lumina-mGPT as a generalist for various multimodal tasks.
Refer to caption: x15.png

Figure: Figure 16: Lumina-mGPT as a generalist for various multimodal tasks.
Refer to caption: x16.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: The comparisons and evaluations are insufficient for the work. Specifically, a few questions are still to be addressed:
* How does Lumina-mGPT compare with diffusion-based methods in terms of image generation?
* How does Lumina-mGPT perform on text-only tasks, comparing with other text-only LLMs?
soundness: 2
strengths: * The proposed Unambiguous image Representation, when used with Flexible Progressive Supervised Finetuning (FP-SFT), allows the method to generate images of varying resolutions.
* While prior method Chameleon can only perform vision-language and text-only tasks, Lumina-mGPT achieves visual recognition tasks (e.g., segmentation, depth prediction) and controllable generation as well as image editing, which makes Lumina-mGPT a unified model for various downstream applications.
* The performance on VQA benchmarks significantly improves over baseline Chameleon, demonstrating the effectiveness of the proposed model.
summary: This work presents Lumina-mGPT, a series of autoregressive models for multimodal understanding and generation. With a pre-trained decoder-only transformer, Lumina-mGPT allows for a unified framework for multi-model modeling. The work shows that multimodal generative pretraining is the key towards general multimodal capabilities. Furthermore, flexible progressive supervised fine-tuning, as proposed by the work, allows high-aesthetic image generation at flexible resolutions. Finally, omnipotent supervised fine-tuning enables Lumina-mGPT to achieve task unification in visual generation and understanding.
weaknesses: * The contributions of this work is not clearly described. While the work claims mGPT to be a key insight, the importance of mGPT is not discovered by the work, as the mGPT model is adapted from Chameleon. The difference is that Lumina-mGPT performs fine-tuning on Chameleon.
* The proposed Unambiguous image Representation (Uni-Rep) is only applied at the supervised fine-tuning stage. This creates gaps between the image representation in pre-training and fine-tuning. 
* The method claims that baseline Chameleon shows degraded visual quality compared to diffusion methods (L242-244). However, this work does not perform quantitative evaluations on the quality of the generated images to illustrate whether the method is able to overcome this limitation. Metrics such as FID and/or results on benchmarks such as T2I-CompBench [1] are needed to show the performance improvements against other related works.
* No comparisons are provided with text-only LLMs on text-only benchmarks such as MMLU. The example dialogues are insufficient in evaluating the model's capabilities in text-only tasks.
* No human evaluation is performed to assess the quality of multi-modal generation.

[1] T2I-CompBench: A Comprehensive Benchmark for Open-world Compositional Text-to-image Generation. K. Huang, et al. https://arxiv.org/abs/2307.06350

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: - In the appendix, you observe that *"with the CFG increasing, the quality of generated images improves, proving the effectiveness of the classifier-free guidance in this context."*. This finding appears to contrast with typical outcomes in diffusion models, where increasing the CFG beyond a certain threshold often results in diminished image quality. Could you elucidate the reasons behind this differing impact of CFG in your model compared to traditional diffusion models?
soundness: 3
strengths: + Omni-SFT within Lumina-mGPT effectively unifies a diverse array of tasks within a single model framework, demonstrating its extensive multitasking capabilities across various complex applications such as text-to-image synthesis, image captioning, image editing, spatial-conditional image generation, and more.
+ The use of mGPT for initial training, followed by sophisticated fine-tuning strategies like FP-SFT and Omni-SFT, establishes a solid foundation for generating high-quality, photorealistic images while adeptly handling a wide range of multimodal tasks.
+ Lumina-mGPT exhibits exceptional flexibility in generating images across different resolutions and aspect ratios—a significant advantage over many existing models.
+ By effectively bridging the gap between autoregressive and diffusion methods, Lumina-mGPT achieves remarkable visual aesthetics and detailed image rendering without the need for cascading models,
+ The paper ingeniously integrates various advanced concepts from the literature and existing models to enhance the training and inference capabilities of Lumina-mGPT. This includes adopting classifier guidance from diffusion models and stabilization techniques used in large language models, which collectively contribute to the robustness and efficiency of the framework.
summary: This paper introduces Lumina-mGPT, a multimodal autoregressive model that excels in generating photorealistic images from textual descriptions, as well as performing a variety of other vision and language tasks. Built on a pre-trained decoder-only transformer architecture, the model adeptly processes multimodal token sequences. Key innovations of Lumina-mGPT include Effective Multimodal Generative Pretraining (mGPT) using extensive interleaved text-image datasets, Flexible Progressive Supervised Fine-tuning (FP-SFT), and Omnipotent Supervised Finetuning (Omni-SFT). These developments enable the model to produce images of varying resolutions and accommodate a broad spectrum of vision-language tasks, marking a notable advancement in flexibility and task integration over traditional autoregressive approaches.
weaknesses: - The paper falls short in providing comprehensive comparative metrics, such as FID scores, with only limited comparisons featured in Table 2 against the Chameleon model. A more extensive range of benchmarking against current state-of-the-art (SoTA) models is essential to objectively evaluate the model’s performance and its advancements over existing methodologies. Including a broader set of metrics would significantly clarify the model's positioning and contribution within the broader field.
- Although the paper showcases zero-shot capabilities in enhancing visual details, it only offers qualitative results. Incorporating quantitative evaluations would provide a more robust comparison against other models and substantiate the claimed improvements.
- While Omni-SFT demonstrates the model's capability to handle diverse tasks, the absence of quantitative results in a controlled evaluation test limits the understanding of its performance, especially in comparison to specialized models. Providing such data would help gauge the effectiveness of Omni-SFT and its relative performance across different tasks.
- The paper highlights the performance benefits of Omni-SFT fine-tuning on top of the Lumina model but lacks a comparative analysis with scenarios where Omni-SFT is fine-tuned starting from the initial mGPT. A direct comparison would help validate the advantage of transitioning from Lumina to Omni for task unification, ensuring that this methodological choice yields tangible benefits.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: The Large World Model (LWM) paper introduces a multimodal autoregressive model capable of processing extensive sequences of video and book data, with a context length of millions of tokens. This enables it to perform tasks such as language understanding, image analysis, and video generation. In contrast, the Lumina-mGPT model is focused specifically on photorealistic text-to-image generation. Is this understanding accurate? Based on reading both papers, LWM appears to use a less effective VQ-VAE compared to Lumina-mGPT's VQ tokenizers; does this difference explain the variation in visual quality? Could the authors provide an analysis to identify the areas contributing the most to these performance gains?

While Lumina-mGPT demonstrates impressive performance across various tasks, additional examples in less conventional vision-language settings would help illustrate its adaptability and generalization. Could the authors provide results or comparisons on more challenging benchmarks or diverse vision-language tasks (e.g., object counting, spatial reasoning) that may better highlight Lumina-mGPT's strengths and weaknesses across the multimodal spectrum?
soundness: 3
strengths: The paper presents a systematic approach to fine-tuning through Flexible Progressive Supervised Fine-Tuning (FP-SFT) and Omnipotent Supervised Fine-Tuning (Omni-SFT). These methods are well-supported by empirical results, demonstrating that Lumina-mGPT generates high-quality, high-resolution images across various aspect ratios. Additionally, an extensive evaluation of zero-shot performance and attention visualizations offers insights into the model's generalizability and internal mechanisms.

The paper is well-organized and clearly articulates its goals, methodology, and findings. The authors effectively communicate complex technical concepts, ensuring that each design choice—such as the unambiguous image representation—is both motivated and illustrated, enhancing readability. While certain sections, such as the detailed explanations of supervised fine-tuning processes, may require technical background to fully appreciate, the overall clarity remains high.

Lumina-mGPT's primary contribution lies in bridging the gap between autoregressive (AR) models and diffusion-based image quality, with its open-source release further amplifying its impact.
summary: The paper "Lumina-mGPT" introduces a multimodal autoregressive model designed for flexible, photorealistic image generation from text, leveraging a decoder-only transformer and multimodal Generative PreTraining (mGPT). Lumina-mGPT unifies text and image processing within a single framework, using Flexible Progressive Supervised Finetuning (FP-SFT) for high-resolution, flexible-aspect image synthesis, and Omnipotent Supervised Finetuning (Omni-SFT) for a wide range of vision-language tasks, including segmentation and multiview generation. The model’s novel unambiguous image representation enhances its ability to generate variable-resolution images, and extensive multimodal pretraining enables impressive zero-shot capabilities across visual tasks.
weaknesses: The introduction and abstract emphasize the decoder-only architecture for text and image as a novel contribution, which is strange, given that several prior works—some even cited in this paper—have already employed similar architectures.

The effectiveness of Flexible Progressive Supervised Finetuning (FP-SFT) would benefit from more detailed analysis, especially around why certain parameter settings or resolution stages were chosen. Currently, the progressive finetuning approach appears somewhat arbitrary, and an ablation study exploring different finetuning configurations or showing specific benefits per stage would substantiate these choices.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 2
questions: What exactly is different about Lumina-mGPT from other large VLMs? Is the training different? Is there some novelty in how the training is done?
soundness: 3
strengths: The paper finetunes the Chamaleon model to add image generation and other multimodal tasks to the model. The original Chameleon model can also do this but the image generation model was not publicly released.
The paper also proposes a simple representation of the images to support various aspect ratios by adding information about height and width, as well as end-of-line tokens after each row of (latent) pixels.
The finetuning on multimodal tasks shows that the training approach also generalizes to tasks besides image generation.
summary: The paper proposes to finetune an autoregressive decoder-only model (initialized from Meta's Chameleon model) for image generation based on multimodal inputs. For this the paper initializes their model from the Chameleon checkpoint and finetunes the model for image generation and also adds additional multimodal tasks (e.g.,image editing, dense predictions like segmentation maps, spatial conditions, VQA, etc) to the training. The resulting model can generate photorealistic images in various aspect ratios and can also perform the multimodal tasks it was finetuned on.
weaknesses: There is limited novelty in the paper. Most of the work seems to come from the original Chameleon checkpoint with the image training being a relatively straight-forward finetuning approach.

The finetuning with multimodal tasks is also a relatively straight forward extension and other works have already shown that AR vision models can handle many different tasks and also can perform in-context learning (e.g., Sequential Modeling Enables Scalable Learning for
Large Vision Models, CVPR 2024).

Overall it's not clear to me what exactly is novel and different from Chameleon or other large VLMs.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see the questions from my review comments above. Overall, my preliminary rating is based on the pros and cons of the current work.
soundness: 2
strengths: - The proposed model achieves very good image generation results on the T2I task in the AR paradigm, as stated as one of the main contributions.
- The paper presents a rather detailed discussion of the implementations and fine-tuning recipes, including failure cases, which is appreciated.
- This work is open-source, with the promise to release the model checkpoints and codes, thus, it may be of interest to the researchers in the open-source community on large models.
summary: This paper introduces a large multimodal autoregressive model, Lumina-mGPT, with a particular focus on its text-to-image (T2I) generation capability after fine-tuning. The model leverages pre-trained token representations from Chameleon 7B and 30B (Meta) and incorporates two fine-tuning techniques—Flexible Progressive Supervised Fine-tuning (FP-SFT) and Omnipotent Supervised Fine-tuning (Omni-SFT)—to adapt the model for T2I tasks and multiple vision and language tasks.

Overall, I think this paper has both clear strengths and limitations, which I will detail in my comments below.
weaknesses: - While this may sound like a platitude, I must note that there is limited novelty in this work. As the authors acknowledge, a well-trained token representation for multimodal data is crucial in this setting, motivating their choice of Chameleon. The proposed fine-tuning techniques, however, follow commonly used methods and widely accepted design choices. Overall, the work reads very much like a technical report to me. However, I recognize that both intellectually novel approaches and engineering-heavy work have their place in the community. That being said, I am fine with either as long as the design choices are well-justified. 
- A more concerning issue is the claim of the “emergent zero-shot capabilities.” While zero-shot is indeed becoming something of a buzzword, I don’t believe the observed reconstruction ability within the VQ-VAE module qualifies as true “zero-shot.” This term should be defined and examined rigorously, with careful attention to the training and inference data at different stages, especially within complex frameworks that combine multiple large models.
- While I appreciate the discussion on failure cases, Figure 6 actually highlights that this work does not address the fundamental challenges in generative modeling—specifically, the limitations in distribution learning with limited data. In other words, much of the performance boost appears to come from improved pre-trained data representations and high-quality, sufficient data during fine-tuning, rather than advancements in the model’s actual ability to learn distributions.

Meta-review: This paper receive the ratings of 6, 6, 5, 5, 8, where the reviewers provided mixed feedback. In this paper, the authors introduce Lumina-mGPT, a decoder-only autoregressive multimodal model, designed to unify vision-language tasks while enabling flexible, photorealistic image generation. The paper emphasizes interesting techniques, such as Flexible Progressive Supervised Finetuning (FP-SFT) and Omnipotent Supervised Finetuning (Omni-SFT), for achieving scalability and task unification.

Strengths:
- The problem addressed by Lumina-mGPT is timely and relevant, with potential applications across multimodal AI.
- The proposed FP-SFT and Omni-SFT techniques are interesting attempts at addressing scalability and unification.
- The paper provides both qualitative and quantitative evaluations, with promising results on photorealistic image generation.

Area for improvements:
- Several reviewers pointed out that the novelty of this work is limited, especially as the differences between Lumina-mGPT and its predecessor, Chameleon, are minimal.
- Reviewers found the evaluation lacking in benchmarking against state-of-the-art diffusion models, particularly under identical training setups. This weakens the claim of performance parity with modern diffusion-based systems.
- The reliance on pretrained components (e.g., Chameleon) without detailed insights into training data and methodologies limits the transparency and reproducibility of the work.
- Ineffective instruction following, inconsistent artifact handling, and limited multilingual capabilities, further undermine the model's claimed generalizability.
- Clarity: the manuscript seems to overclaim certain achievements, which are not convincingly demonstrated.

While the work explores an important direction with some promising ideas, the current status does not meet the high standards of empirical rigor, clarity, and reproducibility required for acceptance at ICLR. We encourage the authors to address the area of improvements and resubmit to a future venue after further refinement and evaluation.

Decision: Reject