Fantasy: Transformer Meets Transformer in Text-to-Image Generation

Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-11 07:33:51

Abstract: We present Fantasy, an efficient text-to-image generation model marrying the decoder-only Large Language Models (LLMs) and transformer-based masked image modeling (MIM). While diffusion models are currently in a leading position in this task, we demonstrate that with appropriate training strategies and high-quality data, MIM can also achieve comparable performance. By incorporating pre-trained decoder-only LLMs as the text encoder, we observe a significant improvement in text fidelity compared to the widely used CLIP text encoder, enhancing the text-image alignment. Our training approach involves two stages: 1) large-scale concept alignment pre-training, and 2) fine-tuning with high-quality instruction-image data. Evaluations on FID, HPSv2 benchmarks, and human feedback demonstrate the competitive performance of Fantasy against state-of-the-art diffusion and autoregressive models.

Main paper: 
 problem formulation: 
 As depicted in Fig. 3 , Fantasy consists of a pre-trained text encoder T , a transformer-based masked image generator G, a sampler S, a frozen VQGAN, and a pre-trained super-resolution model. T maps a text prompt t to a continuous embedding space. G processes a text embedding e to generate logits l for the visual token sequence. S draws a sequence of visual tokens v from logits via iterative decoding [4] , which runs N steps of inference conditioned on the text embeddings e and visual tokens decoded from previous steps. Finally, D maps the sequence of discrete tokens to pixel space Z. To summarize, given a text prompt t, an image x is synthesized as follows:EQUATION)where n is the synthesis step, and l n are logits, from which the next set of visual tokens v n+1 are sampled. M denotes the masking operator that applies masks to the token in v n . We refer to [4, 3] for details on the iterative decoding process. The Phi-2 [24] for T and VQGAN [8] for encoder E and decoder D are used. G is trained on a large text-image pairs D using masked visual token modeling loss:EQUATIONwhere CE is a weighted cross-entropy calculated by summing only over the unmasked tokens.VQGAN [8] is capable of transforming each image into discrete tokens with higher-level semantic information from a learned codebook, while ignoring low level noise. The autoregressive tokens prediction of VQGAN shares the same form as text tokens generated by LLMs. Prior research [46] has shown that unifying vision and language by the same token space could enhance the coherency for vision-text alignment. Furthermore, compared with RGB pixels, the visual token representation has proven to reduce disk storage and improve the capability of robustness and generalization.To reduce the computational burden, we initially compress an RGB image v ∈ R H×W ×3 into a diminished representation with a resolution of h × w × 3, where h = H/f and w = W/f , with f denoting the downsampling factor. We then employ a pre-trained f 16 VQGAN [27] encoder E to quantizate images x ∈ R 3×256×256 into discrete tokens of spatial dimensions 16 × 16 from a pre-trained codebook Z = {z k } K k=1 consisting of K = 8192 vectors, resulting in the quantized representation z = E(x, Z).Recent studies [10, 5, 3] tend to use encoder-decoder LLMs [31] for text encoding over CLIP [30] ,which is adept at handling tasks that involve complex mappings between input and output sequences.Due to the tremendous success of ChatGPT, attention has been drawn to models that consist solely of a decoder. Also, [43] presents an insight that efficiently fine-tuning a more powerful decoder-only LLM can yield stronger performance in long-text alignment. Consequently, to capitalize on the enhanced semantic comprehension and generalization potential of LLMs while simultaneously reducing the training burden, we employ Phi-2 [24], a state-of-the-art, lightweight LLM, as the text encoder.Given the text prompt t, Fantasy first passes it through Phi-2, extracting the text embedding from the last hidden layer L. However, typically, decoder-only architectures are not adept at feature extraction and mapping tasks. [23] proposes that the conceptual representations learned by LLM's are roughly linearly mappable to those learned by models trained on vision tasks. Therefore, the embedding vectors are linearly projected to the hidden size of the image generator G:EQUATIONwhere T (•) denotes the decoder-only Phi-2 and L is the index of the last hidden layer. P represents the projection from text space to visual space, and c is the text feature suitable for the image generator.MIM narrows the gap between its modeling and the extensively studied area of language modeling, making it straightforward to leverage the findings of the LLMs research community. Therefore, we adopt a masked transformer as the image generator backbone of Fantasy [46] .During training, we leave the projected text embeddings c unmasked and the image tokens z are masked at a variable masking rate based on a Cosine scheduling M as [4, 3] . Specifically, for each training example, we sample a masking rate r from [0, 1] from a truncated arccos distribution with density function p(r) = 2 π (1 -r 2 ) -1 2 . While autoregressive methods learn fixed-order token distributions P (z i |z <i ), random masking with variable ratios enables learning P (z i |z ̸ =i ) for any token subset, crucial for our parallel sampling scheme. The sampling of a new state s n+1 at each successive step is conditioned on the previous state and the specified text condition c: EQUATIONEQUATIONAt the output layer, to reduce the training burden, ConvMLP [18] is utilized to transform masked image embeddings into logits sets, aligning with the VQGAN codebook dimensions. Eventually, the reconstructed lower-resolution tokens are restored with the pre-trained 256 × 256 resolution VQGAN decoder to the pixel space, resulting in the generated image x:EQUATIONFig. 3 illustrates Fantasy's two-stage training approach. Following prior works [43, 35, 9] , we employ large-scale pre-training to achieve general text-image concept alignment, and simultaneous fine-tuning of Phi-2 [24] and the masked image generator using high-quality instruction-image pairs.Pre-training Stage. To perform general text-image concept alignment, the VQGAN and LLM weights are frozen, and only the image generator is pre-trained on deduplicated LAION-2B [37] with images above a 4.5 aesthetic score. We exclusively preserve prompts in English, filter out images above a 50% watermark probability or above a 45% NSFW probability, yielding a final set of 9 million images. Since the computational cost of upsampling is much lower than training a super-resolution model, Fantasy is started with training at a resolution of 256 × 256. Note that the pre-training only needs approximate image-text alignment, substantially lowering the training costs.Fine-tuning Stage. [43] has proven that LLMs trained solely on text data lack prior image-text knowledge, and that merely aligning their text embeddings with visual features might not be optimal.Therefore, in the second stage, we gather an internal dataset of 7 million high-quality instructionimage pairs to fine-tune both the Phi-2 model and the image generator of Fantasy, which ensures enhanced compatibility of text embeddings within the text-image pair space, facilitating the use of decoder-only LLMs in text-to-image generation tasks and harnessing their inherent advantages. To prevent catastrophic forgetting in LLMs and preserve their understanding abilities during training, we select questions from BIG-bench [2] and monitor the common sense question-answering ability of Phi-2 in real-time throughout the training process. We construct our training dataset for the fine-tuning stage by incorporating JourneyDB [39] and an internal synthetic dataset to enhance the aesthetic quality of generated images beyond realistic photographs. To facilitate instruction-image alignment learning, we retain only data with descriptions exceeding 30 words, as these provide enough detailed insights into the image objects, including attributes and spatial relations.With this approach, Fantasy trains a 0.6B parameter T2I model in about 69 A100 GPU days, significantly reducing computation compared to existing diffusion-based methods, while maintaining comparable visual and numerical fidelity. Throughout this paper, we present a comprehensive evaluation of Fantasy's efficacy, showcasing the potential in training high-quality transformer-based image synthesis models compared to diffusion-based models in future.To ensure rough alignment in the pre-training phase, we utilize the large-scale dataset LAION-2B [37] and apply the filtering strategy developed by DataComp [14] . Furthermore, we gather longtext prompts and corresponding high-quality images to achieve finer-grained text-image alignment through instruction tuning. CapsFusion [47] employs a fine-tuned LLaMA [40] for recaptioning LAION-2B [37] and LAION-COCO [1] . However, this approach still results in suboptimal image quality and occasional mismatches between images and text. SAM-LLAVA [5] utilizes LLaVA [20] to recaption the SAM dataset [17] , which leads to images with blurred faces, a consequence of the dataset's inherent face-blurring. Therefore, we shift focus to synthesize images, mainly including DiffusionDB [42] and JourneyDB [39] , produced by Stable Diffusion and MidJourney, respectively.To augment the diversity of the images, we minimize the use of datasets from specific domains, such as gaming and anime. Furthermore, we implement filtering to discard texts with special characters and data containing violence or pornography, retaining only instructions exceeding 30 words.Question: Does the paper discuss the limitations of the work performed by the authors?Answer: [Yes]Justification: We have included a comprehensive discussion on limitations, particularly focusing on the scalability of our simulations in extremely large urban environments and potential biases in the modeling processes.Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.• The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.• The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.• The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.• The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.• If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.• While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. 
experiments: 
 In this section, we outline detailed training, inference, and evaluation protocols, followed by comprehensive comparisons across three key metrics.Training Details. Different from the prior works [9, 43, 32, 34] , we used a lightweight but powerful decoder-only large language model Phi-2 [24] as the text encoder. Diverging from prior approaches that extract a standard and fixed short text tokens, we extend the extraction to 256 tokens to master long-term instruction-image alignment, ensuring precise alignment for more fine-grained prompts.For the entire training process, we train Fantasy on 4×A100 80G GPUs and set the accumulation step to 2. At different stages, we employ varying learning rate strategies with single-cycle cosine annealing decay. Furthermore, the AdamW optimizer [22] is utilized with a weight decay of 0.01.Fantasy trains a 0.6B parameter T2I model in about 84.5 A100 GPU days, significantly reducing computation compared to existing diffusion-based methods as shown in Fig. 1 . Inference Details. We use N = 32 sampling steps in all of our evaluation experiments. SinceFantasy is trained at a resolution of 256 × 256, we employ the pre-trained diffusion-based superresolution model StableSR [41] to upscale images to 512 × 512.Evaluation Metrics. We comprehensively evaluate Fantasy via four primary metrics, i.e., alignment on HPSv2 [44] , FID [16] on MSCOCO dataset [19] and human evaluation on a collected dataset.Results on HPSv2. We utilize HPSv2 [44] as our primary automated metric, a preference prediction model which can be used to compare images generated with the same prompt across five categories:anime, concept art, paintings, photography, and DrawBench [36] . We present the results of HPSv2 between Fantasy and other state-of-the-art generative models in Tab. 1. Fantasy exhibited outstanding performance across all key aspects among previous Transformer-based methods like CogView2 [12] , which is expected. The results also reveal its competitive performance compared to prior diffusion-based methods, especially in concept-art and painting, demonstrating similar performance to DALL•E 2 [26] . This remarkable performance is primarily attributed to the text-image alignment learning in fine-tuning stage, where high-quality text-image pairs were leveraged to achieve superior alignment capabilities. In comparison, DeepFloyd-XL and other diffusion-based models achieve better scores, while utilizing larger models with significantly higher compute budget.Results on FID. We employ FID [16] to evaluate our models on COCO-30K [19] . To allow for a fair comparison, all images are downsampled to 256 × 256 pixels. The comparison between our method and other methods in FID, and their training time is summarized in Tab. 2. We observe that the FID of Fantasy is substantially higher compared to other state-of-the-art models. Visual inspections reveal that images generated by Fantasy are smoother than those from other leading T2I models. This discrepancy is most noticeable in real-world images like COCO, on which we compute the FID-metric. Although the state-of-the-art models [43, 11, 29] exhibit lower FID, it relies on unaffordable resources. Furthermore, prior studies [29, 5, 11] have demonstrated that FID may not Figure 4 : User study on prompts with different length. VC. , CV2. , FT. , SD. , and PA. refer to VQGAN+CLIP [8] , CogView2 [12] , our Fantasy, Stable Diffusion v2.0 [33] , and Pixart-α [5] .be an appropriate metric for image quality evaluation, as a lower score does not necessarily reflect superior image generation, and it is more authoritative to use the evaluation of human users.Following prior works [5, 43, 28] , we also conduct a study with human participants to supplement our evaluation and provide a more intuitive assessment of Fantasy's performance. Participants are asked to select a preference of the images based on the visual appeal of the generated images and the precision of alignments between the text prompts and the corresponding images.As involving human evaluators can be time-consuming, we choose the top-performing open-source diffusion-based models (e.g., SD XL [33] , and Pixart-α [5] ) and transformer-based models (e.g., VQGAN+CLIP [8] and CogView2 [12] ) as our baseline, which are accessible through APIs and capable of generating images. We randomly select a total of 600 prompts from existing prompt sets (e.g., ParaPrompt [43] , ViLG-300 [13] , COCO Captions [6] ). To comprehensively contrast the capabilities of Fantasy and other models in interpreting text prompts of varying lengths, we allocate one subset to consist of 300 prompts ranging from 10 to 30 characters and another subset comprising 300 prompts exceeding 30 characters. For each model, we use a consistent set to generate images, which are then evaluated by 50 individuals. A close-up photo of a person. The subject is a male. He was wearing a wide-brimmed hat, a gray-white beard on his face, a brown coat. His facial expression looked pensive and serious, with the clear blue sky in the background.A young man wearing a black leather jacket and tie stood behind an old door, his gaze firmly fixed on the camera. The door had patterns of leaves and flowers on it, revealing a yellow background. His hair was casually curled and he appeared to be deep in thought or contemplating something. ParaDiffusion [43] pioneers the use of decoder-only large language models as text encoders in text-to-image generation. As illustrated in Fig. 6 , our observations suggest that Fantasy more closely aligns details with prompts than ParaDiffusion [43] .Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?Answer: [Yes] Justification: All theoretical results are accompanied by a clear statement of assumptions and are supported by complete proofs provided in the supplementary materials. Each theorem and lemma are properly referenced and numbered for clarity and ease of access.Guidelines:• The answer NA means that the paper does not include theoretical results.• All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.• All assumptions should be clearly stated or referenced in the statement of any theorems.• The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.• Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.• Theorems and Lemmas that the proof relies upon should be properly referenced. 
ablation study: 
 This section analyzes the effects of LLMs fine-tuning, and model scale on Fantasy's performance through ablation studies. More ablation study refers to appendix.To assess the effect of training strategies on the comprehension of complex instructions, we perform a human preference evaluation, as detailed in Sec. 3.3, using a subset of 300 prompts longer than 30 characters. 'Base' denotes general text-image alignment with filtered LAION-2B [1] in the pre-training stage. Compared to the base model, our synergy fine-tuning with Phi-2 demonstrates a notable improvement in all aspects in Tab. 3. A wooden outhouse sitting in the grass near trees.A small kitchen does have plenty of cabinets. L6 L12 L22 Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?Answer: [Yes]Justification: The paper provides detailed descriptions of the experimental setup, including data splits, hyperparameters, and the type of optimizer used. We also provide access to the source code and datasets in the supplementary materials to ensure full reproducibility.Guidelines:• The answer NA means that the paper does not include experiments.• If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.• If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.• Depending on the contribution, reproducibility can be accomplished in various ways.For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.• While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example (d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results. Guidelines:• The answer NA means that paper does not include experiments requiring code.• Please see the NeurIPS code and data submission guidelines (https://nips.cc/ public/guides/CodeSubmissionPolicy) for more details.• While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).• The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.• The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.• The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.• At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).• Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
conclusion: 
 In this paper, we introduce Fantasy, a lightweight and efficient text-to-image model that combines Large Language Models (LLMs) with a transformer-based masked image modeling (MIM), effectively transferring semantic understanding capabilities from LLMs to the text-to-image generation.With our proposed two-stage training strategy and high-quality dataset, Fantasy significantly reduces computational requirements while producing high-fidelity images. Extensive experiments demonstrate that Fantasy achieves comparable performance to models trained with significantly more computational resources, illustrating the viability of our approach and suggesting potential efficient scalability to even larger masked image modeling for text-to-image generation.NeurIPS Paper Checklist Guidelines:• The answer NA means that the abstract and introduction do not include the claims made in the paper.• The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.• The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.• It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes] Justification: The experimental section of the paper provides comprehensive details about the training and test setups, including the rationale behind choosing specific hyperparameters and the types of optimizers used.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.• The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: All experimental results are presented with error bars reflecting the standard deviation across multiple runs. We provide a detailed explanation of how these were calculated and the assumptions underlying our statistical tests.Guidelines:• The answer NA means that the paper does not include experiments.• The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.• The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).• The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)• The assumptions made should be given (e.g., Normally distributed errors).• It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.• For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).• If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes]Justification: The paper details the computational resources required for each experiment, including the types of GPUs used, the amount of memory, and the execution time. This ensures that other researchers can allocate the appropriate resources to reproduce our results.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.• The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.• The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes]Justification: The paper includes content about broader impacts that discusses both the potential positive applications of our method in educational and creative industries, and potential negative impacts, such as the misuse of generated images. We also suggest mitigation strategies for potential negative uses.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.• Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.• The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.• If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our paper poses no such risks. If then, we will describe the safeguards implemented in releasing our models, including usage guidelines and limitations to access, ensuring responsible use and mitigating risks of misuse.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.• Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.• We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Justification: Our research did not involve human subjects, thus no IRB approval was necessary.Guidelines:• The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.• We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.• For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
: 
 Recent advances in text-to-image (T2I) models [3, 5, 12] have become focal points within the computer vision field.Most advances in T2I models, focused on generating highquality images based on relatively short descriptions, struggle with intricate long-text semantic alignment due to inherent structure constraints and data limitations. Text encoders used for T2I fall into three categories: CLIP [30] , encoder-decoder LLMs, and decoder-only LLMs. Models using encoder-decoder LLMs like T5-XXL [31] have shown improved text-image alignment over CLIP by exploiting enhanced text understanding, increasing token capacity, yet without delving into the semantic alignment for longer texts. ParaDiffusion [43] indicates that directly aligning text embeddings with visual features without prior image-text knowledge is not the most effective approach. Previous works [38, 45] have highlighted shortcomings in existing text-image datasets [37] , including image-text mismatches, a lack of informative content, and a pronounced long-tail effect. These deficiencies notably impair training efficiency for T2I models and restrict their ability to learn complex semantic alignment.Existing diffusion-based T2I models [33, 5, 9, 26] have achieved unprecedented quality. However, as detailed in Fig. 1 , these advanced models come with significant computational demands. The Submitted to 38th Conference on Neural Information Processing Systems (NeurIPS 2024). Do not distribute.A snowy Sweden lake in a vibrant, cinematic style with intense detail and raytracing.A furry cat.Studio photo portrait of Lain Iwakura wearing floral garlands over her traditional dress.A tiny planet image of Rio de Janeiro.A 3d render of a cute, blue, anthropomorphic dragon with ice crystals growing off her, sharp focus.Majestic ornate great hall, grand library, baroque, torches, s t a i n e d g l a s s w i n d o w s , moonlight rays, dreamy mood.T h e s o l i t a r y g r e a t t r e e c e n t e r e d i n t h e i m a g e . cloudless s unny sky. little islands in the flooded plain.B r e a t h t a k i n g b e a u t i f u l , aesthetically pleasing, gouache ocean waves ripples, sea foam, sunset, digital concept art.Beautiful warm tavern seen from the outside, middle age, river crossed by a bridge next to the tavern, crepuscular light.Ted bundy in a pixar movie. considerable expenses of these models create significant barriers for researchers and entrepreneurs.Meanwhile, economical text-to-image models [25, 15, 48] compromise on image quality, yielding lower resolution and diminished aesthetic appeal.Given these challenges, a pivotal question arises: Can we develop a resource-efficient, high-quality image generator for long instructions? In this paper, we present Fantasy, significantly reducing training demands while maintaining the capability of instruction understanding and competitive image generation quality, as shown in Fig. 2 . To achieve this, we propose three core designs:Efficient T2I netwrok. To leverage the powerful understanding ability of a decoder-only LLM, we choose the lightweight Phi-2 [24] as our text encoder. We derive discrete image tokens from a pre-trained VQGAN [27] , and employ Transformer-based masked image modeling (MIM) as our T2Iarchitecture. We also utilize the pre-trained VQGAN decoder [27] for pixel space restoration.Hierarchical Training strategy. We propose a thoughtfully two-stage training strategy to address the high computational demands of current leading models while maintaining competitive performance:(1) large-scale concept alignment pre-training, (2) high-quality instruction-image fine-tuning. To facilitate a coarse image-text alignment, we initially train the T2I model from scratch using relatively lower-quality data. We then fine-tune the pre-trained T2I model and LLM on text-image pair data rich in information density with superior aesthetic quality.High-quality data. To achieve rough alignment while pre-training, we select the large-scale dataset LAION-2B [37] and employ the filtering strategy proposed by DataComp [14] . We collect longtext prompts and corresponding high-quality synthesized images for instruction tuning, including DiffusionDB [42] and JourneyDB [39] . We further filter and discard texts with special characters and data containing violence or pornography, retaining only instructions exceeding 30 words.Our main contributions are summarized as follows:1. We present Fantasy, a novel framework that is the first to integrate a lightweight decoder-only LLM and a Transformer-based MIM for text-to-image synthesis, allowing for long-form text alignment.2. We show that our two-stage training strategy with high-quality data enables MIM to achieve comparable performance at a significantly reduced training cost.3. We provide comprehensive validation of the model's efficacy based on automated metrics and human feedback for visual appeal and text faithfulness.  
 
 Appendix:  


Decision: The paper had 3 borderline reviews and 1 accept. 

- All reviewers point out that the quality of the images is not up to par with current models and there are visible artifacts / smoothness.
- The reviewers also pointed out that they fail to see significant contributions in the paper: it combines masked image generators with a decoder LLM (the only part being different is the use of a decoder LLM).
- The reviewers also mention that the advantages of a decoder-only LLM aren't demonstrated in depth. 

The AC agrees with the reviewers -- while fantasy introduces a lightweight training approach, there is a significant image quality gap which would have needed to be resolved for acceptance. The use of a decoder LLM is interesting, but lacks deeper analysis and ablations comparing against encoder LLMs to warrant acceptance.

Reviews: review: paper_summary: paper_summary: To develop a resource-efficient, high-quality image generator for long instructions, the authors presented Fantasy, an efficient T2I generation model that integrates a lightweight decoder-only LLM and a transformer-based masked image modeling (MIM). 

They demonstrate that with appropriate training strategies and high-quality data, MIM can also achieve comparable performance.

By incorporating pre-trained decoder-only LLMs as the text encoder, they observe a significant improvement in text fidelity compared to the widely used CLIP text encoder, enhancing the text image alignment. 

Their training includes two stages: 1) large-scale concept alignment pre-training, and 2) fine-tuning with high-quality instruction-image data. 

They conduct evaluation on FID, HPSv2 benchmarks, and human feedback, which demonstrate the competitive performance of Fantasy against other diffusion and autoregressive models.
strength_weakness: strength_weakness: - the major concern of the work is unclear contributions. The claimed three contributions or core designs are quite similar with existing works.
- Efficient T2I network: there is no justification about why the network is “efficient”. Simply adopting a smaller LLM like Phi-2 can hardly be claimed as efficient network design. 
- The hierarchical training strategy was also proposed before, it is not clear what is the difference with existing work.
- High quality data: the training data utilize Laion-2B and use existing filtering strategy. The collection high quality synthesized images from existing datasets.
- The evaluation metrics are mainly based on HPSv2, which has a limited range of values, e.g., HPSv2 has close values for SDv1.4 and SD2.0, e.g., 27.26 vs 27.48. Why SDXL is missing in Table 1?
- The author acknowledged that their model lags behind diffusion-based models in visual appeal, limited by the 8K size of VQGAN’s codebook and not targeting visual appeal. However, there is no solution or further study for solving this problem, which limits the scalability of the model.
- The scaling study in section 4.2 seems pretty premature and it is unclear what is the limit of the scaling. Increasing the model depth can improve the performance, which has been verified from previous work such as in https://arxiv.org/abs/2212.09748 or https://arxiv.org/abs/2404.02883.
questions: questions: - what is the major difference with existing MIM based methods such as Muse, beside different components and data strategy?
limitations: limitations: I would encourage the authors to emphasize about the core contributions rather than combining everything together, which can hardly show significant performance improvement over existing public models.
paper_summary: paper_summary: This paper proposes a technique for training transformer based masked image modeling in an efficient way. Two main contributions include (1) use of a LLM decoder as text embeddings, and (2) Two-stage training strategy for MIM models. Experimental results show good generation quality.
strength_weakness: strength_weakness: - I don't see anything new proposed in this paper. The authors simply use Phi-2 model as text encoder with MIM models, and use two-stage training. 
- Even two-stage training is not something new to image synthesis. People have been doing aesthetic finetuning to improve image quality in diffusion models (eg. stable diffusion). The authors extend this to instruction-image data.
- The quality of generated images are not very impressive. When zoomed in, we notice a lot of visible artifacts. The generated images are also flat and doesn't have a lot of details.
questions: questions: - Why use Phi-2 model when there are many LLMs available? Is this for efficiency?
limitations: limitations: N/A
paper_summary: paper_summary: This paper proposes an efficient text-to-image generation model that integrates LLM and MIM. It demonstrates that MIM can achieve comparable performance. Unlike commonly used text encoders like CLIP and T5, this study introduces an efficient decoder-only LLM, phi-3, achieving better semantic understanding. The effectiveness of the method is validated through a newly proposed two-stage training approach and sufficient experiments.
strength_weakness: strength_weakness: 1. The quality of the generated images does not yet match that of existing methods (e.g., pixart-alpha, SDXL), with some loss of detail. This is noticeable from the comparison in column B of Figure 5.
2. Some aspects of the methodology could be clearer, and the overall coherence of the approach could be strengthened.
3. While the proposed method demonstrates efficiency advantages, particularly in faster training convergence, this can be influenced by various factors. However, the related experiments in the paper could be more comprehensive.
4. The semantic accuracy of the generated images, a potential strength of Fantasy, is not fully demonstrated in the paper. For instance, the model's ability to handle prompts with multiple entities, color attribute descriptions, or retaining key elements in long text inputs is not adequately showcased.
questions: questions: 1. The results in Figure 1 should be based on a consistent benchmark for image quality. Could you provide more detailed information about this benchmark?
2. Why is Phi-2 used for the LLM? If it is interchangeable, would it be possible to include comparison experiments using CLIP or T5?
3. Does the model also have advantages in inference speed, or is it comparable to existing methods?
4. The semantic accuracy of the generated images should be a strength of Fantasy, but this is not fully demonstrated in the paper. For example, it would be beneficial to show how the model handles prompts with multiple entities, color attribute descriptions, or retains key elements in long text inputs.
limitations: limitations: Yes.
paper_summary: paper_summary: The paper proposes Fantasy, a T2I model based fully on transformers (except for the VQGAN for the latents encoding and decoder):
* A __fine-tuned LLM__ (based on Phi-2) for the text encoding
* A image generator based on the MIM (Masked Image Modelling) approach

The training happens in two stages, a generic stage for aligning the generator the the frozen Phi-2 features, followed by a fine-tuning stage where the Phi-2 encoder is fine-tuned alongside the MIM transformer.

The results on human evaluations are convincing, putting Fantasy alongside models that require larger computational resources, while the FID results are less convincing (due to the image being smooth according to the authors).
strength_weakness: strength_weakness: Performance:
* The FID scores are not competitive and the authors describe why: the image are smooth => it seems that the human evaluations still rank Fantasy at the top on visual appeal, but it might be that if the question was "visual realism" they might prefer a different model
* Results are available for 256px, and a 600M parameters MIM generator, there is no proof that this method scales (we know that diffusion models based on UNet have trouble scaling for instance)
questions: questions: n/a
limitations: limitations: n/a

Comments: comment: Dear Reviewer AAcU,

As the discussion phase ends today, we will not be able to further clarify potential additional concerns. We would be very grateful if you could respond to our further comment and offer us an opportunity to address any questions you might have!

Thank you again for your time and feedback!

Best,

Authors
Dear Reviewer 5cW5,

As the discussion phase ends today, we will not be able to further clarify potential additional concerns. We would be very grateful if you could respond to our further comment and offer us an opportunity to address any questions you might have!

Thank you again for your time and feedback!

Best,

Authors
Dear Reviewer soBM,

Thank you very much for your feedback and for increasing the rating based on our rebuttal. We appreciate your suggestion to incorporate the changes discussed in the rebuttal into the final revision, and will ensure that these adjustments are clearly reflected in the final version of our paper.

Thank you once again for your constructive critique and guidance throughout this review process.

Best regards,

Authors
Thank you for providing the detailed answers. My concerns have been resolved, thus, I will increase the rating by 1. I recommend that the authors incorporate the changes discussed in the rebuttal into the final revision.
Dear Reviewer aiiy,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our **core contributions** and **distinguish our method from other two-stage training models** for all reviewers. We address the **Scaling Study’s significance and future directions**, and **clarify our benchmark selection** as recommended.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors
Dear Reviewer soBM,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our **core contributions** for all reviewers. We explain our **use of Phi-2**, **image detail limitations**. We also **describe the Figure 1** in the paper, and **compare the inference speed** with others. Following your advice, we **detail additional training steps** and include **more generated images** with long texts in the PDF.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors
Dear Reviewer AAcU,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our **core contributions** and **distinguish our strategy from other hierarchical training strategy** both in data and components for all reviewers. We explain our **use of Phi-2** and **image detail limitations** as recommended.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors
Dear Reviewer 5cW5,

We sincerely thank you for your insightful and constructive feedback. We have incorporated your suggestions and responded comprehensively to in the comments.

We highlight our **core contributions** and **distinguish our strategy from other hierarchical training strategy** both in data and components for all reviewers. We refine our **efficiency definition**, **contrast our approach with MUSE**, and **incorporate baseline results** as recommended. We also address the **Scaling Study’s significance and future directions**.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors
Dear Reviewers, Area Chairs, Senior Area Chairs, and Program Chairs, 

We sincerely thank the Reviewers for their insightful and constructive feedback. We have incorporated their suggestions and responded comprehensively to their comments. 

- We highlight our **core contributions** and distinguish our method from other two-stage training models for all reviewers.
- We address the **Scaling Study**’s significance and future directions with 5cW5 and aiiy, and explain our **use of Phi-2** and **image detail limitations** to AAcU and soBM.
- We also refine our **efficiency definition**, contrast our approach with MUSE, and incorporate baseline results as recommended by 5cW5.
- Following soBM’s advice, we detail additional training steps and include **more generated images** with long texts in the PDF.
- Our **benchmark selections** are clarified for aiiy.

We hope that our study will be well-received as a valuable contribution to the NeurIPS' focus on theory & application. We are available for any further discussions or inquiries the reviewers may have during the reviewer-author discussion period.

Best regards,

Authors