Title: UniGen: Enhanced Training & Test-Time Strategies for Unified Multimodal Understanding and Generation
Venue: NeurIPS 2025
Accepted: yes
Keywords: Unified Multimodal Understanding and Generation

Multimodal Foundation Model

Multimodal LLM
Abstract: We introduce UniGen, a unified multimodal large language model (MLLM) capable of image understanding and generation. We study the full training pipeline of UniGen from a data-centric perspective, including multi-stage pre-training, supervised fine-tuning, and direct preference optimization. More importantly, we propose a new Chain-of-Thought Verification (CoT-V) strategy for test-time scaling, which significantly boosts UniGen’s image generation quality using a simple Best-of-N test-time strategy. Specifically, CoT-V enables UniGen to act as both image generator and verifier at test time, assessing the semantic alignment between a text prompt and its generated image in a step-by-step CoT manner. Trained entirely on open-source datasets across all stages, UniGen achieves state-of-the-art performance on a range of image understanding and generation benchmarks, with a final score of 0.78 on GenEval and 85.19 on DPG-Bench. Through extensive ablation studies, our work provides actionable insights and addresses key challenges in the full life cycle of building unified MLLMs, contributing meaningful directions to future research. Code is available at https://github.com/apple/ml-unigen.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents UniGen, a unified multimodal large language model (MLLM) for image understanding and generation. It proposes a comprehensive training pipeline including pretraining, supervised fine-tuning, and direct preference optimization. A test-time Chain-of-Thought Verification (CoT-V) method is introduced to enhance generation quality through self-verification and Best-of-N selection.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n1. **Clear Pipeline:** The paper outlines a clear and systematic methodology, covering multi-stage pretraining, supervised fine-tuning, preference optimization, and test-time reasoning. This end-to-end design provides practical insights into constructing unified multimodal large language models.\n\n2.  **Good Empirical Results:** UniGen achieves notable improvements over prior unified MLLMs on multiple benchmarks, including GENEVAL and DPG-BENCH, demonstrating the effectiveness of its design and training strategies.\n\nWeaknesses:\n\n1. **Limited Scale:**  The current model is constrained to 1.5 billion parameters. While the results are competitive, extending the framework to larger model sizes could further strengthen the contribution.\n\n2. **High Test-Time Overhead:** The proposed Chain-of-Thought Verification (CoT-V) strategy involves evaluating multiple candidates per prompt, which may be computationally expensive. A detailed analysis of test-time cost and latency would improve the practicality assessment of the method.\n\nQUESTIONS:\n1. **Comparison Between Model Scaling and Test-Time Scaling:** I’m curious about the trade-offs between scaling the model size (e.g., training a larger MLLM) versus applying test-time strategies like CoT-V with a smaller model. How do they compare in terms of performance gains and computational overhead during inference?\n\n2. **Open-Source Concern:** Since a major contribution of this work is the unified training pipeline and the use of open-source data (with additional synthetic annotations), there is a concern about whether the authors will release the training code and datasets. Making these resources publicly available would greatly benefit reproducibility and further research.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-22"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a unified image understanding and generation model UniGen. This model borrows the ideas from some established models (e.g., Janus, Show-o) to constitute its architecture and undergoes five training stages: pre-training, supervised fine-tuning (SFT), direct preference optimization, and test-time scaling learning. The authors propose a post-training approach named Chain-of-Thought Verification, which uses UniGen itself as the verifier to evaluate the generated images and produce CoT data. They then conduct SFT on the CoT data to endow the model with thinking ability. In experiments, UniGen surpasses other MLLMs by notable margins.\n\nSTRENGTHS AND WEAKNESSES:\n> Strengths:\n\n1. Unifying image understanding and generation with one MLLM has been a hot and interesting topic in recent years. This paper provides a new unified MLLM with SOTA performance for the community, which can advance this field.\n2.  The proposed CoT-V method is novel and effective. The idea of asking the model to answer a series of atomic questions to serve as the thinking process is interesting and can intuitively enhance model generation performance.\n3. The effectiveness of CoT-V is not only grounded in UniGen but also in unified models, indicating this technique’s generalizability.\n4. The constructed preference dataset would be a valuable contribution to the community if published.\n\n> Weaknesses:\n\n1. The resolution of Fig. 1 is low. Please make a refinement.\n2. In Introduction, the authors state that “we walk through the entire life cycle of the model development” and “draw insightful lessons”. However, in experiments, the authors seem not to show what these “lessons” are. They primarily focus on demonstrating the effectiveness of their specific model design or training strategies, rather than introducing some globally useful knowledge or experiences that can benefit the community in designing such sophisticated models. Can the authors share some experiences (the learned “lessons”) regarding training or model design from a more global perspective?\n3. It is unclear whether the model weights and curated dataset in this paper will be published. If so, the authors should make a clear statement somewhere in the main text.\n4. In DPO training, what is the size of preferred samples $y_w$ and rejected samples $y_l$ used in experiments?\n5. In L193, the authors say that “RV struggles with free-form or complex instructions, such as those in DPG-Bench [21].” and choose CoT-V as a better option. The description is too limited to demonstrate RV’s drawbacks. Since the Best-of-N strategy can also be combined with RV and achieve similar functionality, the superiority of CoT-V over RV remains unclear. Can the authors give some examples of free-form or complex instructions to better illustrate RV’s limitation and give more details regarding CoT-V’s superiority?\n6. The authors use UniGen-DPO as a self-verifier to produce training data for test-time scaling. The model can generate low-quality or erroneous data, and it seems that no human correction has been applied to this data. How do the authors ensure data quality, and will low-quality data hamper model learning?\n7. During training, UniGen first undergoes DPO and then the test-time scaling training. However, it seems that the model could first undergo the test-time scaling stage and then DPO, since training with CoT-V is simply supervised fine-tuning. This also makes sense to first foster the model’s CoT-based reasoning ability and then optimize its preference. What is the reason behind choosing the original order? Will this lead to better model performance? I recommend the authors provide more clarification on this point.\n8. typo: L257: stages -> stages.\n\nQUESTIONS:\nSee weaknesses.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-22"}

comments: SUMMARY:
This paper proposes a unified model for multimodal understanding and generation. To improve image generation quality, the authors introduce a method called Chain-of-Thought Verification (COT-V) for use during the inference phase. The model first generates a set of candidate images, then utilizes its multimodal understanding capabilities to score each image and selects the one with the highest score as the final output.

STRENGTHS AND WEAKNESSES:
Strengths:
- The paper is clearly written and easy to follow. The motivation is reasonable, and the method is simple and reproducible.
- Leveraging the chain-of-thought reasoning process of multimodal models to evaluate image quality is an interesting idea.

Weaknesses:
- The core contribution, Chain-of-Thought Verification, does not appear to emerge naturally from the model but instead relies on fine-tuning with specific datasets. These datasets are based on rule-based scoring, which may limit the potential of the proposed method.
- There is randomness in the model's output during the chain-of-thought scoring process. How can one ensure that the output format consistently matches the expected structure, such as the format shown in Line 187?

QUESTIONS:
Please see the weaknesses.

LIMITATIONS:
yes.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n# Summary\n\n---\n\nThis work introduces a novel 1.5B-parameter model that unifies both image understanding and generation within a single framework. The model is trained through a comprehensive pipeline, including two-stage pre-training, SFT, DPO, and an additional CoT-V post-training phase to enable inference-time scaling with CoT-V techniques. For model design, the authors employ MAGVIT-v2 as the image tokenizer and utilize a cosine masking schedule for image decoding following MaskGIT. The image understanding encoder uses the widely adopted SigLIP. Both the encoder and decoder are kept frozen throughout all training stages, with only the LLM and connector modules being trainable. Notably, during pre-training, the image tokens are also optimized on understanding tasks to enhance the semantics of the image tokens.\n\nSTRENGTHS AND WEAKNESSES:\n# Strength\n\n---\n\n- The model demonstrates impressive performance despite its small 1.5B parameter size.\n- The CoT-V inference-time scaling technique is both innovative and interesting.\n- The implementation is solid, with comprehensive training stages that span from pre-training to preference learning.\n\n# Weakness\n\n---\n\n1. **What are the differences between the PT-1 and PT-2 stages?** Section 3.2 does not clearly state the distinction between these two stages. The datasets, augmented datasets, and tasks (t2i, i2t, t) all appear to be the same. Further clarification on how PT-1 and PT-2 differ would be helpful.\n2. **What is the key novelty of UniGen compared to prior work?** UniGen seems to combine multiple existing modeling designs. For example, its architecture is highly similar to Janus, with both featuring decoupled generation (MAGViT) and understanding (SigLIP) encoders, as well as similar training strategies. While some differences exist, such as the use of a cosine masking schedule for parallel visual decoding, multi-stage pretraining, and frozen encoders throughout all stages, there is a lack of ablation studies to assess the impact of these techniques. The explanations for improved performance (lines 240–241) do not sufficiently address differences compared to Janus, leaving the true reasons for UniGen’s superior performance underexplored.\n3. **Regarding Table 2**: Does only UniGen utilize BoN samples for image generation? If so, this could make the comparison unfair, as such techniques like contrastive reranking [1] can significantly affect generation performance.\n4. **Additional ablation studies are needed**: Ablations should cover not only different training stages (PT-1, PT-2, SFT, DPO, CoT-V post-training), but also the key novelties of this work. For example:\n- The effect of the generation encoder and decoder (vs. diffusion-based or flow matching-based methods)\n- The impact of masking-based parallel visual decoding vs. causal decoding [2, 3]\n- Frozen vs. tunable visual encoders during SFT (for comparison with Janus)\n- The effect of different tokenizers\n- The effect of not using the generation encoder for understanding tasks during pre-training\n- (At least) Comparisons among OV, RV, and CoT-V\n- Among others.\n5. **Lack of evaluation on text-only tasks**: The paper does not report results on text-only tasks. How do the authors draw conclusions regarding preserved text performance?\n\n[1] Scaling Autoregressive Models for Content-Rich Text-to-Image Generation\n\n[2] Making LLaMA SEE and Draw with SEED Tokenizer\n\n[3] MIO: A Foundation Model on Multimodal Tokens\n\nQUESTIONS:\n# Questions (bonus)\n\n---\n\n- I am also curious about the scaling performance of UniGen. In your scaling experiments (if any), did you encounter any bottlenecks caused by the generation decoder? Specifically, as the model size increases, did you observe that the image generation ability tends to saturate? If so, is this likely due to limitations of the image tokenizer, especially the generation decoder?\n- How do the authors recap the PT-1 stage data using Qwen2.5-VL-7B? What prompts did you use for this process?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-14"}