Title: UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
Venue: NeurIPS 2025
Accepted: yes
Keywords: Unified Vision-Language Model

Personalization

Understanding and Generation
Abstract: Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation \textit{\textbf{personalized attribute-reasoning generation}}. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{https://github.com/arctanxarc/UniCTokens}{https://github.com/arctanxarc/UniCTokens}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UniCTokens, a novel personalization framework that enhances both personalized generation and understanding in a unified VLM by fine-tuning unified concept tokens. Unlike prior methods that treat these two tasks independently, UniCTokens enables mutual enhancement by sharing learned representations across tasks. The authors also propose UnifyBench, a new benchmark designed to evaluate concept understanding, concept generation, and knowledge-driven generation within a unified framework. Experimental results on UnifyBench demonstrate the effectiveness and efficiency of the proposed approach.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n\n-\tThis paper proposes a new framework for personalizing unified VLMs by fine-tuning unified concept tokens for both understanding and generation.\n-\tThe proposed UnifyBench benchmark seems to contribute to the evaluation of concept understanding, concept generation, and knowledge-driven generation of a personalized unified model.\n-\tUniCTokens achieves state-of-the-art performance while maintaining a smaller model size and requiring fewer training samples.\n\nWeaknesses\n\n-\tThe term “Personalized Knowledge-Driven Generation” might be misleading. As presented, it largely refers to image generation guided by personalized textual attributes (e.g., \"⟨x⟩ wearing its hat\"), rather than a more complex or knowledge-integrative reasoning process. Clarifying this terminology would improve reader comprehension. \n-\tThe presentation of results lacks clarity in Table 1. The content (L208-214, L216-219) suggests that it includes results for \"Personalized Concept Generation\" and \"Personalized Knowledge-Driven Generation\", but only \"Personalized Generation\" appear to be explicitly reported in this table. \n-\tThe paper does not address how the model handles prompts that include multiple concepts (e.g., “⟨x⟩ and ⟨y⟩ sitting together”), which is a common scenario in real-world personalization tasks. Including an analysis or discussion of multi-concept generation would strengthen the work.\n\nQUESTIONS:\nWhile UniCTokens enables efficient personalized understanding and generation with only a few training samples per concept, the framework seems to require training for each new concept. How scalable is this approach in scenarios where a large number of personalized concepts need to be integrated? Additionally, what are the computational and time costs associated with adding new concepts?\n\nLIMITATIONS:\nPlease refer to \"weakness\".", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes UniCTokens, a framework integrating personalized information into a unified vision language model (VLM) for understanding and generation. The paper proposes a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between these tasks.  Moreover, it presents a benchmark UnifyBench to evaluate the unified VLM personalization.\n\nSTRENGTHS AND WEAKNESSES:\nPros\n1. The unified concept tokens proposed in the paper enable learnable tokens to be simultaneously applied to both understanding and generation tasks, breaking the limitations of tasks' isolation.\n2. The three-stage training method progressively enhances the model's understanding and generation capabilities while enabling mutual improvement between them.\n3. The paper presents the first personalized dataset, which covers three categories: humans, pets, and objects.\n\nCons\n1. Regarding Section 1.3.1 describing the training objectives of unified concept tokens M, provide more detailed differentiation between Objective 2 and Objective 3 to better highlight their differences and implementation challenges.\n2. For image generation evaluation, introducing more challenging tests with noisy backgrounds (e.g., generating the target concept <bo> alongside similar category instances) would provide a more rigorous assessment of the model's concept retention and robustness in complex scenarios.\n\nQUESTIONS:\nL44, mutual information, what does it mean? \nThe current evalution on several tasks, such VQA, how about other complex VL tasks, such as visual dialogue.\n\nLIMITATIONS:\nK-means is adopt for clustering. So some ad-hoc parameter is required. \nThe training of the model has three stages. So the reproducibility is a little in doubt.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces UniCTokens, a novel framework for personalizing a unified VLM to handle both understanding and generation tasks for user-provided concepts. The key idea is to move away from separate concept tokens for each task and instead train a single, unified set of concept tokens. A core contribution is a three-stage progressive training strategy. To evaluate their method, the authors also introduce UnifyBench, a new benchmark designed to assess personalized understanding, generation, and a new task they define as \"knowledge-driven generation,\" where the model must generate images combining visual concepts with attributes described only in text.\n\nSTRENGTHS AND WEAKNESSES:\nThis paper presents a novel approach to personalization in unified VLMs by using a single set of concept tokens for both understanding and generation. This contrasts with prior work like Yo’Chameleon and enables more effective cross-task transfer, particularly for the “knowledge-driven generation” task. The three-stage training strategy is technically sound and the feedback loop introduced in Stage 3 is especially insightful.\n\nExperiments are thorough, with strong baselines and a new benchmark (UnifyBench) supporting the claims. Ablations clearly show the benefit of each component.\n\nHowever, the framework adds complexity by introducing multiple token types and training phases, and it's unclear if a simpler setup could achieve similar results. The model scale is relatively small (1.3B), so it's uncertain how well the approach generalizes. Also, evaluation relies heavily on GPT-4o, which may introduce bias.\n\nQUESTIONS:\nI'm mostly curious about the complexity and scalability. Is the three-stage process truly necessary, or could a simpler joint-training scheme work? Meanwhile, beside 1.3B model, have you tried on 7B Show-o?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}