UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
Authors: Ruichuan An
Sihan Yang
Renrui Zhang
Zijun Shen
Ming Lu
Gaole Dai
Hao Liang
Ziyu Guo
Shilin Yan
Yulin Luo
Bocheng Zou
Chaoqun Yang
Wentao Zhang
Venue: NeurIPS 2025 poster ()
Keywords: Unified Vision-Language Model
Personalization
Understanding and Generation
Primary area: applications

Abstract: Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept $\langle bo\rangle$, generating "$\langle bo\rangle$ wearing its hat" without additional textual descriptions of its hat. We call this kind of generation \textit{\textbf{personalized attribute-reasoning generation}}. To address the limitation, we present UniCTokens, a novel framework that effectively integrates personalized information into a unified vision language model (VLM) for understanding and generation. UniCTokens trains a set of unified concept tokens to leverage complementary semantics, boosting two personalized tasks. Moreover, we propose a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-art results in personalized attribute-reasoning generation. Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: \href{https://github.com/arctanxarc/UniCTokens}{https://github.com/arctanxarc/UniCTokens}.

UniCTokens: Boosting Personalized Understanding and Generation via Unified Concept Tokens
Ruichuan An Sihan Yang*
Renrui Zhang Zijun Shen'
Ming Lu Gaole Dai Hao Liang' Bocheng Zou Chaoqun Yang'
Ziyu Guo Shilin Yan
Yulin Luo'
Wentao Zhang'
Peking University
Xi'an an JiaoTTong University CUHK Intel Labs, China
Nanjing University
University of Wisconsin- Wisconsin-Madison
Tsinghua University
User-provided concept.
Name: thos
boo is Shiba Inu dag with round face, fluffy cream and curly toil.
reddish-brown
-bor has red hat.
roo's resting spot soft corpet.
Mul timadol unders tertenngg
Deepen understanding from generation & Personalized attribute- reasoning generation
Text imoge generation
Cleistre coming
TE Berigher
Wes
What kind of should
Genernte photo of -us roeing
<hee??
for
fuufly Stabo
spot white carpet.
The werting Mack
Ter 0a tap ol
Daserbe image
shows Shite comdonshly.....
visbile in the photo
Figure The capability overview of UniCTokens. UniCTokens achieves personalized under- standing and generation of unified VLM using user-provided concept images and texts. This is accomplished by fine-tuning a set of unified concept tokens, which harness the mutual benefits un- derstanding and generation. Notably, UniCTokens supports complex personalized attribute-rcasoning generation, which has never been achieved by previous methods.
Abstract
Personalized models have demonstrated remarkable success in understanding and generating concepts provided by users. However, existing methods use separate concept tokens for understanding and generation, treating these tasks in isolation. This may result in limitations for generating images with complex prompts. For example, given the concept (bo). generating "(bo) wearing its hat" without addi- tional textual descriptions of its hat. We call this kind of generation personalized attribute-resonnin generation. To address the limitation, we present UniCTokens, novel framework that effectively integrates personalized information into uni- fied vision language model (VLM) for understanding and generation. UniCTokens trains set of unified concept tokens to leverage complementary semantics, boost- ing two personalized tasks. Moreover, we propose progressive training strategy with three stages: understanding warm-up, bootstrapping generation from under-
'Equal contribution. Email: arctanxarcOgaail.com
I Project Leader.
1 *Coresponning author. Email: wentao www..oaaangguu....omomm edu. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

standing, and deepening understanding from generation to enhance mutual benefits between both tasks. To quantitatively evaluate the unified VLM personalization, we present UnifyBench, the first benchmark for assessing concept understanding, concept generation, and attribute-reasoning generation. Experimental results on UnifyBench indicate that UniCTokens shows competitive performance compared to leading methods in concept understanding, concept generation, and achieving state-of-the-at results in personalized attribule-ressonnng generation, Our research demonstrates that enhanced understanding improves generation, and the generation process can yield valuable insights into understanding. Our code and dataset will be released at: httiss//github.com coaob/cco//aaaaaaaa Saaam//reaaaaa//miiii Unniaaarr//nCTTooeen
Introduction
Personalized tasks focus on understanding and generating information that users provide. Recently, there has been growing interest in personalizing understanding models, especially in transforming general-purpose models into daily life assistants [12]
2 3]. As result, significant effort has been devoted to improving the personalization capabilities of Large Language Models (LLMs) [4.5 (.66) and Vision- Laniun-L Models (VLMs) 7 8
9
5
12 In terms of generation, with the rapid 15
18 personalization techniques can generate highly realistic and diverse images based on user-specified concepts. They mostly employ paradigms such test-time fine-tuning 19 20 and retrieval augmentation (21). LLMs, VLMs, and T2I models have shown remarkable personalization performance in their respective task domains.
advancement of text-to-image (T2I) models 13
14
16
17
Despite significant advancements in personalized generation and understanding, current methods often treat these as independent tasks, failing to effectively leverage complementary semanties [22] Personalized understanding utilizes langua-l models (VLMs) [23] 24 (25,, while personalized generation primarily employ diffusion models 266 19 (7]. This leads lack of a unified personal- ized model for users to perform both understanding and generation. Meanwhile, personalized tasks are complex and conceptually diverse in reality, as shown Fig. [ For instance, when training data includes only the concept (bo), diffusion models would struggle to generate suitable images of " (bo) wearing its hat". if additional textual descriptions of its hat are provided. Additionally, concepts often include subtle visual features that critical for identifying them. Traditional under- standing models typically prioritize high-level semantic information over these subtle yet critical features 28 29], Ignoring the mutual semantics of the two tasks makes current methods insufficient for fully understanding and efficiently generating concepts provided by users.
Although a recent work Yo' Chameleon 22 achieves understanding and generation upon a unified VLM 30), it still presents several challenges in personalization:
First, Ch'hhamel utilizes distinct concept tokens for understanding and generation, whereas isolating these tasks may not fully leverage their complementary benefits.
Second, Yo Chameleon assesses personalized understanding and generation separately, without quantifying how understanding facilitates generation, referred as attribute-reasoning generation.
To this end, instead of fine-tuning distinct concept tokens like Yo' Chameleon, we propose UniCTo- kens, a personalization framework that efficiently personalizes unified VLM by fine-tuning unified concept tokens. Additionally, we utilize a progressive training strategy with three stages, mimicking the general process of human understanding of concepts. Given a new concept, we first establish a preliminary understanding of it, and then enable the ability to visualize it through drawing, thereby enhancing the comprehension of the concept. Specifically, our method begins by warming up unified concept tokens through understanding task. We then share the concept representations learned from this task in generative learning. Finally, during the generation process, we utilize intermediate results to provide detailed information for the understanding task. This progressive training strategy explicitly promotes mutual enhancement between personalized understanding and generation.
To better assess the personalization capabilities of the unified model, we introduce new benchmark, UnifyBench, aimed at evaluating models' abilities in concept understanding, concept generation, and attribue-reasonnng generation. When we assess our UniCTokens using UnifyBench, we consistently achieve competitive or better results than all other personalization methods. Our analysis indicates that better understanding enhances generation, while the generation process can also provide

---

User provided concept:
Apph 18 <bobens, ctolanee 2D Mire
Unified concept Lokens
Nome: rosh
"poy ceremic may designed a pi
featuring round eers, pirk blus cheeks,
CANMNA...
with sculpted peash.
belonge young
Peach lid top Round Ears
<roiens
Tekenieer
Autoregnostiviccno enrroyy los
Unic Tokens
Pig face
Owner
UniCTokens
aall p0000
Blush cheeks
Ceramic mug
Text Tokeniser & Image Tekenirer
TAmMN.....
photo
Mattinad cndorcadiingg....
Temherty
(l)
The overview of UniCTokens. Rather than training separate concept tokens for under- standing and generation, we train unified concept tokens that take advantage of the mutual benefits of both tasks. Linked with shared tokens, achieve cross-task transfer.
Figure 2:
information that supports understanding, providing valuable insights for the development of general unified models. We believe that UnifyBench will serve as a strong foundation for future research in unified model personalization. To sum up. our contributions can be concluded as:
We propose UniCTokens, effective framework for personalizing unified VLMs by fine-tuning unified concept tokens instead of separate tokens for understanding and generation.
We propose a progressive training strategy consisting of three stages to facilitate the transfer of information between tasks, promoting both personalized understanding and generation.
We construct UnifyBench, a benchmark to evaluate concept understanding, concept generation, and attribute-reasoning generation of personalized unified model.
We conduct extensive experiments UnifyBench. UniCTokens demonstrates competitive performance compared to leading methods in concept understanding and generation, achieving state-of-the-ard results in attribute-reasoning generation.
Related Work
Personalized Understanding and Generation Model. Model personalization mainly involves integrating concept-related information into the outputs of the model. Recent text-to-image methods depend on recontextualizing text conditions. Text inversion [26] utilizes soft prompt tuning for
special token adjustments, while Dreambooth [1] modifies the entire network weights to ensure
34 inject concept-related information through varied training strategies. Large Language Models and Vision-Language Models have also witnessed trends in personalization. (35] employs a patch-based LoRA for character, while (25) utilizes a dual-tower architecture for user-aware outputs. Furthermore, 23 24 21 achieve personalized responses in multimodal scenarios through fine-tuning or eetrievl-auugmente generation, linking user information with content in images. Yo" Chameleon [22 the first attempt unified personalized models. However, its separate training strategy limits cross-task information sharing. We train unified concept tokens to enhance information transfer between understanding and generation tasks.
subject fidelity. Additionally, (27 31
32
33
UniCTokens
propose UniCTokens, novel framework that efficiently manages personalized understanding and the generation of unified VLM, as shown in Fig. 1). In order to transfer information between understanding and generation, we propose a three-stage training strategy: (1) Personalized understand- ing warm-up (2) Bootstrap generative learning from understanding and (3) Deepen understanding of
3

---

Stage
Stage
Stage
UniCTokers
UniCTokens
UniCTokems
Training
MMU data
MMU data T21 data
MMU data
T21 data
Figure 3: Overview of Training Stages of UniCTokens.
representation from generation. This section first defines the personalization of unified VLM using unified concept tokens and details the three-stage training procedures.
3.1 Unified VLM Personalization with Unified Concept Tokens
To model the complexity of real-world personalization, we train unified concept tokens rather than using separate tokens for understanding and generation Yo' Chameleon [22]. Given target concept
C, users provide personalized inputs comprising: the name of C, a set of images {T';=- (typically
3 to 10) that exclusively depict C, and set of textual descriptions {T") containing additional attributes C' that are not visually inferable (c.g.. "C"s favorite hat is red").
The goal is to train unified concept tokens M that can: (1) concept understanding: generate per- sonalized textual responses Piex related to C' (e.g., answering "What is C' doing in the photo?"). (2) concept generation: synthesize conditional images Ping of C' under various prompts, and (3) attribute-reasoningg generation: produce personalized attribute-reasoning images Ppimg that integrate the textual information. As shown in Figure D the commonality between Object 2 and Object 3 lies in their classification as image generation tasks that necessitate the production of high-quality personalized concepts. The distinction between them is that the prompt for Object 2 contains only the personalized concept, whereas the prompt for Object additionally incorporates certain information that is only within the understanding data (c.g.. "[object Object] has red hat" in understanding data, "a photo of [object Object] wearing its hat" for attribute-reasoning generation). Failure to leverage this information would preclude the generation of the hat in an appropriate color. Our objective is to utilize this task to measure the extent of information transfer across tasks. Detailed task and metric settings can be found in the Appendix. This process can be expressed in formula follows:
Puett Pamg: P Ppimg M ((T' (TT)))
(1)
3.2 Stage-1: Personalized Understanding Warm-up
Soft prompt tuning has proven effective in integrating new concepts for both personalized under- standing and generation tasks [36]
26
Moreover, learnable prompts are often utilized as conduits for
information transfer across task 377 38 model 39 40 and modality [4 43 Based on this
paradigm, we represent the concept C' as prompt containing learnable tokens for unified models:
(sks) is (tokens_s).
(2)
where (sks) is a learnable unique identifier for this new concept and (tokens_s) is shared tokens (token,,) ... (tokensg) encode semantic attributes specific to that concept. This personalized prompt serves as the system prompt during training. After understanding task training, (tokens_s) encapsu- lates various characteristics of the concept (e.g., height, hairstyle, shape; see Fig. 2)).
To enable effective cross-task task information transfer in subsequent stages, warm-up training is es- sential. To stabilize the training process, adopt the token initialization strategy proposed in MC-LLaVA [24]. During training. instruction tuning is employed to optimize the initial tokens. The training dataset contains three types of Visual Question Answering (VQA) samples: (1) positive
recognition VQA pairs, (2) random recognition VQA pairs, and (3) conversational VQA pairs. Detailed descriptions of the data construction process can be found in MC-LLaVA (24 Notably, user-provided textual information {T" H is transformed by LLMs into a set of text-only QA pairs, which are also incorporated into training. Implementation details are provided in the Appendix.
3.3 Stage-2: Bootstrap Generative Learning from Understanding
Training solely on understanding tasks does not equip the unified model to directly generate images containing C. Existing personalized unified models (22) require substantial number of samples

---

Step
Scep 15
Step
Step
Stap
Step
Step
Seep
User provided concept:
>gg> centered
in the image.
Fine- tume
Freeze
Chustering
mug designed rpgr
pig face, featuning ears, pink cheaks, topped scuiphed peach.
<pg> <tokens SP <tokeks
<tokens
1pgh belongs young girl.
Figure Generation as Perception. The first row depicts the generation process, while the differences at different timestamps capture concept details (e.g., pig noses and cup handles).
(~1,000) to train single concept, which is impractical for real-world applications. For humans, a preliminary understanding of concepts facilitates artistic creation. The more thoroughly humans comprehend concept, the more accurately they can replicate it in their artwork. Thus, we aim to explore the potential of leveraging understanding information facilitate generation.
Directly training generation tasks on shared tokens presents a straightforward strategy. However, this direct training approach results in the model losing its general conditioning capability on C and significantly diminishes performance on understanding tasks. We posit that this discrepancy is due to variations in task distributions, which encompass specific information that cannot be directly shared. Thus, we integrate new tokens to enhance model training tailored for text-to-image (T2I). Inspired by DreamBooth 19], the prompt for Stage 2 of T2I training formalized as follows:
A photo of (tokens_s) (tokens_g) (sks).
where (tokens_g) gg) is (tokeng,) (token consisting M learnable tokens. The token initialization strategy is outlined as follows: If the concept is human-related, the features acquired from under- standing tokens inherently capture aspects of style, preferences, and overall appearance. To better maintain the concept characteristic, inspired by [4 45), a facial encoding model is employed to derive embeddings for initializing the tokens. If the concept is non-human, simply apply the same method mentioned Stage 1, due to the lower complexity of generating non-human concepts.
Leveraging priors from understanding within shared tokens, UniCTokens can generate images that effectively retain concept features with only 3 training samples and transfer extra information into generation, thereby demonstrating its efficiency and benefiting from understanding.
3.4 Stage-3: Deepen Understanding Representation from Generation
Existing literature 28 shows that understanding models focus primarily on high-level semantic infor- mation, while the generation model is more responsive to low-level features. Experimental findings indicate that the training in Stage may not effectively address tasks related to low-level character- istics, such as fine-grained recognition. After completing the generation training, model capable of producing visually similar images, which may have the potential to enhance its understanding capabilities through effective utilization of information derived from generation process.
Show-o [46] utilizes decode methodology following MaskGIT [47). removing masked tokens to derive the final image. To generate an image Ir, the process can be formularized as follows:
T
po = II (21-1111
[=1
where denotes the latent variables at step and (0 1 represents the conditional probability distribution defined by the model parameters 7. Visualization of the process shown in the top row of Fig. 4 We also present the results of image differencing between different time steps.
5

---

Experimental observations reveal that Show-o typically generates images of the target concept C in coarse-to- torsee manner, starting with subject components and gradually completing the background. Semantically meaningful subject features (e.g., human eyes) begin to emerge as early as the first g of the timesteps, while background generation generally starts around? of the diffusion process.
&
The subject is generated gradually in distinct components. This process reflects the model's assess- ment of the relative difficulty of different parts of the concept. Areas where the model has higher confidence are prioritized for earlier generation, while regions with greater uncertainty are produced later. We analyze the differences between intermediate results at various timestamps to identify areas where the model encounters more challenges. To improve the localization of regions, we utilize the k-means algorithm to determine the most valuable visual tokens:
Uy set (e- =x,), m m,
(5)
where or represents relative hard regions selected by the model and mn and ñ represent discrete time steps, where in denotes a later timestamp than ñ. These regions contain fine-grained concept information and will serve as initialization for the newly added tokens, denoted as (tokens u) (token, )... ( ). The final prompt for concept C' in understanding tasks is as follows:
(sks) is (tokens s) (tokens_uj,
(6)
During Stage 3, understanding samples are exclusively constructed from recognition VQA pairs to further strengthen the model's 's identification capability. In parallel, the T2I training data continues to update the concept identifier (sks). At this stage, the only learnable tokens are (sks) and (tokens u).
This strategy enhances the model's ability to identify and extract critical details requiring improvement during the generation process. The understanding task lacks perception of low-level concept details, which leads to complementary information provided by the generation.
4 Experiment
4.1 Experiment Setup
Implement Details. We set the number of learnable tokens as K = 16, M = 8, and N = 8 respectively. All training stages optimized using AdamW and each stage is trained for 20 epochs. The batch size is set to 4 for understanding tasks stage 1, and 1 for both stage and stage 3, as well as for T2I generation. All experiments are conducted A800 GPUs. For the backbone, we adopt Show-o5 1x512 [46] as the base model. More training details can be found in the Appendix.
Dataset. We collect Unifybench of 20 concepts: Person (10), Pets (5), Objects (5). Each concept is associated with 10-15 images for training and testing. In addition, each concept is annotated with 1-2 extra attributes that are not visually inferable from training images (c.g.. "this person owns dog"). To comprehensively evaluate both understanding and T2I capabilities, We design specific test data for each task. This includes standard understanding QA pairs and generation prompts, well as queries. Please refer to Appendix for more details on our dataset.
personalized attribute-reasonet
Our approach is primarily evaluated against four distinct categories of methods. The most direct comparison utilizes unified base model incorporating personalized text and image prompts. Show-o's 's inability to support interleaved image prompts precluded this comparison. The textual prompts are derived from the captions generated by GPT-4o for each concept and subsequently summarized to meet the required token length. Another significant baseline for comparison is the recent unified customized model, Yo' chameleon 22). We retrain the model according to the original paper, utilizing 1,000 images per concept and 7B base model. Additionally, we evaluate the current GPT-4o on our benchmark to serve as an upper bound. Lastly, we conduct comparisons between models focused solely understanding or generation. More details are provided in the Appendix.
Baselines.
Metries. For personalized attribute-reasoning image generation, we use VLMs to score the generated images (from 0 to 1) based on their consistency with the extra attributes of concepts embedded in the prompt. Metries for separate personalized understanding and generation are detailed in Appendix. Final results are obtained by averaging the scores across all concepts for each evaluation metric.

---

Personalized Understanding
Personalized Generation
Type
Methods
Model Size
Token
Training
Rec.
VQA
QA
Pure Gen.
People Gen.
PARG
Images
Weight
BLEU
GPT
BLEU
GPT
CLI--T DINO
Face-Simi
Score
CLIP-
GPT-40-TP
2008
-100
0.742
0,473
0.676
0.610
0.685
0.689
0.301
0.626
0.198
0.780
0.690
Upper
GPT-1o-1P
2008
w01,00
0.773
0.543
0.685
0,589
0.652
0.794
0,310
0,722
0.559
0.782
0.792
Bound
Real Images
0.832
0.728
0.739
0.832
Yo'LLaVA
138
16
100
0.919
0.609
0.629
0.612
0.595
MC-LLaVA
13B
16
-10
0.924
0.628
0.637
0.601
0.583
Und.
RAP.MLLM
13B
~1,000
0.910
0.616
0.616
0.712
0.722
Ouly
Owen2.5-VL TP
3B
~100
0.660
0.407
0.727
0.574
0.774
Ye'LLaVA(Ph--155)
1,38
100
0.765
0.488
0.497
0.510
0.494
Text inversion
1.0B
~0
0.630
0.217
0.569
0.371
0.070
0.628
Gen Onky
DreamBooth (SD)
1.08
~10
0.649
0.281
0.591
01436
0.071
0.650
Chumllou+TP
7B
~100
0.690
0.413
0.188
0.509
0.564
0.547
0.176
0.509
0.011
0.329
0.549
Chamelsoma1P
78
~1,000
0.493
0.498
0.335
0.523
0.160
0.409
0.006
0.499
Unifled
Model
Show-o#TP
1.38
~100
0.566
0.461
0.400
0.504
0.579
0.664
0.264
0.553
0.048
0.770
0.660
Yo Chameleon
11,000
0.764
0.507
0.697
0.224
0.698
Ours
1.38
~10
0.790
0.523
0.750
0.646
0.334
0.749
Table I: Quantitative Comparison on UnifyBench. TP Text Prompt. IP = Image Prompt. PARG
= Personalized Atrrbute-Reassonigg Generation. The best and second best are highlighted.
4.2 Our Unified Personalized Benchmark (UnifyBench)
Personalized Concept Understanding. This task requires responding to queries with concept conduct experiments on personalized recognition, Tab. T our proposed UniCTokens significantly enhances
identifiers and images. Following [23] 36 24,
VQA, and QA tasks. As demonstrated the performance of the vanilla Show-o model by an average of 8.9%, while utilizing fewer tokens. Notably, when compared with unified models with a much larger number of parameters, our model also achieves decent performance over all understanding tasks while keeping smaller training sample(~~10 10 v.s. ~ ~,,00)). Given such promising results, we envision UniCTokens as a potential next-generation paradigm for unified personalized understanding.
Personalized Concept Generation. Personalized image generation is more challenging than language generation. It requires controlling many pixels with a few tokens that contain conceptual information. As shown in Tab. [ we achieve state-of-the-art results in personalized generation across three evaluation metrics among unified models. Our method also outperforms the unified models in individual generation, producing realistic faces with conceptual features. Utilizing image prompts, GPT-4 demonstrates effectiveness, but it requires a large number of tokens. Simply adding image tokens does not guarantee improvements, as the effectiveness also depends on the model's performance, as demonstrated in Chamelcon. Fig. Sishhwcases the generated image of UniCTokens, illustrating the consistency of conditional control generation and concept-related features.
Personalized Attribute- Reasoning Generation. This task evaluates the capability of unified models to generate images with additional textual personalized knowledge (c.g.. "( sks } likes playing with yellow ball."). This task challenging because the training data for T2I does not include this information. Thus, the model must possess a capacity for cross-task information transfer to handle this challenge. As shown in Tab. T while directly utilizing text prompts appears improve the T2I scores, it decreases in CLIP-I, indicating a quality reduction of generated images. Our model achieves optimal results balancing these two aspects, effectively incorporating additional personalized knowledge into generated images while preserving the characteristics of the concept. Qualitative results in Fig. show that generated picture can precisely reflect the textual description.
4.3 Existing Personalized Understanding and Generation Benchmarks
In order to facilitate a fair comparison, we also evaluated our method on benchmarks for pure personalized understanding and generation. The results of the evaluation on Yo VLAA [36] and MC-LLaVA Datasets (24 are presented in Tab. fleft). Due to the limitations of the unified model's capabilities multi-concept tasks, only conducted tests the single-concept portion of MC-
7

---

stteestyed whyld dork
aloadd rols
rith can sereet rese lemee sargianss
of
Jackt
Figure 5: Qualitative Comparisons among UniCTokens, Yo'Chameleon and GPT-4o. Our proposed UniCTokens demonstrates its controllable and personalized generation.
YoTLaVA
MC-LLaYA
Aedde
Tiken
Training
Images
Weight
Ret
Ace
QA
Acc
WWigght
BL.UU
VQA
Methol
Media
Telom
Training
CLIP C
DeeanBeach
VrLLaVA
CLIP-1
LLaVA-TP
13R
-.5
0:19 09919
0.901
0.594
0.424
0.997
Beal
Und.
0.924
Daly
Deanflach
1.0
-3000
ME-HaaA
0992
09911
0.910
01912
0.679
0.735
Toa mandce
1.00
-110
01087
0.271
0.539
0.671
0.709
1.38
100
0.792
0.615
0.326
0.714
0,512
0.68
Unilel
Chanchonu: TP
Clasedoun1PP
78
--10
0.500
0.190
0.566
Chandoore1PP
-64
0.722
0.116
Unifad
Yo Clummias
32
-1,000
0.845
0.604
0.721
0.741
0.597
0.670
SoChorasieee
-11000
Shm--o1PP
0.691
0.991
Oknnn
1.38
-130
(1:00)
02267
11:94
Oanx
1.38
32
<110
0.852
0.615
0.738
0.734
0.630
0.679
Table 2: Performance Personalized Understanding and Generation Benchmarks, Text
Prompt. IP Image Prompt. The best and second best performances highlighted.
LLaVA. Our method outperforms the current leading unified personalized model, utilizing only 1.3 parameters and fewer training images. Notably, UniCTokens outperforms all understanding tasks with an average of 5. 5.13% when compared to an important baseline, Yo' Yo'LLaVA(1.3B), demonstrating the potential of scaling UniCToken achieve state-of-the-at performance.
We evaluate personalized generation capabilities of UniCTokens on Dreambench
19 and Yo'LLaVA Dataset [36]. Our approach significantly enhances the capabilities of vanilla Show-o. Additionally, our performance surpasses that of Yo 'o'hhmeleon, particularly on CLIP-T, which measures the model's s ability for controllable generation. This improvement be attributed prompt design and mutual enhancement between personalized tasks. Notably, we also outperform the significant generative baseline, Dreambooth, with the same training data, demonstrating that method can generate high-quality and realistic images containing user-provided concepts.
effective
4.4 Ablation Study and Analysis
Better Understanding Leads Better Generation. Through modulation of training epochs and data requirements, we developed unified models with varying understanding capacities. Models' understanding capabilities benefit from increased training epochs and data. Fig. (left) shows that as the understanding capability of the unified models improves, their generative ability also enhances. Moreover, our analysis shows that factors influencing enhanced understanding have different effects on generative performance, indicated by the varying slopes. For models with insufficient training

---

data, limitations on generative capabilities may not primarily result from the duration of training. This finding could offer crucial insights for future research on general unified models.
Understanding V.S. Gevention Capakilty
proor-proviee concept:
Peerie raie and smotion WWis tha
Equdictooe
Daoec 15%
1poch:100%
Dota 50%
Dot:
Epoch:50%
0.74
Cany Capanilyy (Sagd))) (CU10-
Epoch:50%
Nome: HMI
1000 ie the sonuffic maa
blaccc
0.73
Epoch:25%
100%
hoir and necker amile
proniemss chessed dark-
0.72
-11 owndd pellow her.
90:71
aamiil en fece
sazahino, und toomiiin
550ch:2.5%
Stage
Full imsage
Stage
0.70
51%
0.59
1.poch25%%
11000
Undenzanding Capabiirs (Stggel)) (Mocoggitoio
map
image
(1)
(2)
Figure 6: (1) Ablation study on relationship between personalized understanding and generation. (2) Visualization of different token initialization methods for stage 3 and their corresponding model outputs. With generation process as perception, UniCTokens focuses more on the concept.
Generation Process as Perception for Understanding. The process of identifying challenging regions through generation can be conceptualized as a form of perception. To investigate how this process facilitates understanding, we evaluated four main baselines: (1) Pure stage 1, without additional generative assistance; (2) Direct finetuning with the same data; (3) Utilizing the entire image for token initialization; (4) Initialization via randomly selected patches. As illustrated in Tab. (right), additional training only brings margin improvements, while general initialization methods still fall short of the performance achieved by our proposed generation as a perception process. Fig. (right) indicates that models trained with our method more likely produce detailed sentences. The attention map of UniCTokens demonstrates a greater focus on the provided concepts, reflected by higher scores, whereas utilizing the full image for initialization introduces the issue of attention dispersion, resulting in uneven distribution. These suggest that generative priors may be effectively integrated into the understanding component, resulting better understanding for concepts.
Personalized Understanding
Personalized Generution
Rec
VQA
QA
Init strutegy
Stage
Rec.
VQA
Pure Gen.
People Gen.
Weight
Score
Score
Weigh[ Soore BLEU Score
CLIP
CLIPT DINO
Pace-Simi
Score CLIIP1
Stage
0.637
0.538
0.427
Stage
0.51
0.899 0.38 0.500 0.600
0.524
0.78
0.446
0.60
0.201 0.111
No Init
0.670
0.520
0.429
Sugg wil
0616
0.479
0.438 0.527 0.581
0.695
0.243
0,582
0210
0.297 0.695
Full Image
0.723
0.529
0.423
Stage
D622
0.497
0.532 0.558 0.605
0.280
0.349
03449
0,750
Random Patch
0.681
0.520
0.421
Stage
0.790
0.50
0.933 0.544 0.603
0.750
0.982
0.646
0.334
0.334 0.740
Stage
0.790
0.523
0.431
Table 3: Performance of different training stages (left). Different initialization strategies (right).
Different Training Strategies. Our approach utilizes three-stage training strategy to facilitate information transfer across tasks. We then validate this design, beginning from generation, especially examining the inclusion of Stage Omitting Stage yields a variant of Text inversion distinguished by an increasing parameter count. Tab. 3 emphasizes the essential role of Stage 1, particularly in personalized atrrbute-reasoning generation, omitting Stage leads to significant degradation in generation quality. This further corroborates the existence of information transfer across tasks in our design. Stage does not update the generative parameters, resulting in a subtle influence on the generation task. In the context of the understanding task, we present the results of the evaluations conducted across all stages. Although performance in understanding task declines Stage 2 to facilitate generation, the subsequent injection of fine-grained visual information from Stage 3 enhances understanding capabilities. Results demonstrate that our approach achieves optimal performance, facilitating mutual enhancement between the two tasks.

---

5 Conclusion
We present UniCTokens, an innovative framework designed to personalize a unified VLM by training unified concept tokens. Our proposed three-stage training strategy enables UniCTokens to efficiently enhance personalized understanding and generation, facilitating cross-task information transfer to achieve mutual enhancement. Experimental results demonstrate that, while maintaining smaller model size and fewer training samples, UniCTokens achieves state-of of-the-art performance in concept understanding, concept generation, and from our analysis are noteworthy, shedding light on the development of unified VLMs. Furthermore, the ad vdancement of personalized AI holds significant promise for improving human lives by enabling tailored interactions, enhancing creativity, and offering solutions that align with individual needs and preferences, thereby fostering more intuitive and responsive technological ecosystem.
generation tasks. The insights derived
Acknowledgments
This work is supported by the National Key R&D Program of China (2024YF11014003), National Natural Science Foundation of China (92470121, 62402016), and High-performance Computing Platform of Peking University.
References
[1] Dhruvitkumar Talati, Ai (artificial intelligence) daily life. Authorea Preprints, 2024.
[2] Hamed Rahimi, Adil Bahaj. Mouad Abrini, Mahdi Khoramshahi, Mounir Ghogho, and Mohamed Chetouani. User-vim 360: Personalized vision language models with user-aware tuning for social human- robot interactions. arXiv preprint 2xX::2502210666 2025.
[3] Hamed Rahimi, Jeanne Cattoni, Meriem Beghili, Mouad Abrini, Mahdi Khoramshahi, Maribel Pino, and Mohamed Chetouani. Reasoning Ilms for user-aware multimodal conversational agents. arXiv preprint arXiv:2504.01700 2025.
[4] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Lconi Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprini 23xi=2303300774, 2023.
[5] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng. Yang Fan, Wenbin Gc, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 2023.
[6] Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten. Alex Vaughan. et al. The llama 3 herd of models. arXiv preprint arXiv:2407.2173 72733, 2024.
[7] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
[8] Bo Li, Yuanhan Zhang, Dong Guo, Renrui Zhang. Feng Li, Hao Zhang, Kaichen Zhang, Peiyuan Zhang, Yanwei Li, Ziwei Liu, et al. Llava-onevision: Fasy visual task transfer. arXiv preprint arXiv:2408.03322 2024.
[9] Feng Li, Renrui Zhang, Hao Zhang, Yuanhan Zhang, Bo Li. Wei Li, Zejun Ma, and Chunyuan Li. Llava-next-inte intt-intereaave: Tackling multi-image, vidco, and 3d large multimodal models, arXiv preprint arXiv:2407.07895 2024.
[10] Renrui Zhang, Jiaming Han, Chris Liu, Aojun Zhou, Pan Lu, Yu Qiao, Hongsheng Li, and Peng Gao. Llama-adapter: Efficient fine-tuning of large language models with gero-initialize attention. In ICLR 2024, 2024.
[11] Renrui Zhang, Dongzhi Jiang, Yichi Zhang, Haokun Lin, Ziyu Guo, Pengshuo Qiu, Aojun Zhou, Pan Lu, Kai-Wei Chang, Peng Gao, et Mathverse: Does your multi-modal Ilm truly see the diagrams visual math problems? ECCV 2024, 2024.
[12] Jack Hong. Shilin Yan, Jiayin Cai, Xiaolong Jiang. Yao Hu, and Weidi Xie. Worldsense: Evaluating real-world omnimodal understanding for multimodal 1lms. arXiv preprint arXiv:2502.04322 04326, 2025.
10

---

[13] Florinel-Alin Croitoru, Vlad Hondru. Radu Tudor lonescu, and Mubarak Shah. Diffusion models in vision: A survey, IEEE Trunsactions on Puttern Analysis and Machine Intelligence, 45(9): 10850-1069. 2023.
[14] Brian B Moser, Arundhati S Shanbhag, Federico Raue, Stanislav Frolov, Sebastian Palacio, and Andreas Dengel. Diffusion models, image super-resolution, and everything: A survey. IEFE Transactions on Neural Networks and Learning Systems, 2024.
[15] Ziyu Guo, Renrui Zhang, Chengzhuo Tong, Zhizheng Zhao, Peng Gao, Hongsheng Li, and Pheng-Ann Heng. Can we generate images with cot? let's verify and reinforce image generation step by step. arXiv preprint 2501. 13926, 2025.
[16] Dongzii Jiang, Ziyu Guo, Renrui Zhang, Zhuofan Zong, Hao Li, Le Zhuo, Shilin Yan, Pheng-Ana Heng. and Hongsheng Li. T2i-r1: Reinforcing image generation with collaborative semantic-level and token-level cot, arXiv preprint arxiv:2505.00777 2025.
[17] Ziyu Guo, Renrui Zhang, Xiangyang Zhu, Yiwen Tang. Xianzheng Ma, Jiaming Han, Kexin Chen, Peng Gao, Xianzhi Li, Hongsheng Li, et al. Point-bind & point-llm: Aligning point cloud with multi-modality for understanding, generation, and instruction following. arXiv preprint (rXivi3309.00611 2023.
[18] Shanglin Li, Bohan Zeng, Yutang Feng, Sicheng Gao, Xiuhui Liu, Jiaming Liu, Lin Li, Xu Tang, Yao Hu, srruuddion-
Jianzhuang Liu, and Baochang Zhang. ZONE: Zero-shot instruction guided local editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6254-6264, 2024.
[19] Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstcin, and Kfir Aberman. Dream- booth:
Fine tuning text-to-image diffusion models for subject-driven generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 22500-22510, 2023.
[20] Renrui Zhang, Zhengkai Jiang. Ziyu Guo, Shilin Yan, Junting Pan, Xianzheng Ma. Hao Dong, Peng Gao, and Hongsheng Li. Personalize segment anything model with one shot. arXiv preprint arXiv:2305.03045 2023.
[21] Haoran Hao. Jiaming Han, Changsheng Li, Yu-Feng Li, and Xiangyu Yue. Remember, retrieve and generate: Understanding infinite visual concepts as your personalized assistant. arXiv preprint arXiv:2410.13360. 2024.
[22] Thao Nguyen, Krishna Kumar Singh, Jing Shi, Trung Bui, Yong Jae Lee, and Yuheng Li, Yochamelcon: Personalized vision and language generation. arXiv preprint arXiv:2504.2099. 2025.
[23] Yuval Alaluf, Elad Richardson, Sergey Tulyakov, Kfir Aberman, and Daniel Cohen-Or. Myvim: Personal- izing vims for user-specific queries. In European Conference Computer Vision, pages 73-91. Springer, 2024.
[24] Ruichuan An, Sihan Yang, Ming Lu, Renrui Zhang, Kai Zeng, Yulin Luo,, Jiajun Cao, Hao Liang, Ying Chen, Qi She. al. Mc-llava: Multi-concept personalized vision-language model. arXiv preprint urXiv:2411.11766, 2024.
[25] Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, and Tong Zhang. Personalized visual instruction tuning. arXiv preprint arXiv:2410.07113 2024.
[26] Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit Bermano, Gal Chechik, and Daniel Cohen-Or Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.0/16 01618, 2022.
[27] Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman. and Jun Yan Zhu. Multi-concept customization of text-to-image image diffusion. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1931-1941, 2023.
[28] Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, al. Janus: Decoupling visual encoding for unified multimodal understanding and generation. arXiv preprint Aiiv:2410.1384P 2024.
[29] Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint 2001 17811, 2025.
[30] Chameleon Team. Chameleon: Mixed-mode early-fusion foundation models. arXiv preprint arXiv:2405.098 09818, 2024.
11

---

[31] Qixun Wang. Xu Bai. Haofan Wang, Zekui Qin, Authony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot identity- peeesrty-r generation in seconds. arXiv preprint arXiv:2401.075/1 2024.
[32] Hu Ye, Jun Zhang. Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models, arXiv preprint arXiv:2308.067211 2023.
[33] Bohan Zeng, Shanglin Li, Yutang Feng, Ling Yang, Hong Li, Sicheng Gao, Jiaming Liu, Conghui He, Wentao Zhang, Jianzhuang Liu, Baochang Zhang, and Shuicheng Yan. Ipdreamer: Appearance--ommrllabbe 3d object generation with complex image prompts. arXiv preprint 2rXiv:2300003377 2023.
[34] Hong Li, Yutang Feng, Song Xue, Xuhui Liu, Bohan Zeng, Shanglin Li, Boyu Liu, Jianzhuang Liu, Shumin Han, and Baochang Zhang. UV- IDM: Srentityccondiiionee latent diffusion model for face UV-texture generation. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10585- 2024.
[35] Zhaoxuan Tan, Zheyuan Liu, and Meng Jiang. Personalized pieces: Efficient personalized large language models through collaborative efforts. arXiv preprint arXiv:2406. 10471, 2024.
[36] Thao Nguyen, Haotian Liu, Yuheng Li, Mu Cai, Utkarsh Ojha, and Yong Jae Lee. Yo Volllava: Your personalized language and vision assistant. arXiv preprini arXiv:2406. 2024.
[37] Haeju Lee, Minchan Jeong, Se- Young Yun, and Kee-Eung Kim. Bayesian multi-task transfer learning for soft prompt tuning. arXiv preprint arXiv:2402.0R550 08594, 2024.
[38] Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. arXiv preprint (xXiv:2/00009000 2021.
[39] Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transfcrability of prompt tuning for natural language processing. arXiv preprint or www::iiv:i11..0. 11.01..066 1.06119, 2021.
[40] Robert Belanec, Simon Ostermann, Ivan Srba, and Maria Bielikova. Task prompt vectors: Effective initialization through multi-task soft-prompt transfer. arXiv preprint arXiv:2408.01110 2024.
[41] Zichen Wu, Hsiu-Yuan Huang, Fanyi Qu, and Yunfang Wu. Mixture-of- promtteeeesstttttttttett.s for multi-modal semantic understanding. arXiv preprint arXiv:2403.I 1131,, 2024.
[42] Juncheng Yang. Zuchao Li, Shuai Xic, Wei Yo, Shijun Li, and Bo Du Soft foot-prompting with graph-of- thought for multi-modal representation learning. arXiv preprint arXiv:2404. 04538, 2024.
[43] Zehao Xiao, Shilin Yan, Jack Hong, Jiayin Cai, Xiaolong Jiang, Yao Hu, Jiayi Shen, Qi Wang, and Coes GM Snock. Dynaprompt: Dynamic test-time prompt tuning. arXiv preprint 2501:2501.1640 2025.
[44] Lianyu Pang, Jian Yin, Haoran Xie, Qiping Wang. Qing Li, and Xudong Mao. Cross initialization for face personalization of text-to-image models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 8303, 2024.
[45] Yuxiang Wei, Yiheng Zheng, Yabo Zhang, Ming Liu, Zhilong Ji, Lei Zhang, and Wangmeng Zuo. Person- alized image generation with deep generative models: A decade survey. arXiv preprint arXiv:2502.1308.. 2025.
[46] Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang. Weihao Wang. Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. 12528, 2024.
[47] Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 11315-11325, 2022.
[48] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine tramslation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pages 31 1-318, 2002.
[49] Jiankang Deng. Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Arcface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pages 4690-4699, 2019.
[50] Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J Hewett, Mojan Javaheripi, Piero Kauffimann, et al. Phi-4 technical report. arXiv preprint arXiv:2412.08900 2024.
12

---

[51] Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianminn Zheng. Siyuan Zhuang, Yonghao Zhuang. Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90% e chatgpt quality, See https://vicua. Imsys. org (accessed 14 April 2023), 2()):, 2023.
[52] Dhruba Ghosh, Hannanch Hajishirzi, and Ludwig Schmidt. Geneval: An foccus-- framework for evaluating text-to-image alignment. Advances in Neural Information Processing Systems, 36:52132-522 52152, 2023.
[53] Xiang Yue, Yuansheng Ni, Kai Zhang, Tianyu Zheng. Ruoqi Liu, Ge Zhang. Samuel Stevens, Dongfu Jiang, Weiming Ren, Yuxuan Sun, et al. Mmmu: A massive multi-discipline multimodal understanding and reasoning henchmark for expert agi. In Proceddings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9556-9567, 2024.
[54] Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, and Ji-Rong Wen. Evaluating object hallucination in large vision-language models, arXiv preprint arXiv:2305. 10355, 2023.
[55] Yuying Ge, Yixiao Ge, Ziyun Zeng, Xintao Wang, and Ying Shan. Planting a seed of vision in large language model. arXiv preprint arXiv:2307.0800 2023.
[56] Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi- cratiiaraaulartty comprehension and generation. arXiv preprint arXiv:2404. 14396, 2024,
[57] Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, et al. Dreamllm: Synergistic multimodal comprehension and creation. arXiv preprint arXix /499, 2023.
[58] Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang. Fan Zhang, Yueze Wang. Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint wrXiv:2409.1866 2024.
[59] Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zetllemoyer and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi- model. arXiv preprint arXiv:2408. 11099, 2024.
[60] Yulin Luo, Ruichuan An, Bocheng Zou, Yiming Tang. Jiaming Liu, and Shanghang Zhang. Llm as dataset analyst: Subpopulation structure discovery with large language model. In European Conference on Computer Vision, pages 235-252. Springer, 2024.
[61] Zheng Liu, Hao Liang, Xijie Huang, Wentao Xiong, Qinhan Yu, Linzhuang Sun, Chong Chen, Conghui He, Bin Cui, and Wentao Zhang. Synthvlm: High-efficiency and high-quality synthetic data for vision language models. arXiv preprint urXiv:2407.2053 2024.
[62] Lisa Dunlap, Alyssa Umino, Han Zhang, Jiezhi Yang, Joseph E Gonzalez, and Trevor Darrell, Diversify your vision datasets with automatic diffusion-based augmentation. Advances in neural information processing systems, 36:79024-79000 2023.
[63] Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint arXiv:2412.04332. 2024.
[64] Shengbang Tong, David Fan, Jiachen Zhu, Yunyang Xiong, Xinlci Chen, Koustuv Sinha, Michael Rabbat, Yann LeCun, Saining Xie, and Zhuang Liu. Metamorph: Multimodal understanding and generation via instruction tuning. arXiv preprint ar1iv:2412. /4164, 2024.
[65] Lijie Fan, Tianhong Li, Siyang Qin, Yuanzhen Li, Chen Sun, Michael Rubinstein, Deqing Sun, Kaiming He, and Yonglong Tian. Fluid: Scaling autoregressive text-to-image generative models with continuous tokens. arXiv preprint arXiv:2410.1 102210.18800 13863, 2024.
[66] Weifeng Lin, Xinyu Wei, Renrui Zhang, Le Zhuo, Shitian Zhao, Siyuan Huang, Huan Teng. Junlin Xie, Yu Qiao, Peng Gao, and Hongsheng Li. Pixwizard: Versatile image-to-image visual assistant with open-language instructions, 2025.
[67] Rongyao Fang. Shilin Yan, Zhaoyang Huang, Jingqiu Zhou, Hao Tian, Jifeng Dai, and Hongsheng Li. Instructseq: Unifying vision tasks with cnstruction-ooniiiiionn multi-modal sequence generation. arXiv preprint arXiv:2311.1883 2023.
13

---

[68] Xichen Pan. Satya Narayan Shukla, Aashu Singh. Zhuokai Zhao, Shiok Kumar Mishra, Jialiang Wang. Zhiyang Xu, Jiuhai Chen, Kunpeng Li, Felix Juefei-Xu, et al. Transfer between modalities with metaqueries. arXiv preprint arXiv:2504.00 06256, 2025.
[69] Daya Guo, Dejian Yang. Haowei Zhang, Junxiao Song. Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang. Xiao Bi, et al. Deepseek-rl: Incentivizing reasoning capability llms via reinforcement learning. arXiv preprint arXiv:2501. 129488 2025.
[70] Renrui Zhang. Xinyu Wei, Dongzhi Jiang, Ziyu Guo, Shicheng Li, Yichi Zhang, Chengzhuo Tong, Jiaming Liu, Aojun Zhou, Bin Wei, Shanghang Zhang, Peng Gao, Chunyuan Li, and Hongsheng Li. Mavis: Mathematical visual instruction tuning with an automatic data engine, 2024.
[71] Junxian Li, Di Zhang, Xunzhi Wang. Zeying Hao, Jingdi Lei, Qian Tan, Cai Zhou, Wei Liu, Yaotian Yang. Xinrui Xiong, et al. Chemvim: Exploring the power of multimodal large language models in chemistry area. In Proceedings of the AAAI Conference Artificial Intelligence, volume 39, pages 415-423, 2025.
[72] Di Zhang, Jingdi Lei, Junxian Li. Xunzhi Wang, Yujie Liu. Zonglin Yang, Jiatong Li, Weida Wang. Suorong Yang. Jianho Wu, et al. Critic-v: Vlm crities help catch vim errors in multimodal reasoning. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 9050-9061, 2025.
[73] Niklas Muennighoff, Zitong Yang, Weijia Shi, Xiang Lisa Li, Li Fei-Fei, Hannanch Hajishirzi, Luke Zettlemoyer, Percy Liang, Emmanuel Candès, and Tatsunori Hashimoto. sl: Simple test-time scaling. arXiv preprint arXiv:2501. 19393, 2025.
[74] Chengzu Li, Wenshan Wu, Huanyu Zhang, Yan Xia, Shaoguang Mao, Li Dong, Ivan Vulié, and Furu Wei. Imagine while reasoning in space: Multimodal visualization-o--thie of-thought. arXiv preprint arXiv:2501.0754 2025.
[75] Huanyu Zhang, Chengzu Li, Wenshan Wu, Shaoguang Mao, Yifan Zhang, Haochen Tian, Ivan Vulić, Zhang Zhang, Liang Wang, Tieniu Tan, et al. Scaling and beyond: Advancing spatial reasoning in milms requires new recipes. arXiv preprint ar 20::::5504. 2025.
[76] Yizhen Zhang, Yang Ding, Shuoshuo Zhang, Xinchen Zhang, Haoling Li, Zhong zhi Li, Peijie Wang, Jie Wu, Lei Ji, Yelong Shen, Yujiu Yang, and Yeyun Gong. Perl: Permutation-enhanceed reinforcement learning for interleaved vision-language reasoning, 2025.
177] Shuoshuo Zhang, Zijian Li, Yizhen Zhang, Jinginn Fu, Lei Song, Jiang Bian, Jun Zhang, Yujiu Yang, and Rui Wang. Pixelcraft: A multi-agent system for high-fidelity visual reasoning on structured images, 2025.
[78] Haozhe Wang, Chao Qu, Zuming Huang, Wei Chu, Fangzhen Lin, and Wenhu Chen. Vl-rethinker: Incentivizing self-reflection of vision-language models with reinforcement lcarning. arXiv preprint arXiv:2504.00 088377 2025.
[79] Haozhe Wang, Qixin Xu, Che Liu, Junhong Wu, Fangzhen Lin, and Wenhu Chen. Emergent hierarchical reasoning in Ilms through reinforcement learning. arXiv preprint arXiv:2509.03644, 2025.
[80] Weiye Xu, Jiahao Wang, Weiyun Wang, Zhe Chen, Wengang Zhou, Aijun Yang, Lewei Lu, Houqiang Li, Xiaohua Wang, Xizhou Zhu, et al. Visulogie: benchmark for evaluating visual reasoning multi-modal large language models. arXiv preprint arXiv:2504.1522 15279, 2025.
[81] Yanshu Li, Tian Yun, Jianjiang Yang, Pinyuan Feng, Jinfa Huang, and Ruixiang Tang. Taco: Enhane- ing multimodal in-context learning via task mapping-guide sequence configuration. arXiv preprint wrXiv:2505.17090 2025.
[82] Yanshu Li, Yi Cao, Hongyang He, Qisen Cheng, Xiang Fu, Xi Xiao, Tianyang Wang, and Ruixiang Tang. MIIV: Towards efficient and fine-grained multimodal in-context learning via representation engineering. In Second Conference Language Modeling, 2025.
[83] Xinyu Wei, Jinrui Zhang, Zeqing Wang, Hongyang Wei, Zhen Guo, and Lei Zhang. Tiif-bench: How does your 12i model follow your instructions? arXiv preprint urXiv:2506.02 02666, 2025.
[84] YiFan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wa, Feng Li, Kun Wang. Qingsong Wen, Zhang Zhang, et al. Mme-realworid: Could your multimodal I1m challenge high- resolution real- world scenarios that are difficult for humans? In The Thirteenth International Conference on Learning Representations, 2025.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Our contributions are clearly stated in the introduction section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We have discussed the limitations in appendix H
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
15

---

Justification:
Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We have included the implementation detailed and important hyperparameters in appendix A
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
16

---

Answer: [NA]
We will publish our code and relevant dataset on GitHub after the work is published.
Justification:
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips... cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to to reproduce the results. See the NeurIPS code and data submission guidelines (https
ssicnyolicy) for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer:
[Yes]
Justification: All details are provided in the Appendix A
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: evaluation.
We run each quantitative experiments multiple times to own a robustness
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
17

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have included the implementation details in experimental setup in main paper 4.1.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't It make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: We have reviewed and have confirmed that our work aligned with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: Our work has no negative societal impacts, while we discuss the potential positive impacts that our method will bring Appendix
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
18

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: Our work doesn't pose such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g.. code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We use some open-source models as our backbone and evaluate our model on dataset released by community obeying their license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the Datasets.paaeesstt
package should be provided. For popular datasets, paperswi thcode. cow/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
19

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
New dataset well-documented and will be open sourced after the paper accepted.
Justification:
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Question:
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: Our paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
20

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: The LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standarr components.
Please refer to our LLM policy (https: //euriip// wwweerripp..ccooofefeeeeeeee cc/Confferences/22022 for what should or should be described.
21

---

A Additional Implement Detail
Loss. We use standard autoregressive loss based on masked language modeling. Given a training instance (X Xa) Xa)-where where Xg denotes the question and X2 the answer- we apply the standard masked language modeling loss to compute the likelihood Xa:
L(Xa xx. 0)
Σ log P(Xa, Xa,ct,))
Here, is the length of the answer Xax Xa.: denotes the t-th token, and P(X..1 Xg. X Xa,ct, is the probability of predicting Xa,t given the image I, question Xę, and preceding tokens Xa,<t.
Stage Configuration. We summarize the full three-stage training setup in Tab, 4 Unless otherwise stated, before Stage 3 we run k-means with = using cosine distance (d( u, v) = each selection step, we retain the cluster containing the largest number of patches.
" [ullii ). At
Stage
Stage
Stage
Positive recognition VQA pairs Random recognition VQA pairs
Positive recognition VQA pairs Random recognition VQA pairs
Positive recognition VQA pairs Random recognition VQA pairs T21 data
Training Data
Conversational (VJQA pairs
Conversational (V)QA pairs
T21 data
Trainable Tokens
(sks); (tukens,)
(sks); (lokens.); ) (tokens,)
(sks); (tokens,)
Batch Size T2T
N/A 4
Batch Size MMU
Learning Rate
1 1x10
1x 10-3
1x10- Adam
Epoch Optimizer
20
AdamW
Adamm
Table Multi-stage training configuration.
The MMU training data used for Stage 2 is essential to maintaining the model's conversational abilities. The presence or absence of these data has minimal impact on model's generative capabilities.
B Additional Experiment Setup
Metries. For understanding, we evaluate the model on three tasks: personalized recognition, VQA, and QA. In the Recognition task, we compute the recall for both positive and negative samples and report their arithmetic mean. For VQA and QA, we evaluate the predicted answers using BLEU [48 scores against ground truth, and further apply LLM- evaluation that scores responses from 0 to based on alignment with key points in the reference answers. For general personalized image generation, adopt prompts from the DreamBooth [19] dataset, and evaluate image quality using CLIP-based metrics: CLIP-I (image-to-image similarity) and CLIP-T (image-to-text similarity). Since half of concepts are human subjects, we additionally employ the off-the-shelf AreFace model 49 to measure facial similarity between generated and reference images. For the evaluated GPT-4o and GPT-4o used for scoring, we utilize different versions of them make fair comparisons.
Comparing Baselines.
We supplement the baselines not described in the main text:
LLaVA+Prompt:
We first prompt LLaVA to generate captions for all training images
concept, then prompt LLaVA summarize the captions into concise, personalized description. During inference, we add relevant captions to the input to supply concept-specific information.
Vo 36):
One of the earliest work to explore VLM personalization. Following the original paper, manually construct hard negative datasets and train Yo" 'LaVA with different sizes of base model(Phi-1.5 [50] 1.3B and Vicuna [51), 13B) to make a fair comparison.
MC-LLaVA 24): model designed for enhancing multi-concept personalization tasks. Utilizing dual textual and visual prompts, it serves strong baseline for personalized understanding.
22

---

We utilize the RAP-LLaVA model and follow the RAP-MLLM approach to construct a personalized database for each concept. To obtain the capability of understanding personalization,, RAP-MLLM conducted a post-training on dataset of 260K in size.
RAP-MLLM (2]::
DreamBooth enhances the personalization capability of generative models by allowing users to fine-tune models with limited number of images, thereby producing highly specific and context-awand outputs. For fair comparison, we utilize different number of training data (10, 3,000) to better evaluate Dreambooth.
Dreambooth
Text Inversion 26] Text inversion a technique that transforms textual prompts into corre- sponding visual representations, enabling the generation of images based on description.
C Catastrophe Forgetting
When a model acquires new knowledge, there exists the risk of catastrophic forgetting of previously learned information. Following Yo' fo'Chameleon [22], we evaluate the personalized model on several benchmarks assessing general capabilities. We conducted a comparative analysis against the original Show-o across well-established multimodal benchmarks: GenEval [52], MMMU [53], and POPE (54). The experimental results shown in Fig. | iindicate that, despite training through three-stage process and having task-specific tokens, model performance does not diminish after each stage, effectively preserving the model's 's general capabilities.
GPT-40
Show-o
Ours
Masa Lisa the Van Gogh.
Mona Lisa in the atvie of Goph.
<AB> the style of Gioch.
GenEval
MMMU
POPE
Original
0.68
26.7
80.0
Stage
0.67
26.6
80.0
Stage 2
0.66
26.6
80.0
Stage 3
0.66
26.6
80.0
Figure 7: Catastrophic Forgetting Evaluation. UniCTokens maintains performance similar to vanilla Show-o.
Figure 8: Limitations. Example images under
different style controls.
D Additional Related Work
Unified Understanding and Generation. A multitude of efforts have been dedicated to employing
single model for both understanding and generation. SEED [5 adapted image representation
through discrete tokenization for language modeling, leveraging autoregressive conditioning for
generation via an external decoder. 156] [5] restructured conditioning information aggregation but still relied on extra modules for generation. Chemeleon (30] is an early hybrid unified model capable of generating and understanding intert-leaved text-image content. Janus 28) claims that understanding and generation require distinct information, employing different tokenizers for each task. Emu3 [58) converts images, text, and video into discrete tokens, enabling joint training of Transformer on multimodal sequences. [46, [59] utilize a hybrid of autoregressive and diffusion methods for text and image processing. The above-mentioned work all focuses general tasks, neglecting exploration in personalization scenarios. In this paper, we employ Show-o [46] as the backbone to efficiently achieve unified personalization without forgetting the general capability.
Bridging Understanding and Generation. The establishment of unified models seeks to optimize the strengths of understanding and generation, allowing information transfer between tasks and mutual improvement. Early efforts to link these tasks mainly utilized a serial processing paradigm. [60, 61] (62 employed image captioning models to generate textual conditions for text-to-image models. While these methods highlight the potential of leveraging understanding for generation, they lack end-to-end optimization and frequently suffer from information loss. [63
64
65
66
67
explored deeper integration of understanding and generation. MetaQuery (68] utilizes learnable queries on frozen VLMs to extract conditions for generation, but it primarily emphasizes how
23

---

understanding aids generation while neglecting the inverse. In this paper, we propose three- stage training strategy that achieves information transfer and mutual enhancement between tasks in personalization scenarios, providing insights for the broader development of unified models.
Reasoning in Understanding and Generation. The success of Deepseek- R1
669 promotes the
exploration of reasoning. The recent work focuses designing reward 71
72 constructing
valuable samples 74 and building reasonable reinforcement learning a algorithm [76
77 78
fully unleash the potential of LLMMs [79] and VLMS [0 81
82]. Many works 83
84 tend
to transfer the reasoning capability into image generation process. 15 analysis the effectiveness
of Coo) in generation, while [16) utilizing semantic- and token-level CoT to
handle complex reasoning scenarios. Inspired by there work, we propose complex personalized attribute-ressoning generation, which better modeling the real world user query.
Additional Qualitative Results
<hutlaa>>
ploto <hadle>
A plonto of <herta the jungle
A.hhett of zhartics notiag
phats abh
Acalue with airee and astuenn tadkgronnd
plots of <alan Settak
<bater
Sahai deaemeed wis Eaneers and bauzs daning Rottooll
mocketed lib.
<butars' lurne
onry wonder catnn
Aphota
of
Aphata of
-EM- oftn wezts
LL Bequesily wear casii black bow te formal evere. enjoyo scacide vauntions.
alegent back dreses formed evente hao eaagical
salall-
A-llay vakadi
<daat Uhsbee dhilee
Aphata
Andiigher vatiit
ploto magiolns-conn coners
potterted drosed in blak
Aphata anyliep
plate of Stydice>
A photo expdie- nearleg coler
platou
Aphata of Sancnee ucaring pisk plancs
A plone of Cesanie galgg
"myyhiu alwaye wain pink collarin the hosso.
SHaRHE bb whet going <marees res ona sofo indrons
pink
anydlessts home cazy wooder hoonz deep julle Eeps
Figure 9: More qualitative results generated from our model.
F Additional Quantitative Experiments
Learnable Token Length. We systematically varied the quantity of learnable tokens, as illustrated TO As the number of learnable tokens increases, the model's performance personalized understanding and generation improves. This improvement is attributed to the increased parameter count providing more learning capacity. However, simply increasing the number of parameters not always beneficial. When the parameter count becomes excessive (e.g., 64 tokens), the CLIP-T score declines, which may be due to excessively long contexts that make conditioning more difficult. The improvement rates vary across different tasks, reflecting a shift in task domains.
Cost of Adding New Concept. Tab. reports per-concept FLOPPs and wall-clock time un- der unified setup. Methods that personalize with large per-concept image sets cetss-xeepllfiie by You Chamelcon (about 1,100 images) incur the highest cost (7x 10 FLOPs: 120 min). whereas

---

Effoctiveness Token Length
0.800
Und.
Gen.
Learnable Token
0.775
Rec.
Pure Gen.
Length
Weight
CLIP-I
CLIP-T
0.725
(only <sks>)
0.608
0.677
0.271
Meric
(1+0+0)
0.641
0.680
0.274
0.700
(2+1+1)
0.682
0.688
0.275
0.675
8 (4+2+2)
0.727
0.707
0.277
16 (8+4+4)
0.754
0.731
0.279
0.650
32 (16+8+8)
0.790
0.750
0.282
0.625
Recognition Acc.
64 (32+16+16)
0.793
0.763
0.271
T21 CLIP-I
0.600
Figure 10: Ablations amount of tokens.
Token Length
generator-only tuning (DreamBooth) is far cheaper (2x10'8 FLOPs; 7 min) but limited in scope. Among unified models, UniCTokens achieves favorable cost-capability trade-off: 1.3x10' FLOPs and 25 min per concept, yielding lower time than Yo" Chamelcon.
Type
FLOPs/ Concept
Time Concept
Yo'LLaVA-133B
1.5 x 10
30 min
Yo'LLaVA-Phi
1.6 x 10t6
10 min
DreamBooth (SD)
x 10
7 min
Yo Chameleon
7 x 10"7
120 min
UniCTokens (ours)
10a7
25 min
Table 5: Per-concept training cost.
Training Scheme: Three-Stage vs. Joint. Tab. coontrasts single-stage joint baseline with our three-stage schedule. The staged schedule delivers consistent gains personalized understanding and generation the largest improvements on personalized attribute-reasoning generation. These results indicate that staging is redundant: an understanding warm-up first stabilizes concept bindings; generation bootstrapped from these bindings improves conditioning quality; the final stage feeds generation signals back understanding. Consequently, cross-task transfer strengthened without increasing model capacity.
Rec.
VQA
QA
Pure Gen,
People Gen. Face-Sim.
PARG
Method
Weight
GPT
GPT
CLIP-I
CLIP-T
Score
CL.IP-I
Joint Training
0.709
0.489
0.565
0.681
0.258
0.288 0.334
0.269
0.678
3 3-Stage (Ours) 0.790
0.523
0.603
0.750
0.282
0.359
0.749
Table Joint vs. three-stage. PARG = Personalized Attribute- Reabbuuinn Generation.
Judge Models and Human Alignment. Beyond GPT-based scoring and the classical metrics reported in the main paper, we additionally evaluate with Gemini-2.5-PPr and human study on 300 samples (five per concept per task). As shown in Tab. [ the relative ordering of methods is consistent between Gemini and human judges across VQA, QA, and personalized attribute- reasoiing generation: UniCTokens ranks highest where reported. These trends support using LLM judges as practical proxy while remaining aligned with human preferences.
G Details of Dataset
The dataset's data sources comprise animals and objects obtained from MC-LLaVA 24],
Yo'LLaVA (36 and MyVLM 233, with images of individuals sourced from Yo'LLaVA and various online platforms. images and generated training data have been subjected to rigorous human validation processes. We present a set of images along with extra textual information in Fig.
25

---

Gemini-2.5-Pro
Human Study
Methods
VQA
QA
PARG
Methods
VQA
QA
PARG
Yo'LLaVA-Phi DreamBooth (SD)
0.492
0,498
Vo
0.679
0.622
0.066
DreamBooth (SD)
0.012
Chameleon
0.489
0.495
0.279
Yo Chammeeeon
0.670
0.623
0.272
UniCTokens (ours)
0.527
0.601
0.371
UniCTokens (ours)
0.700
0.683
0,352
Table 7: LLM and human scores on bench. PARG Personalized Attribute- Generation.
H Limitation and Discussion
While our method demonstrates notable strengths, is not without limitations. The first limitation is that, similar to other personalized models, our approach is constrained by the inherent limitations of the base model. Show-o struggles to generate outputs in varying styles, as illustrated in Fig. g Consequently, our method inherits the limitations of its underlying model. The second limitation pertains to the model's inability to effectively handle domain shift inputs, particularly in specialized fields such medicine. However, this challenge presents opportunity for improvement, as the development unified models could enhance gencralization and address this issue in future work. Finally, although have attained state-of of-the-art results (e.g., achieving facial similarity of 0.334) in personalized individual generation compared with other unified models, there remains gap when comes personalizing human faces. For reference, the recommended threshold for facial recognition similarity is around 0.4-0.5, representing a significant avenue for further research.
I Broader Impacts
The development of UniCTokens and the accompanying UnifyBench benchmark holds significant potential for advancing the field of personalized AI. By effectively integrating user- sser-provided con- cepts into unified vision language model, our approach not only enhances the performance of personalized understanding and generation tasks but also opens up new avenues for applications across various domains, such as education, healthcare, and creative industries. The ability to generate contextually relevant images based on minimal prompts can greatly benefit creative professionals by streamlining content creation processes. Additionally, our research emphasizes the importance of mutual reinforcement between understanding and generation, which could lead to more intuitive human-Al interactions. As we release our code and dataset, we aim to foster further research in this area, encouraging the development of more sophisticated models that can better understand and respond to user needs while ensuring ethical considerations in Al deployment.
26

---

Animal
<bo>
<mam>
<maeve>
<bo> a Shiba dog
<macve> is a small white dog flufly fur, bushy and distinctive dark markings around the eyes and cars.
<mam? is British Shortbair cal
round flufly cream
with silver labby für, large round
reddish-brown für, and a curly lail.
amber eyes, and stocky build.
<mam? has red hat.
red hat. <boo's resting on soft
"maeve" wears pink going out.
<mamm's 's home is cozy wooden
cabin.
white carpet.
<maeve'> rests on sofa indoors.
Are Person
<>>
<B>
<W>
<N> is young woman with long
<B> is man with a close- cropped beard, medium-dark skin, and clean fade haireut.
<W> is a well-groomed man with neatly styled black hair, fair skin, and defined jawline.
black hair, porcelain skin, and delicate facial fentures often highlighted by stylish makeup.
<B> owns dog his pet. <B> has a pair of diamond stud carrings.
<W' home is by the sea.
<N> likes to wear white atbletic
<W> wears subtle lapel pins as his jackets.
short-alecved shirts.
<WN>
WL
<WN> is cheerful young man with short wavy brown and fair skin.
<W
professional
well-groamed beard, short
player known for her athletic build and signature braidod hairstyle.
black hair, and radiant smile.
frequently appears at premieres dressed in dark-colored suits.
<WN> enjoys eating street food
<C> often purple tank top on the court.
while easually dressod.
<WN> has blue T-shim.
a yellow
owns pearl nocklace.
Object
<pg>
<nha>
<nhar is Gothic Revival
"pgo> ceramic mag designed
circular red, white, and blue shield with metallic finish and the center.
cathedma with hell towers,
like a pig's face, featuring round ears, pink blush cheeks, topped with sculpted peach. "pg> belongs to young girl.
pointed arches, and a large window on the facade.
lid
appears rusty when exposed to ruin.
<nha> decorated with flowers and banners during festivals.
Figure 11: UnifyBench Dataset. Example images for partial concept in constructed dataset.
27


Reviews:
[{"review_id": "dBBQiGzhtb", "reviewer": "Reviewer_mWTp", "summary": "This paper introduces UniCTokens, a novel framework for personalizing a unified VLM to handle both understanding and generation tasks for user-provided concepts. The key idea is to move away from separate concept tokens for each task and instead train a single, unified set of concept tokens. A core contribution is a three-stage progressive training strategy. To evaluate their method, the authors also introduce UnifyBench, a new benchmark designed to assess personalized understanding, generation, and a new task they define as \"knowledge-driven generation,\" where the model must generate images combining visual concepts with attributes described only in text.", "questions": "I'm mostly curious about the complexity and scalability. Is the three-stage process truly necessary, or could a simpler joint-training scheme work? Meanwhile, beside 1.3B model, have you tried on 7B Show-o?", "limitations": "Yes.", "rating": 5, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "This paper presents a novel approach to personalization in unified VLMs by using a single set of concept tokens for both understanding and generation. This contrasts with prior work like Yo’Chameleon and enables more effective cross-task transfer, particularly for the “knowledge-driven generation” task. The three-stage training strategy is technically sound and the feedback loop introduced in Stage 3 is especially insightful.\n\nExperiments are thorough, with strong baselines and a new benchmark (UnifyBench) supporting the claims. Ablations clearly show the benefit of each component.\n\nHowever, the framework adds complexity by introducing multiple token types and training phases, and it's unclear if a simpler setup could achieve similar results. The model scale is relatively small (1.3B), so it's uncertain how well the approach generalizes. Also, evaluation relies heavily on GPT-4o, which may introduce bias.", "was_revised": true, "final_justification": "The authors addressed my concerns during the rebuttal. I believe that incorporating the experiments presented in the rebuttal into the final version will strengthen the paper.", "rebuttal": "## Response to Official Review by Reviewer mWTp: \n**Rebuttal:**\n\nWe sincerely appreciate your recognition of our paper and your valuable comments. We are encouraged by the recognition that our method is novel and insightful. After thoroughly reading the comments, we provide the following point-by-point responses：\n\n> W1 & Q1: It's unclear if a simpler setup could achieve similar results; Is the three-stage process truly necessary, or could a simpler joint-training scheme work?\n\nTo further elucidate the necessity of the three-stage strategy, we conducted additional exploratory experiments. Our experimental setup is as follows: To ensure a fair comparison, we maintained the same total number of learnable parameters and training data as in the UniCTokens, but directly executed single-stage joint training. For your convenience in comparing, we present the results of the single-stage approach alongside those of UniCTokens as follows:\n\n|   Task Type     |     Rec   |   VQA   |   QA    | Pure Gen | Pure Gen | People Gen | Knowledge-driven | Knowledge-driven |\n|:---------------|:-------:|:-------:|:-------:|:--------:|:--------:|:----------:|:----------------:|:----------------:|\n|     Metric      | Weight  |  GPT    |  GPT    |  CLIP-I  |  CLIP-T  | Face-Simi  |      Score       |      CLIP-I      |\n| Joint Training  | 0.709   | 0.489   | 0.565   |  0.681   |  0.258   |   0.288    |      0.269       |      0.678       |\n| 3-Stage(Ours)   | 0.790   | 0.523   | 0.603   |  0.750   |  0.282   |   0.334    |      0.359       |      0.749       |\n\nAs can be observed, our UniCTokens outperforms joint training across personalized understanding and generation tasks, especially on knowledge-driven generation. We believe that it is our three-stage training that facilitates better cross-task information transfer, and this paradigm is superior to direct joint training.\n\n> W2 & Q2: It's uncertain how well the approach generalizes; Meanwhile, beside 1.3B model, have you tried on 7B Show-o?\n\nThank you for your insightful question and suggestion. First, we would like to clarify that when we completed this work, the Show-o2-7B model had not yet been released (2025.06.19). Second, to validate the generalizability of our method, we supplement the results of UniCTokens on Show-o2-7B with the learnable parameters and training data remained consistent with Show-o in the main text:\n\n|     Task Type     |   Rec  |       VQA      |       QA      | Pure Gen | Pure Gen | People Gen | Knowledge-driven | Knowledge-driven |\n|:-----------------|:-------:|:--------------:|:-------------:|:------------------:|:---------------:|:----------------:|:----------------:|:----------------:|\n|      Metric       | Weight  |      GPT       |     GPT       |   CLIP-I  | CLIP-T |   Face-Simi     |      Score       |      CLIP-I      |\n| UniCTokens(Show-o2-7B)    | 0.857   |     0.595      |    0.679      |   0.799   | 0.297  |     0.381       |      0.372       |      0.801       |\n| UniCTokens(Show-o)  | 0.790   |     0.523      |    0.603      |   0.750   | 0.282  |     0.334       |      0.359       |      0.749       |\n\nWhen the model is scaled, the performance on the task is observed to improve correspondingly. Our approach is designed to be compatible with the base model; it is evident that the improvement of the base model's capability leads to a significant enhancement in the results produced by our method, thereby demonstrating the generalizability of our approach across models of diverse scales.\n\n> W3: Also, evaluation relies heavily on GPT-4o, which may introduce bias.\n\nThank you for pointing this out. We would like to clarify two key points regarding our evaluation methodology:\n\n1. **More Metrics Utilized:** We employed not only GPT-4o for evaluation in our main text but also utilized several classical metrics for understanding and generation (e.g., BLEU, CLIP-I). GPT scoring also increasingly becomes a widely accepted evaluation method in both general[1] and personalization[2] scenarios. However, to further mitigate potential biases in evaluation, we employed Gemini-2.5-Pro as a judge for the extra evaluation on the Unify Bench. \n2. **Human Study:** To ensure that the evaluation results of large models align with human preferences, we also sampled 300 results from the test set (each concept has 5 samples for each task) and recruited professionals from technology companies and college students to conduct a detailed human study. The scoring range for human graders is consistent with that of large language models. We compare against several baselines:\n\n|    Methods      | Training images |   VQA (Gemini)   |   VQA (Human)   |   QA (Gemini)   |   QA (Human)   | Knowledge-driven (Gemini) | Knowledge-driven (Human) |\n|:---------------|:--------------:|:----------------:|:---------------:|:---------------:|:--------------:|:-------------------------------:|:------------------------:|\n| Yo'LLaVA-Phi     |     ~100       |      0.492       |      0.679      |      0.498      |     0.622      |                -                |            -             |\n| Dreambooth(SD)  |     ~10        |        -         |        -        |        -        |       -        |              0.066              |         0.012            |\n| Yo'Chameleon    |     ~1000      |      0.489       |      0.670      |      0.495      |     0.623      |              0.279              |         0.272 \n| UniCTokens      |     ~10        |      0.527       |      0.700      |      0.601      |     0.683      |              0.371              |         0.352            |\n\nIn comparison with Table 1 in the main text, the trends of Gemini and GPT scores are consistent and are generally aligned with the results from the human study. We will add extra metrics in the revised version to ensure unbiased results.\n\nConsidering the limited time during the rebuttal period, we will include more comparisons on understanding-only and generation-only benchmarks of the above mentioned experiments in the revised version.\n\n[1] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena\n\n[2] MC-LLaVA: Multi-Concept Personalized Vision-Language Model"}, {"review_id": "p8RyLZ8PKz", "reviewer": "Reviewer_Sy58", "summary": "The paper proposes UniCTokens, a framework integrating personalized information into a unified vision language model (VLM) for understanding and generation. The paper proposes a progressive training strategy with three stages: understanding warm-up, bootstrapping generation from understanding, and deepening understanding from generation to enhance mutual benefits between these tasks.  Moreover, it presents a benchmark UnifyBench to evaluate the unified VLM personalization.", "questions": "L44, mutual information, what does it mean? \nThe current evalution on several tasks, such VQA, how about other complex VL tasks, such as visual dialogue.", "limitations": "K-means is adopt for clustering. So some ad-hoc parameter is required. \nThe training of the model has three stages. So the reproducibility is a little in doubt.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Pros\n1. The unified concept tokens proposed in the paper enable learnable tokens to be simultaneously applied to both understanding and generation tasks, breaking the limitations of tasks' isolation.\n2. The three-stage training method progressively enhances the model's understanding and generation capabilities while enabling mutual improvement between them.\n3. The paper presents the first personalized dataset, which covers three categories: humans, pets, and objects.\n\nCons\n1. Regarding Section 1.3.1 describing the training objectives of unified concept tokens M, provide more detailed differentiation between Objective 2 and Objective 3 to better highlight their differences and implementation challenges.\n2. For image generation evaluation, introducing more challenging tests with noisy backgrounds (e.g., generating the target concept <bo> alongside similar category instances) would provide a more rigorous assessment of the model's concept retention and robustness in complex scenarios.", "was_revised": true, "final_justification": "Some of my concerns have been addressed. I would like keep my score.", "rebuttal": "## Response to Official Review by Reviewer Sy58: \n**Rebuttal:**\n\nThanks for your thoughtful response and the time you have invested in reviewing our paper. After carefully reviewing the comments, we present the following responses in a point-by-point format:\n\n> C1: Provide more detailed differentiation between Objective 2 and Objective 3. \n\nWe sincerely apologize for any potential ambiguities. As shown in Figure 1, the commonality between Object 2 and Object 3 lies in their classification as image generation tasks that necessitate the production of high-quality personalized concepts. The distinction between them is that the prompt for Object 2 contains only the personalized concept, whereas the prompt for Object 3 additionally incorporates certain information that is only within the understanding data (e.g., \"<bo> has a red hat\" in understanding data, \"a photo of <bo> wearing its hat\" for knowledge-driven generation). Failure to leverage this information would preclude the generation of the hat in an appropriate color. Our objective is to utilize this task to measure the extent of information transfer across tasks. We will clarify this point in the revised version to avoid any misunderstandings.\n\n> C2: Introduce more challenging tests with noisy backgrounds. \n\nThank you for your suggestion. This essentially represents a special case of multiple concept generation. In fact, personalization tasks often focus more on the consistency between the generated subject and the reference object. Meanwhile, there are currently no established benchmarks available for evaluation. We appreciate your advice and have manually constructed prompt templates such as \"a photo of <sks> and a <cls>\" for evaluating on Unified Bench. Here, the <cls> refers to the same category as <sks>. For example, we want to generate a Tom cat, the prompt will tend to be like \"a photo of <Tom> and a cat\". The results are as follows:\n\n| Method| Training images | Personalized Generation(CLIP-I) | Personalized Generation(CLIP-T) |\n|:-----|:---------------:|:-------------------------------:|:-------------------------------:|\n| Dreambooth(SD) | ~10 | 0.627 | 0.234 |\n| Textinv | ~10 | 0.600 | 0.222 |\n| Show-o+TP | - | 0.659 | 0.240 |\n| Yo'Chameleon | ~1000 | 0.691 | 0.207 |\n| UniCTokens | ~10 | 0.723 | 0.243 |\n\nIt can be observed from the table that under such challenging and noisy background conditions, the performance of personalized generation is somewhat weakened. Nevertheless, our UniCTokens still achieved SOTA performance, demonstrating the effectiveness and robustness of our approach.\n\n> Q1: L44, mutual information, what does it mean?\n\nWe apologize for any potential misunderstandings that the paper may have caused. Mutual information refers to information that can be mutually utilized across tasks. This is primarily manifested through the pre-training of understanding models, which leverage the implicit priors encoded in the model parameters to assist in generation, as well as utilizing intermediate results from the generation process to explicitly aid in fine-grained understanding. The specific results of this mutual enhancement are quantitatively presented in Table 3 of the main text. To express this more clearly, we will revise it to incorporate the term \"mutual semantics\". We welcome any suggestions you may have regarding the naming.\n\n> Q2: How about other complex VL tasks, such as visual dialogue.\n\nThank you for your insightful question. To the best of our knowledge, none of the existing works[1,2,3] have mentioned this type of evaluation, and there are currently no established benchmarks available for this evaluation. However, to address your concern, we created a small-scale testing set that includes two multi-turn dialogues for each concept generated by GPT, with each dialogue consisting of 2 to 3 turns. The rationale behind this construction is to use GPT to generate multiple questions about the concept in the images, selecting question pairs that can form a dialogue and linking them to create a coherent conversation, thereby addressing the evaluation gap.\n\n| Method | Backbone | Visual Dialogue(BLEU) | Visual Dialogue(GPT) |\n|:------:| :---:|:---:|:---:|\n| Show-o+TP | Show-o | 0.391 | 0.388 |\n| UniCTokens | Show-o | 0.459 | 0.450 |\n\nIt can be observed that the performance of the unified model backbone declines on complex VL tasks. Additionally, performance of UniCTokens also drops when building upon Show-o. Complex VL tasks present both a significant challenge and a promising research direction, and we plan to explore solutions in future work.\n\n> L1: Some ad-hoc parameter is required; So the reproducibility is a little in doubt.\n\nThank you for pointing this out. In fact, we place a high priority on reproducibility, and have presented several hyperparameters and training details in Table 4 of the appendix. We will include more hyper parameter setup to further ensure the reproducibility of our results. Regarding the aforementioned ad-hoc parameter in K-means for identifying the most valuable visual tokens, our experimental setup involves setting K to 2, using cosine similarity as the distance metric, where we select the cluster with the highest number of retained patches.\n\n[1] MyVLM: Personalizing VLMs for User-Specific Queries\n\n[2] Yo'LLaVA: Your Personalized Language and Vision Assistant\n\n[3] RAP: Retrieval-Augmented Personalization for Multimodal Large Language Models"}, {"review_id": "hK5X0OB0DA", "reviewer": "Reviewer_MqL1", "summary": "This paper introduces UniCTokens, a novel personalization framework that enhances both personalized generation and understanding in a unified VLM by fine-tuning unified concept tokens. Unlike prior methods that treat these two tasks independently, UniCTokens enables mutual enhancement by sharing learned representations across tasks. The authors also propose UnifyBench, a new benchmark designed to evaluate concept understanding, concept generation, and knowledge-driven generation within a unified framework. Experimental results on UnifyBench demonstrate the effectiveness and efficiency of the proposed approach.", "questions": "While UniCTokens enables efficient personalized understanding and generation with only a few training samples per concept, the framework seems to require training for each new concept. How scalable is this approach in scenarios where a large number of personalized concepts need to be integrated? Additionally, what are the computational and time costs associated with adding new concepts?", "limitations": "Please refer to \"weakness\".", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "Strengths\n\n-\tThis paper proposes a new framework for personalizing unified VLMs by fine-tuning unified concept tokens for both understanding and generation.\n-\tThe proposed UnifyBench benchmark seems to contribute to the evaluation of concept understanding, concept generation, and knowledge-driven generation of a personalized unified model.\n-\tUniCTokens achieves state-of-the-art performance while maintaining a smaller model size and requiring fewer training samples.\n\nWeaknesses\n\n-\tThe term “Personalized Knowledge-Driven Generation” might be misleading. As presented, it largely refers to image generation guided by personalized textual attributes (e.g., \"⟨x⟩ wearing its hat\"), rather than a more complex or knowledge-integrative reasoning process. Clarifying this terminology would improve reader comprehension. \n-\tThe presentation of results lacks clarity in Table 1. The content (L208-214, L216-219) suggests that it includes results for \"Personalized Concept Generation\" and \"Personalized Knowledge-Driven Generation\", but only \"Personalized Generation\" appear to be explicitly reported in this table. \n-\tThe paper does not address how the model handles prompts that include multiple concepts (e.g., “⟨x⟩ and ⟨y⟩ sitting together”), which is a common scenario in real-world personalization tasks. Including an analysis or discussion of multi-concept generation would strengthen the work.", "was_revised": true, "final_justification": "The authors addressed most of my concerns. I would like to remain my score with borderline accept.", "rebuttal": "## Response to Official Review by Reviewer MqL1\n**Rebuttal:**\n\nThank you very much for taking the time to review our paper. After carefully considering the feedback, we offer the following detailed responses organized point-by-point:\n\n> W1: The term “Personalized Knowledge-Driven Generation” might be misleading. Clarifying this terminology would improve reader comprehension.\n\nThank you for dedicating your time to reviewing our paper and pointing out. We will clarify the terminology after introducing the newly defined task to avoid potential misleading. We propose renaming \"Personalized Knowledge-Driven Generation\" to \"Personalized Attribute-Driven Generation\" to emphasize the core of the task, which may mitigate potential misunderstandings. If you have any better suggestions for naming, we would be glad to discuss them with you.\n\n> W2: Only \"Personalized Generation\" appear to be explicitly reported in this table.\n\nThank you for highlighting this issue. In fact, in Tables 1 and 3, we placed \"Personalized Knowledge-Driven Generation\" under the \"T2I\" column of \"Personalized Generation\", which may have caused some misleading. Overall, your suggestion is highly valuable and practical. We will revise the table structure based on the corrected descriptions in W1 to avoid further misunderstandings.\n\n> W3: The paper does not address how the model handles prompts that include multiple concepts.\n\nFirst, the current unified model has limitations when it comes to generating multiple concepts, which is orthogonal to our proposed method. This limitation has been explicitly discussed in the appendix of Yo'Chameleon[1]. Due to the NIPS Rebuttal policy, which prohibits the inclusion of images or links in the rebuttal, we designed a simple experiment to demonstrate that Show-o shares the same shortcoming in generating multiple concepts. We utilized GPT-4 to generate 100 prompts containing multiple concepts, consistent with the types in the Unify Bench, and allowed Show-o to generate outputs directly. To evaluate the ability to generate multiple concepts simultaneously, the metric is straightforward: a score of 1 is given if multiple concepts are generated simultaneously; otherwise, a score of 0 is assigned. To assess the quality of the generated images, we also had GPT score the outputs. We have listed the performance below:\n\n| Model | GPT |\n|:---|:---:|\n|Show-o| 0.37|\n|Show-o2-7B| 0.51 |\n|Chameleon | 0.21 |\n\nIt can be observed from the table that both Show-o and Chameleon struggle to generate multiple concepts. Building on Show-o, UniCTokens also inherits these limitations. If better unified models become available in the future, our method has the potential to benefit from these advancements. Finally, we would like to clarify that the primary focus of our paper is on unified model personalization and facilitating mutual enhancement between understanding and generation tasks. We will consider addressing the issue of multiple concept generation in future work.\n\n> Q1: How scalable is this approach in scenarios where a large number of personalized concepts need to be integrated?\n\nFor general fine-tuning based methods (e.g., Dreambooth, Yo'LLaVA), each concept has its own set of parameters. To evaluate the scalability of our method, we selected five concepts from the Unify Bench and employed a progressively continuous training approach. After learning one concept, we freeze the parameter and prepare new parameters for the next concept. After training on all the concepts, we conducted tests on each of them:\n\n|    Type           | Rec                        | VQA                        | QA                         | Pure Gen                | Pure Gen                | People Gen               | Knowldeg-driven        | Knowldeg-driven        |\n|:---------------|:----------------------------:|:----------------------------:|:----------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|:-------------------------:|\n| Metric        | Weight                     | GPT                         | GPT                        | CLIP-I                  | CLIP-T                  | Face-Simi              | Score                  |  CLIP-I                  |\n| Yo'LLaVA-Phi   | 0.737                      | 0.377                      | 0.382                      | -                       | -                       | -                       | -                       | -                       |\n| Dreambooth    | -                          | -                          | -                          | 0.611                   | 0.259                   | 0.101                   | 0.050                    | 0.608                   |\n| Yo'Chameleon   | 0.742                      | 0.391                      | 0.433                      | 0.623                   | 0.187                   | 0.156                   | 0.199                    | 0.623                   |\n| UniCTokens    | 0.758                      | 0.422                      | 0.480                     | 0.691                   | 0.270                   | 0.221                   | 0.307                    | 0.694                   |\n\nContinuously integrating new concepts may lead to catastrophic forgetting. However, UniCTokens still outperforms Yo'LLaVA-Phi, Dreambooth and Yo'Chameleon in this setting, which may be attributed to the well-designed progressive training strategy.\n\n> Q2: What are the computational and time costs associated with adding new concepts?\n\nAs you suggested, in accordance with the Q1 setting, we have presented the computational and time costs comparison results of adding new concepts. From this table, we can draw the following observations:\n\n| Type | FLOPs/Concept | Time/Concept |\n|:---|:---:|:---:|\n|Yo'LLaVA-13B|$1.5×10^{17}$|30min|\n|Yo'LLaVA-Phi| $1.6×10^{16}$ |10min|\n|Dreambooth(SD)| $2×10^{15}$ |7min|\n|Yo'Chameleon| $7×10^{17}$ |120min|\n|UniCTokens| $1.3×10^{17}$ |25min|\n\nIt can be observed that Yo'Chameleon takes a long time because it trains each concept using approximately 1,100 images. In contrast, UniCTokens achieves stronger MMU and T2I capabilities while keeping the training computational and time costs under control.  \n\nWe sincerely hope the above explanations can alleviate your remaining concerns and any potential misunderstandings about our paper. And we will include all the above-mentioned tables in the revised version of our paper. We wish this work could bring new insights to the field of personalization for unified models and provide feasible solutions for cross-task transfer and mutual enhancement between tasks in more complex scenarios.\n\n[1] Yo'Chameleon: Personalized Vision and Language Generation"}]

Author rebuttal:

Meta-review: All three reviewers recommend acceptance of this submission, highlighting the strengths of its framework and method. While no major issues remain, their acceptance is conditioned on the authors incorporating the changes promised in the rebuttal, specifically including, adding the new experiments, clarifying the task definition, improving table presentation, expanding the discussion on multiple concepts, and providing clearer descriptions along with additional ablation studies.
Decision: Accept (poster)