UniF$^2$ace: A $\underline{Uni}$fied $\underline{F}$ine-grained $\underline{Face}$ Understanding and Generation Model
Authors: Junzhe Li
Sifan Zhou
Liya Guo
Xuerui Qiu
Linrui Xu
TingTing Long
Chun Fan
Ming Li
Hehe Fan
Jun Liu
Shuicheng YAN
Venue: ICLR 2026 Poster (2026)
Keywords: Human-centric AI
Face Generation
Face Understanding
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Unified multimodal models (UMMs) have emerged as a powerful paradigm in fundamental cross-modality research, demonstrating significant potential in both image understanding and generation. However, existing research in the face domain primarily faces two challenges: **(1) fragmentation development**, with existing methods failing to unify understanding and generation into a single one, hindering the way to artificial general intelligence. **(2) lack of fine-grained facial attributes**, which are crucial for high-fidelity applications. To handle those issues, we propose UniF$^2$ace, the first UMM specifically tailored for fine-grained face understanding and generation. **First**, we introduce a novel theoretical framework with a Dual Discrete Diffusion (D3Diff) loss, unifying masked generative models with discrete score matching diffusion and leading to a more precise approximation of the negative log-likelihood. Moreover, this D3Diff significantly enhances the model's ability to synthesize high-fidelity facial details aligned with text input. **Second**, we propose a multi-level grouped Mixture-of-Experts architecture, adaptively incorporating the semantic and identity facial embeddings to complement the attribute forgotten phenomenon in representation evolvement. **Finally**, to this end, we construct UniF$^2$aceD-1M, a large-scale dataset comprising *130K* fine-grained image-caption pairs and *1M* visual question-answering pairs, spanning a much wider range of facial attributes than existing datasets. Extensive experiments demonstrate that UniF$^2$ace outperforms existing models with a similar scale in both understanding and generation tasks, with 7.1% higher Desc-GPT and 6.6% higher VQA-score, respectively. Code is available in the supplementary materials.

Published as a conference paper at ICLR 2026
UNIF²ACE; A UNIFIED FINE-GRAINED FACE UNDER- STANDING AND GENERATION MODEL
Junzhe L Sifan Zhou?; Liya Guo", Xuerui Qiu Linrui Xu², Delin Qu?, Tingting Long?. Chun Fan?, Hehe Fan", Jun Liu", Ming Li Shuicheng Yan
1 School of Computer Science, Peking University, *Comptter Center, Peking University, 3 *Carnegie Mellon University 'Tsinghua University
"Institute of Automation, Chinese Academy of Sciences Central South University
Fudan University ZZhejiang University Lancaster University
10 Guangdong Laboratory of Artificial Intelligence and Digital Economy (SZ)
National University of Singapore "Zhongguaneuu Academy
ABSTRACT
Unified multimodal models (UMMs) have emerged as a powerful paradigm in fundamental cross moosslto research, demonstrating significant potential in both image understanding and generation. However, existing research in the face domain primarily faces two challenges: (1) fragmentation development, with existing methods failing to unify understanding and generation into single one, hindering the way to artificial general intelligence.
(2) lack of fine-grained facial attributes, which are crucial for high-fide fidelity applications. To handle those issues, we propose UniF2 the first UMM specifically tailored for fine- men-grained face understanding and generation. First, we introduce novel theoretical framework with Dual Discrete Diffusion (D3Diff) loss, unifying masked generative models with discrete score matching diffusion and leading to a more precise approximation of the negative log- ligeliklioood. Moreover, this D3Diff significantly enhances the model's ss ability to synthesize high-fidelity facial details aligned with text input. Second, we propose a multi-level grouped Mixture-of-Fxpere incorporating the semantic and identity facial embeddings to complement the attribute forgotten phenomenon in representation evolvement. Finally, to this end, we construct UniF grained image-caption pairs and IM visual question-answering pairs, spanning a much wider range of facial attributes than existing datasets. Extensive experiments demonstrate that UniF2ace outperforms existing models with a similar scale in both understanding and generation tasks, with 7.1% higher Desc-GPT and 6.6% higher VQA-score, respectively. Codes and datasets are available at Unee
architecture, adaptively
a large-scale dataset comprising 130K fine-
INTRODUCTION
Recently, unified multimodal models (UMMs) have emerged as a vibrant research area enabling cross-modality understanding and generation within a single "any-to-any" framework, marking
significant step toward artificial general intelligence (AGI) (Wu et al. [2024a] Shi et al. 2024 [i et al.
2024a; Zhou al. 2024 Zhou 2025 Team 2024 Xie et 2024 2025 Zhang et al. 2025 Hu et al.
2025 Chen et al. 2024a 2025a bi Liu et al. 2025a b 2026 Liu et al.) [i et al. 2025 Yu et al. (025).
Given the central role of faces in daily life, applying this unified paradigm to achieve fine-graines face understanding and generation is essential for developing numan-centria AGI. The practical appli- cations are vast and critical: accurate face understanding is pivotal for identity verification (Srinivasan)
et al. 2024 Roshdy et al. 2024) and human-computer interaction (Liu) 2024 Chowdary et al., 2023). while high-fidelity face generation drives progress in creative industries (Melnik et al. 2024). virtual
avatars Yan et al. 2024), and data augmentation for model robustness (Melzi et al.
2023 These
"Equal contribution. (lijunzhe 1028 stu. stu, stu.pku.edu.co)
1 Corresponning author.

---

Published as a conference paper at ICLR 2026
Face Understonding-Oly
Face Generation-One Only
Froegis: The wons Far tapresiii oyes ore freeas by arcenl syelines.
Can you describe his hair and beard?
Oraete sort natch diperes tare, cenrieutis 2
T2I Model (Diffusion)
VLM (AR)
Flae-grained X
Fine-grained X
MBUMS The se ia the yocure has oull, doke how heir that oppiar wrlngee
legtt, neatty trimasd
Face Undertandinng
Face Geseration
X
Faee Uiderstanding X
Frce Generation
Unified Face Understanding and Generation
Question:
Q:
Con describe his hair ond beard?
Prompts:
The woman smilss..... Her experssive eys.....
UniF2ace (AR+Diffusion)
Fine-grained
A[Ours): Cartairlyl medium length with mix block The
wary well-maintiine
beard full, neatly greemed,
Fine-grained
Additionally, he slightly receding
Face Understanding
Face Generation
Figure 1: UniF ace is the first unified multimodal model designed for face understanding and genera- tion, encompassing tasks such visual question Inswering(VQA. and text-to-image generation. The generated responses and images demonstrate Unil" potential in fine-grained face attributes.
demanding real-world needs urge facial research to push the boundaries of multimodal understanding and generative modeling.
As shown in Fig. despite the critical importance of human faces, existing research faces two funda- mental and intertwined challenges: First, the field remains fragmented, with current methodologies treating face understanding and generation as separate endeavors, failing to unify these capabilities into a single framework. Typically, face understanding models are often based on fine-tune pretrained multimodal large language models (MLLMs) on facial images with coarse text descriptions (Chet taoui et al. 2025; Sun et al. 2024at Xing al.) 2024). Face generation models (Huang al. 2023: Nair et al. 2023; Kim et al. 2024) often incorporate visual information, such as semantic masks and sketches, to guide high-fidelity face synthesis, but they cannot achieve direct generation from detailed captions to faces. This leads to disjointed workflows that are both computationally inefficient and functionally restrictive. Crucially, the absence of a unified framework represents a significant hurdle towards the realization of AGI within the domain of human faces. Second, there a pervasive lack of fine-grained information processing across both understanding and generation tasks. This challenge stems from three problems: (a) The discrete diffusion model inherits the advantages of diffusion for image generation while enabling better scalability modeling with text tokens in UMMs (Yang et al. [2025] Xie et al. 2024). However, its specific implementation primarily relies on masked generative models (Chang et al. (022). lacking a combination with accurate score matching (Lou et al.) 2024), making it challenging
generate precise image details; (b) detailed attribute representations are prone to being discarded during the learning evolution in multimodal models (Zeng et al.) 2024 He al.) datasets featuring fine- mne-grained attributes. Existing text-face datasets fall into two types: web-scraped low-resolution facial images with inaccurate captions (Li et al., 2024c| Zheng et al. (022). and close-up facial datasets with limited attributes per caption (only 2 to 7) (Xi et al. 2021] Yu et al., 2023a), which lack detail. Moreover, current facial datasets do not include VQAs, limiting their use for fine-grained understanding tasks. Furthermore, this deficiency directly impacts high-quality face
2025); and (c) the inaccessibility of cross-modality facial
generation (Xiao et al. 2025 Deng et al. 2025 Wang et al. 2025a).
To handle these challenges, we propose UniF2ace (see Fig. 1. the first UMM specifically tailored for unified and fine-grained face understanding and generation. Uni aims to address the aforementioned critical challenges by simultancously performing both tasks and capturing fine- grained facial attributes within a single model. Specifically, we firstly introduce Dual Discrete Diffusion (D3Diff) loss within a novel theoretical proof that optimizes the negative log-likeling ligliikllhood, significantly improving generation quality. After that, we propose an integrated token-level and sequence-leve Exxure-of-ExxperssMMon facial embeddings, effectively addressing the attribute forgotten phenomenon in representation evolvement and specialized fine- fine-grained representation learning for both understanding and generation tasks. Finally, recognizing the critical role of data, we construct UniF2 aceD- IM, a large-scale, specialized dataset containing 130K facial image-text pairs and /M visual question-anse -nestinn-answering
architecture that adaptively handling semantic and identity
2

---

Published as a conference paper at ICLR 2026
(VQA) pairs, with 17.7 attributes per caption. Extensive experiments on UniF aceD-1M and other benchmarks demonstrate that Une significantly outperforms various top-leading single-task models UMMs with similar scale and dedicated face models across both understanding and generation tasks, with 7. % higher Desc-GPT and 6.6% higher VQA-score. Besides, our method achieves comparable even better accuracy than larger-scale models and establishes a strong baseline. Our main contributions are as follows:
A unified face understanding and generation framework: We introduce Unie the first unified multimodal model for fine-grained face understanding and generation, establishing solid baseline.
novel Dual Discrete Diffusion (D3Difff) loss function and hybrid MoF architecture: We introduce D3Diff, a novel loss function within that theoretically unifies score-based diffusion and masked generative models, leading to a better approximation of the negative likelleoooood for high-fidelity generation and fine-grained attribute control. Additionally, we explore hybrid Mixture-of-Expert (MoE) architecture at the token and sequence levels, adaptively incorporating the semantic and identity facial embeddings complement the attribute forgotten phenomenon in representation evolvement.
We construct JceF2aceD-Ill IM, a dataset containing IM VQAs with automated pipeline. Extensive experiments demonstrate that UniF2ace significantly outperforms or is on par with existing state- of-the-art models with similar scale on various benchmarks, all while providing more unified and efficient solution.
RELATED WORK
The field of Unified Multimodal Models (UMMs) has seen significant progress in integrating diverse understanding and generation tasks within single frameworks for generic domains (Ma al. 2024
Team. 2024 Xie et al. 2024). However, their application to fine-grained visual analysis, especially in the complex domain of human faces, remains largely unexplored. Within the face domain, existing
research is primarily fragmented into separate understanding models (often MLLM-based) (Sun et al.
2024a| Xing al., (024) and generation models (typically diffusion-basee based) (Dai et al.) 2025 Wang et al. 2024b). Crucially, these approaches often struggle with fine-grained attribute processing and fail to unify understanding and generation effectively. This dual deficiency represents a significant
gap that UniFPace addresses. We also provide more comprehensive review of Unified Multimodal Models and Face Multimodal Models in Appendix A
METHODOLOGY
We introduce our unified multimodal model, UniF2ace, designed to model both the understanding and generation of fine-grainee facial attributes. Our approach is realized from two perspectives: loss function (Sec. 3.1) and network architecture (Sec. 3.2). Regarding the generation strategy, we recognize that the generation of fine-grained facial attributes is significantly more challenging than understanding tasks, as highlighted in prior studies (Du et al.) 2017 Zhou et al. 2024 Xie et al.] 2024). To address this, we harness the theory of score matching in discrete diffusion (Lou al. (024) and propose the dual discrete diffusion (D3Diff) training strategy, ensuring the meticulous synthesis of facial details. For network architecture, existing UMMs typically focus on dense architectures (Zhou
et al. 2024 Xie et al. 2024) or solely on achieving token-level MoE (Deng et al. 2025), lacking the selective integration of instance features. To overcome these limitations, we introduce token-level and sequence-level MoE. layers. Distinct MoE modules are designed for generation and understanding tasks, selectively integrating information such as facial embeddings to enhance the model's ability to capture subtle facial attributes.
3.1 DUAL DISCRETE DIFFUSION
In generative modeling, masked generative models (Chang et al. 2022) are widely adopted approach. However, in this section, we introduce discrete score matching and theoretically prove that it offers better approximation to the negative log- likeliiooodd. We also establish theoretical connection between the two approaches and finally propose a new loss function to ensure stable optimization, thereby improving the alignment between the generated faces and fine-grained attributes in prompts.
3

---

Published as a conference paper at ICLR 2026
In discrete diffusion process, each image Xy is associated with a probability g(xo), and its latent distribution at time t under noise adding is denoted by ((x,). The forward diffusion is modeled as a continuous-timg time Markov chain, governed by the linear ordinary differential equation (ODE):
d dt ( x) (3) ax) Qt
(I)
which converges stationary distribution as oc. Here, Q: denotes time-dependent sequence of transition matrices. The closed-form solution of this ODE can be expressed as Q exp((o(!)-
(s)) Q), where o(t) = So (ss) ds represents the cumulative noise level and exp is the matrix exponential. The reverse process is given by Lou al. (2024):
dqr-t QT QT-1 dt
4: (3) Qx( (x)
where Q is the reverse diffusion matrix. In our work, focus the absorbing state, which is widely
used masked generative models (Chang et al. 2022: Xie et al. 2024). Assuming independence
among tokens, supported by Sahoo al. (2024); ; Shi al. (2025), ) the exact formulation is
deferred to Appendix D The score-based discrete diffusion model (Lou et 2024 introduces a
wrainin-sttbble loss socore (sp) estimate the denoising score. It is defined as:
q(y) ((x)
((y)
Lxxone (80) = Exxup
Σ y#x
Way (x(()))
log 80 (x)y +
(3)
q(x))
where se (x(xx,)) t)
(v))
is the predicted score from the neural network, and K(a)
ge
M.E
allog 1) a normalizing constant ensuring Lucore 2 O.
To illustrate the advantage of Lycore, we start from the negative log-likel (NLL), which serves as a fundamental criterion for evaluating the quality of training in generative models. Since exact computation of the NLL is generally infeasible, prior works have derived two different surrogate formulations that upper bound the NLL while remaining computationally tractable. Specifically, one is L1 Lscorc(se) + DKL (2rio(- (x)) Ponse). which predicts the score (Lou et al. 2024). And the
other is L2 = Ei= Eeg(Ko) 9(xx) xx.)l log (xo) (xo|xx)) x - C. widely used to predict masked tokens (Xie al. 22244. where C' is residual constant independent of the model parameters (see Appendix for details). The following theorem formally establishes the relationship between these two surrogates and demonstrates that L1, which incorporates the score loss, yields a tighter upper bound on NLL.
Theorem 1. Let- log (x3) denote the negative log-tikelihood of the original data distribution. Then the following inequality holds:
- -log (xo) S S2-
The proof deferred Appendix Importantly, this result implies that Lwore provides a tighter relaxation of the maximum likelihood objective compared to the masked generative related loss C2. thereby offering a more precise approximation of the NLL, In practice, the marginal distribution qy)) often intractable, and the exact analytical form of (xx) is unknown. A key insight is that, in masked generative models, the posterior probability model po (xu xx) can be related to the discrete diffusion score function via Bayes' theorem:
qe (xx)
(x xx) se (x;
(x0)
= qe (Xp Xo) (x::).
(5)
ge (xx))
Leveraging this relation, we propose the dual discrete diffusion (D3Diff) loss for training posterior networks:
T
CDDDiff - Σ (gxxa))(xx)xx)) [log pe (xo xx)] + acCoorr (xo (xx))x/))xxxx())))))
(6)
where q(x,) is the data distribution, q(,, xx) is the forward diffusion distribution, and po (xxx))))) is the learned posterior parameterized by 0. The score loss Lraore is weighted by hyperparameter Eq. establisses a computationally tractable connection between masked generative models and score-based models. Unlike traditional masked generative losses, which rely solely on likelihood, our D3Diff loss jointly optimizes two distinct upper bounds of the maximum likelihood objective, enabling stable optimization and fine- mnaiird generation (See Tab. 7).
4

---

Published as a conference paper at ICLR 2026
Next foken Loss
D3biff Loss
Condy
SxtuleXuuul (kuootssjae) infowee (si(e))
Nacket
Muit Geaeette
bone Macking
Grouped Sequence Level MoE
toppree
LoyerNorm
Grouped Token Level MoE
CLIF Expert
Foos Expert
Copy Expert
Copy Expert
Zaro Expert
No Expert
Expert
Expert
LayerNorm
Causal Attention
Utper Taker
Imogeli encoder:
Text Tekemizer
Add Noise
(Takenizon
Shared Expert
N-1
Image encader
Tou
hill
Task
Gen
Task
Figure 2: Our UniF centered on two key innovations. First, we design the Transformer with Mfxture-of- Experts (MoE) hierarchy: token-level MoE provides task-specific routing for individual tokens, while sequence-level MoE injects holistic, Nomain-specinf features. Second, the model's generative capability is optimized by our proposed D3Diff loss, which unifies masked generation with score matching to ensure high-fidelity synthesis graigrai facial details.
3.2 MULTI-LEVEL GROUPED MIXTURE-ON OF-EPERT
To capture fine-grained facial attributes while maintaining facial embeddings, design distinet MoE layers, termed Multi-level Grouped MoF, tailored for both generation and understanding subtasks. This ensures optimal performance for each task, illustrated in Fig. 2 We incorporate sequence- level MoE layer after the token-level MoE layer to effectively process instance-level inputs, such as images and facial embeddings.
Token-Level MoE. We partition a feedforward neural network (FFN) into multiple experts with reduced hidden dimensions and use a Top-K activation strategy (Fig. (2) We also employ integrate generalized knowledge across contexts. Unlike prior methods, we introduce grouped MoE, dividing experts into two groups based on the different tasks of Text-to-Image (T2I) and Multimodal Understanding (MMU). Each group combines shared and routed MoE, with expert-level balance loss com- puted independently per group:
Figure 3: Clip/Face Expert enhances the model's understanding of fine- grained facial attributes by incor- porating semantic and identity fea- tures.
Output
Na
Namn
Lallance
Apzi 2 fiP: Amnnnu Σ J2P
where ipz and Ammu are balance factors; Nuz and Nmma means the number of routed experts for T2I and MMU tasks, re- spectively; fo and P, denote expert selection frequency and probability.
Resampler
We
Ideenty Feeture
CLIP/InsightFace
Sequence-Level MoE. We propose sequence-level MoE. where distinct experts process the entire image feature. We design three experts for the T2I group: copy expert (skip op- eration), zero expert (discard operation), and noise expert. The copy and zero experts require no additional parameters.
Image Input
Input Hidden States
Easpy (x:
and Exro (x: O,
(8)
where Ecspy (-) the copy expert and E-gro is the zero expert. For the noise expert Enoise (-), we first integrate the time-step embedding, which operates the noise level o()) obtain the noise
5

---

Published as a conference paper at ICLR 2026
embedding vector Vooises following score-based discrete diffusion models (Lou et al. 2023). Then, a resampler S R + RR maps Vnoise into the sequence feature space (see Appendix for resampler details). The resampled noise embedding is added matrix to the sequence feature. Formally, the noise expert's output is:
Enoise (x) = -x + (1 - (l)))(((nsss))
Softmax W Flatten(x));
(10)
where W Wooise Boise 2)) is trainable weight matrix. In the MMU task, we include copy experts and introduce CLIP experts and face experts (See Figure (3). which are similar to noise experts. Next we extract image embeddings by CLIP (Radford al.] (20TT) and face embeddings using Antelope
as supplementary features to enhance fine-grained facial attribute capture. Formally, the outputs of the CLIP and face experts are:
(x) Welip + (1 Welpp (x())
(11)
Eace (x) = face + (1 face (x)))
(12)
where G and F are the image encoder and face encoder, respectively, X is the input face image.
3.3 OVERALL TRAINING OBJECTIVES
To perform both auto-regressive and discrete score-based diffusion modeling, we employ two learning objectives: 1) Next Token Prediction (NTP) and 2) Dual Discrete Diffusion. Given sequence with N image tokens X = {X1, , and A text tokens y = {Y, .e...., Yu ). Then we maximize the likelihood of text tokens y by employing the standard language modeling objective (NTP loss):
M
LMMU
log P(Y, X),
(13)
Next, the overall training objectives of ane are formulated as:
Lioual = LuMU + Lawaar
(14)
EXPERIMENT
4.1
UNIF"ACED-I M DATASET FOR FINE-GRAINED FACE UNDERSTANDING AND GENERATION
Dataset
Face Resolution
Availability
Caption Tokens
Face Attributes (Avg. per Caption)
Nastaal
Emustion
Viale
LAION-Face PLIP-BIM
Appertion
Read
UniiF
MM--Calbhemm Text
FRHO-Teu Zhou
Oifanrs
dctia
D-IM
2(M))
Glan
Figure 4: UniFPaceD- 1M contains high-resolution facial images, the largest number of facial at- tributes, 130K fine-grained image-caption pairs and million VQAs.
Existing datasets for multimodal facial modeling frequently suffer from significant limitations, hindering advancements in fine-graie fine-grained understanding and generation. Common deficiencies include low-resolution imagery, imprecise or web-scraped captions lacking subtle attribute details, and pervasive absence of comprehensive visual
(VQA) pairs tailored to facial specifics (Li et al., 2024c. /Zhnng et al., 2022] Xia et al., 2021 Yu et al., 2023a Karras 2019). These shortcomings mean that current models struggle synthesize nuanced facial expressions, comprehend intricate visual semantics, or reason effectively about complex facial attributes. To overcome these challenges and truly enable unified and fine-grained multimodal facial intelligence, we introduce aneD- This high-quality dataset serves as a comerstone of our framework, meticulously designed to bridge these critical data gaps.
As shown in Fig. 4 our UniF2acel aceD- 1M provides a resource distinguished by its fine- fine-grained detail and scale. It comprises nearly 130K high-fidelity facial images, each paired with richly detailed

---

Published as a conference paper at ICLR 2026
captions that encompass wide spectrum of 46 attributes related to appearance, actions, and emotions. This meticulous level of detail is paramount for both robust model training and the generation of highly controllable and realistic facial outputs. Furthermore, a key innovation of UniF2aceDD IM is the inclusion of approximately 1M specialized VQA pairs. Unlike general VQAs, ours are meticulously crafted to probe diverse facial appearances, emotions, and provide detailed reasoning for character actions. This unique VQA collection is specifically designed to enhance MLLMs ability understand and reason about fine-grained facial attributes through instruction tuning. By offering a substantial collection of high-quulli -uallit facial imagery. richly detailed captions, and unique, large-scale set of facial VQAs, UniF2acel 1M sets a new standard, providing the indispensable resources for developing next-generation unified models capable of sophisticated fine-grained facial intelligence. More collection and operation details can be found in the Appendix
4.2 METRICS AND OTHER FACIAL DATASETS
We rigorously evaluate UniFFace's performance across both generation and understanding tasks on our UniF2aceD- test set. To provide comprehensive assessment and verify the generalizability of our method, we also conduct evaluations on other public benchmarks, including FFHQ-Text (Zhou & Shimada 2021), MM-CelebA (Xia et al., 2021), and CelebV-Text (Yu et al. 2023b). For generation tasks, we used VQAscore to measure the relevance of generated images to captions, reporting results based on CLIP-FlanTS:
(VQAscore-CEF CFS) (Lin et al.. 2024b) and LLaVA-v1.5-13B (VQAscore- LV) (Liu et al. (024c) for robust assessment. We also employ Fréchet Inception Distance (FID) to measure similarity to ground truth and VLM-score to evaluate facial realism. For understanding tasks, we follow LLaVA (Liu et al., 2023) and use GPT-4o (Hurst et al. 2024) and DeepSeek-v3 v3 (Liu ctt al. 2024a) to score responses on a 1-10 scale across two dimensions: detailed captioning (Desc- GPT, Desc-DS), assessing accuracy in capturing face attributes, and VQA (Conv-GPT, Conv-DS), measuring precision in responding to fine-grained queries. To fully validate UniF2 ace, we compare it with SOTA models. This includes generative models such autoregressive LlamaGin (Yu et al., 2023ch and diffusion bbase Stable Diffusion (SD3) (Esser et al. 2024), as well as leading unified
multimodal models (UMMs) like TokenFlow (Qu et al. 2024) and OmniFlow (Li al. 2024 More
implementations details are in Appendix C
4.3 FACE GENERATION
Table I: Comparison of face generation of UniF2ace ace with generative-only and UMMs. Bold indicates the best, while underlined denotes the best. We use red to highlight the larger-scale model.
Type
Model
Method
# Params
VQAscoreCCF5t
VQAscore-LV.
FID.
VLM-score
LlamaGion al 2024b
AR
0.8B
0.746
0.551
183.465
49,773
DALL-E Pookker el du 2025
AR
0.845
0.644
106.477
50, 122
Gen. Only
SDS Esser 01
Diff
2B
0,903
0.671
93.471
75.944
SDXL
Diff
2668
0.876
0.660
123.095
72,764
Flux. 1-dey 10b Z1241
Diff
128
0.893
0.674
76.427
84,513
TokenFlow
Quet
AR
0,871
0.664
98.194
73.177
OmniFlow
F 80740
Diff
3.48
0.798
0,585
180.933
24.96 160
Und. and Gen.
JanusFlow
AR Dif''
1.3B
0,881
0.653
72,825
61.593
Show-o
Xic a
2024
AR + Diff'
1.38
0.855
0.650
142.557
75.618
UniF"nos(uirs)
AR Diff
1.88
0.894
0.679
66.005
88.049
Table 2: Comparison of face generation other public datasets. The experimental setup utilized the built-in short captions of datasets as text prompts for generation.
Model
Params
FFHIQ-Tes
MM--Cabb
VQAScoeee
FID:
VIM.-Scere?
VQJASoore!
FID.
VL3M-Scare
VQAScore
FID)
VLM-Scord
Llunsiieo Sureet
0.388
1.356
201.341
46.412
0.31%
157.311
49.121
(1721
250 541
66,906
503
286
04423
155.120
34,992
0.459
74.127
SDN.
2A6
01356)
161.261
64.25
0.420
174
73.149
01785
271.319
70,991
17B1
04434
136.300
13.021
0.467
124 463
87.764
03066
254
84401
Tide
16
0,409
160023
0.411
71.002
0.781
79.525
Und.
1.38
0.413
231
601964
0.485
131
03.418
0.797
250
66.557
Show-a
1.30
0.391
177.53
13.14]
0.418
141.311
24,342
0.785
250.210
210
70-452
UniPachturgg
1300
125.287
87.412
0.281
85.179
36.978
01804
234.012
943336
Generation Performance on UniF 1M and Public Dataset. On our UniFaceD-IM benchmark (Tab. D. our 1.8B parameter Uni sets new state-of-the-art outperforming all competing UMMs on key generation metrics including FID, VQA-score, and VLM-score. Furthermore, the model also demonstrates robust generalization, consistently achieving leading scores public cross-facial datasets such as FFHQ-Text (Zhou & Shimada (02T), MM-Celeb,, (Xia et al., 202T). and Celeb Text YYu et al. 2023a) (Tab. This strong and consistent performance validates the effectiveness of
7

---

Published as a conference paper at ICLR 2026
SDXL
TokenFlow
OmniFlow
Show-o
UniFrace(Ours)
child
hat.
The child's cheelss and her siggtll pornedl She has light-cslned syea, expressiio curisaity wonder. breased in linght
cokeet she odck a wroant controst to the scene. The beckground a sefilly Slamned curtiion nenter aetting?
The image Festures Temole with Strosght block holr. Siee hos e forr comple onlor with srooth skir Texture and will ceffieeddd arched eyebrows that lis with naturdl suorle The persoon wearing hoop earninga imiling
complement desp-aet brewn syncs nomec sarnty laughnn commeying hoppinsce,
exprossa
Figure 5: Comparative analysis of face images generation quality across SDXL (Podell et al. 2023 TokenFlow (Qu et al. 2024). OmniFlow (Li et al. 2024a). Show-o (iie et al. 2024), and Unil ace. Our proposed Unil-ace effectively captures more detailed information from prompts.
Und, Experts Activations
Layer
Layer
Und, Sequence Moe Activations
Gen. Sequence Moe Activations
Gap 0.3 Page
Expert Nape
Expert
Figure 6: Activation frequency of Token-Level and Sequence-Level MoE in different layers. The left column indicates understanding tasks, while the right column indicates generation tasks. Larger circles indicate experts who are activated more frequently.
our D3Diff loss and multi-level grouped MoE architecture for high-quality, fine-grained facial image generation across diverse settings.
Visualization Analysis. As shown in Fig. 5 we conduct qualitative evaluation on challenging Uni test scenarios involving complex facial details. UniF excels at generating
realistic faces that capture fine-grained details from complex prompts (e.g., "rosy cheeks," "hoop carrings"), visibly outperforming other models. More examples can be found in Fig. S and Fig. 9 Besides, shown in Fig. 6 analyze MoE activation frequencies across layers. For token-level MoEs, high activation frequencies are concentrated between experts 5 and 8, indicating limited token feature variability in the generation task. For sequence-level MoEs, noise and expert activations are evenly distributed, indicating effective training with selective noise embedding and truncation.
4.4
FACE UNDERSTANDING
Table 3: Comparison of face understanding of UniFFace with understanding-onf, and UMMs.
Type
Model
Method
Parmss
Desc-GPT*
Conv-GPT:
Desc-DS
Cony-DS:
V.L. iin et
2023
AR
3B
4.76
5.20
6.56
6.54
Und. Only
Qwen2-VL
Wang
2224
7B
5.16
6.27
5.50
6.86
LLaVA-v1.5
Liu a.. 2024D
AR
7B
4.28
5.48
4.84
6.20
IntemVLL25
8B
5.62
5.89
6.30
6.55
Qwen2.5-VL
Bai et 2023
AR
3B
4,88
6.38
4.98
6.75
TokeaFlow
Qt
2024
AR
7B
5.80
5.82
6.39
OmniFlow
20746
Diff
3.4B
1.62
1.90
Und. and Gen.
JanusFlow Mi
202
+
1.38
4.88
6.06
6.77
Show-o Xic
2024
AR Diff
1.38
3.88
4.17
5.24
4.90
UniF"ae(Uus)
+ Diff
1.8
6.02
6.53
7.38
7.29
8

---

Published as a conference paper at ICLR 2026
Table 4: Comparison of face understanding on other public datasets. The experiments utilized the dataset's captions as labels for captioning task evaluation, showing the robustness of UniF2aae
Type
Model
Params
FFHQ-Text
MM-CelebA
Vexe
Dese-GIT
Desc-DS
Desc-GIT
Dese-DS
Dese-GIT
Desc-DS
VILAI.5
at
2023
3B
4.29
4.79
4.48
4.59
4.61
4.76
Qwen2-Vi V1.
2024a
7B
4.68
5.41
5.11
5.40
4.90
4.95
Und.Only
LLaVA-v1.5
el
20241
7B
4.01
4.60
4.29
4,26
4.54
4.50
IntemVV1.2.5
[Ceen
21244
8B
5,09
5.58
4.75
4.98
5.07
5.01
Qwen2.5.VL,
Bai al
2025
3B
4.38
4.92
4.72
4.70
5.20
5.10
TokenFlow
Quu at.
2024
7B
5.04
5.75
4.99
5.01
4.86
5.10
OmnilFlow
[. a
20.44
3,4B
2.83
3.06
3.41
3.38
2.90
3.03
Gen.&Und,
JanusFlow
1.3B
4.31
5.15
4.60
4.71
4.54
4.86
Show.o
et et
2024
1.3B
3.86
4.67
4.38
4.39
4.49
4.57
UniF"ace (Uurs)
1.8B
5.12
5.92
6.24
6.80
5.87
5.29
Table 6: Performance comparison of non- Table 5: Performance comparison of understanding celebrity face understanding. on FaceXBench (Narayan et al. 2025).
Model
Parnns
Dese-GPT
Desc-D
Medel
ABc Exp.
Pare Rec.
Qwen2-VI. Ware et 2013463
7B
4,97
5.13
50
13.75
13
71.33
Qwen2.5-VE V.
3B
5.30
29.0
Unil" (Ours)
5.54
Understanding Performance on UniF- and Public Dataset. On our UniF IM benchmark (Tab. 3). UniF2ace sets a new state-of-the-ard in fine- fine-grained facial understanding, achiev- ing the highest scores across all metrics. Crucially, it surpasses even larger, specialized models like Intern (8B) and all competing UMMs. Furthermore, as shown in Tab. 4 this superior under- standing capability demonstrates strong generalization, Unil ace also consistently achieves top captioning scores on public fross-facial datasets. This robust performance across diverse benchmarks validates the effectiveness of our approach in learning transferable, fine aeneigrieed facial representations, affirming UniF2ace 's leading position in comprehensive multimodal facial understanding.
Evaluation Bench. We further evaluated our model on FaceXBench (Narayan al. 2025), a comprehensive benchmark designed for assessing Multimodal Large Language Models (MLLMs). Since the original FaceXBench includes multi-image reasoning tasks that exceed the current single-image inference capability of Unil-ace we curated a specialized evaluation subset by extracting all single-image test instances. This robust subset comprises total of 2,150 instances, specifically covering Attribute Expression (400), Bias Fairness (450), Face Recognition (150). Face Localization (1,000), and Face Anti-Spoofing (FAS) & Deepfake (150). Given its significant scale and task diversity, as shown in Table 5 this refined evaluation set provides rigorous and comprehensive assessment of single-image facial understanding capabilities.
Evaluation on non-celebrity inputs. We benchmarked our model against Qwen2- VL.(B) (Wang al. 2024a) and Qwen2.5-VL(3BB) (Bai et al. al.] (2025) using 3,000 random samples from FLIP-80M et al. 2024c) to assess robustness in face understanding. Distinguished from standard celebrity- centric datasets, FLIP-80M captures the complexity of real-world data, including web-scraped and Al-generated content. The dataset presents substantial challenges regarding inconsistent lighting, blur, and unconstrained compositions, consisting largely of low-resolution, non-celebrity images. The results are shown in Table 6 demonstrating the robustness of Une
Quantitative Evaluation. We compare UniF2ace ace with advanced autoregressive multimodal models like Qwen2-VL (Wang et al.) 2024a) (7B) and similar UMMs (Wu et al. 2024c Xie et al. 2024). Visual comparisons (right Fig.) confirm Unil-acees superior fine-grained understanding VQA tasks, where it ac- curately identifies subtle details like "sideburns" and "earrings" that competitors miss. We also provide more examples for captioning in the Appendix (Fig, TO). Be- sides, as shown in Fig. (left column), we analyze MoE activation frequencies in the understanding task. Token-level MoEs often select the same expert in the top (the closest to the prediction head) and bottom lay- ers. For sequence-level MoEs, face and CLIP experts more frequently activated in layers closer to the
Q: Con you describe his hairstyle? The man in the image has a messy, unkempt hairstyle with his hair sticking up all directions.. stterees
VILA1. 5:
messy, spiky hairstyle lot of volume and texture, The hair is light brown and appoars be somewhat unkompt:
JanusFlow: The person
UniéFace: Certainlyl has brown hair that thick and wavy, styled slightly tousled manner. His hairstyle also includes sideburns that complement the overall look
Q: Is she wearing earrings? InternVL2 The image is not clear enough to determine if she is wearing earnings. citereaas.
Show-o: Yes, the woman is wearing earrings. GRoeends
Uni Yes, she wearing carrings, which her stylish oppearance. (store7s)

---

Published as a conference paper at ICLR 2026
top, indicating that deeper layers benefit from visual embeddings to better understand face images. Notably, activation patterns are different across the generation groups, highlighting the effectiveness of our group-basee based strategy.
4.5 ABLATION STUDIES
Coefficient in Dual Discrete Diffusion. As shown in Table 7 we analyze the D3Diff loss.
The optimal weight is a = 0.01 which balances the 200x magnitude difference between the score-matching and masked generative
Table 7: Ablation study with different loss weights.
losses. The complete D3Diff loss sig-
nificantly outperforms using either loss
Loss Type
Weight a
VQAscureCCPS
VQAsore-L.
FID.
VI.M-cure'
component individually. Crucially, the
Only Maak
0
0.879
0.661
77,463
85.993
superiority of the score-only loss over the
Only Scure
0.01
0.886
0.670
69.691
87.951
0.1
0.887
0.673
68.00
86 378
masked-only loss empirically validates our theoretical analysis in Appendix
DADMT
0.01
0.891
0.679
66,005
88 88.019
Table 8: Ablation of Face and CLIP Expert.
Table Ablation of Top-k in Seq-level MoE.
Export Type
Understanding
Top-K
Generation
Understanding
CLIP
Dese-GPT
Conv-GPT)
Desc-DI
Cony-DS
VQAscure
FID,
VLM-score
Desc
Convi
X
X
5.21
3.31
6.27
6.36
0.879
74.914
74.314
6.57
6.42
X
5.67
5.93
6.86
7,30
0.894
66.005
88.049
7.38
7.29
0.895
65.413
7.34
6.02
6.53
7.38
7.29
0.897
63.632
87.795
7:23
7.36
Ablation of MoEs Architecture. We ablate our multi-level MoE design (Tab. g Tab. 9 Tab. T0 The
results (Tab. [O) first confirm that combining token-level and sequence-level MoEs achieves the best performance, with each component individually outperforming the non- MoE baseline. We further analyze the sequence-level MoE, finding that: (I) for understanding tasks, using both CLIP and Face experts is optimal in fine-grained facial understanding (Tab. [); and (2) Top-k=2 selection strategy provides the best balance be-
Table 10: Ablation study of token- and sequence-level MoE.
Token MoE
Sequence MoE
Generation
Understanding
VQAscore?
FID
VLM-score
Desc
Conv
0.878
72.877
84.432
4,988
6.031
x
0.887
67.415
87.917
5.678
6.495
69.312
0.894
66.005
88.049
6.023
6,532
tween performance and efficiency (Tab. 19. These findings validate our hierarchical and specialized
MoE design.
5 CONCLUSION
This paper introduces Unee the first unified multimodal model (UMM) designed for fine- grained face understanding and generation. The model bridges the gap between score-based models and masked generative models in discrete diffusion, while leveraging token-level and sequence- level mixture-of ex-errssss (MoE) to sparsify the model. Extensive experiments show that Uni ace outperforms existing UMMs and even surpasses larger generation-only understanding-ont models. This underscores the potential of our improvements to guide future research in face applications of UMM. Additionally, we constructed large-scale face-text aligned dataset, aneD- MM, to further advance multimodal research in the community.
6 ACKNOWLEDGMENTS
This work was supported by the National Natural Science Foundation of China (Grant Nos. 62502317, 92570101, and 62320106007) and the Science and Technology Development Special Fund Project of Xinjiang Production and Construction Corps (Grant No. 2025AB027). Partial support was also provided by the NUS Start-up Grant (A-0010106-00-00) and the Zhongguaneun Academy (Grant No. 1800302002).
10

---

and the Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
I read all respects with the ICLR Code of Ethics research conducted in the paper complies all respects.
REPRODUCIBILITY STATEMENT
This paper fully discloses all the source code needed to reproduce the main experimental results in the supplementary material. Besides, also provide the complete description of the proposed dataset for their data processing steps in the Appendix B Finally, we also provide clear explanations of our assumptions, and a complete proof of the claims can be included in the Appendix D
F and F
REFERENCES
Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Iige Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.087 087774 2023.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report. arXiv preprint 13323 2025.
James Betker, Gabriel Goh, Jing, Tim Brooks, Jianfeng Wang, Linjie Li, Long Ouyang, Juntang Zhuang, Joyce Lee, Yufei Guo, et al. Improving image generation with better captions. Computer Science. http:://cdn. openai, com/papershuall--33. pdf. 2(3):8, 2023.
Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1 11315-11322 2022.
Kaixiang Chen, Pengfei Fang. and Hui Xue. Multi-modal interactive agent layer for few-shot universal cross-domain retrieval and beyond. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025a.
Kaixiang Chen, Pengfei Fang, and Hui Xue. Depro: Domain ensemble using decoupled prompts for universal cross-domain retrieval. In Proceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 958-967, 2025b.
Xiaohong Chen, Canran Xiao, and Yongmei Liu. Confusion-resistant federated learning via diffusion- based data harmonization non-iid data. In Proceedings of the 38th International Conference on Neural Information Processing Systems, pp. 137495-137520, 2024a.
Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling. arXiv preprint arXiv:2501.178. 2025c.
Zhe Chen, Weiyun Wang, Yue Cao, Yangzhou Liu, Zhangwei Gao, Erfei Cui, Jinguo Zhu, Shenglong Ye. Hao Tian, Zhaoyang Liu, et al. Expanding performance boundaries of open-source multimodal models with model, data, and test-time scaling. arXiv preprint 24Xi::24122.0271, 2024b.
Tahar Chettaoui, Naser Damer, and Fadi Boutros. Froundation: Are foundation models ready for face recognition? Image and Vision Computing, pp. 105453, 2025.
M Kalpana Chowdary, Tu N Nguyen, and D Jude Hemanth. Deep saariin--based facial emotion recognition for homanecomputer interaction applications. Neural Computing and Applications, 35 (32):23311-23322 2023.
Damai Dai, Chenggi Deng, Chenggang Zhao, RX Xu, Huazuo Gao, Deli Chen, Jiashi Li. Wangding Zeng, Xingkai Yu, Yu Wu, et al. Deepseekmoe: Towards ultimate expert specialization in mixture- of-experts language models. arXiv preprint MxXiv:240100000 2024.

---

Published as a conference paper at ICLR 2026
Dawei Dai, Mingming Jia, Yinxiu Zhou, Hang Xing, and Chenghang Li. Face-makeup: Multimodal facial prompts for text-to-image generation. arXiv preprint arXiv:2501.02523, 2025.
Chaorui Deng, Deyao Zhu, Kunchang Li, Chenhui Gou, Feng Li, Zeyu Wang, Shu Zhong, Weihao Yu, Xiaonan Nie, Ziang Song. Guang Shi, and Haoqi Fan. Emerging properties in unified multimodal pretraining, arXiv preprint 2rXiv:2505.14683 2025.
Xinya Du, Junru Shao, and Claire Cardie. Learning to ask: Neural question generation for reading comprehension. In Proceedings the 55th Annual Meeting of the Association for Compulational Linguistics. Association for Computational Linguistics, 2017.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. Multi-modal cross-domain alignment network for video moment retrieval. IEEE Transactions Multimedia, 25:7517-7532, 2022.
Xiang Fang, Wanlong Fang, and Changshuo Wang. Cogniverse: Revolutionizing multi-modal retrieval-augmmened generation with cognitive reflection and geometric reasoning, Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2026a.
Xiang Fang, Wanlong Fang, and Changshuo Wang. Unveiling the fragility of vision-language models: Multi-modal adversarial synergy via texture- conttraied perturbations and cross-modal optimization. In Proceedings of the AAAI Conference on Artificial Intelligence, 2026b.
Tianle Gu, Zeyang Zhou, Kexin Huang, Liang Dandan, Yixu Wang, Haiquan Zhao, Yuanqi Yao, Yujiu Yang. Yan Teng, Yu Qiao, et al. Milmguard: A multi-dimensional safety evaluation suite for multimodal large language models. Advances in Neural Information Processing Systems, 37: 2256-7295, 2024.
Tianle Gu, Kexin Huang, Lingyu Li, Ruilin Luo, Shiyang Huang, Zongqi Wang, Yujiu Yang, Yan Teng, and Yingchun Wang. From sparse decisions dense reasoning: A multi-attribute trajectory paradigm for multimodal moderation. arXiv preprint arXiv:2602.02536, 2026.
Hulingxiao He, Geng Li, Zijun Geng, Jinglin Xu, and Yuxin Peng. Analyzing and boosting the power of fine-grained visual recognition for multi-modal large language models, arXiv preprint arXiv:2501.15 151400 2025.
Jua Hu, Shanshan Zhao, Qing-Guo Chen, Xuerui Qiu, Jialun Liu, Zhao Xu, Weihua Luo, Kaifu Zhang, and Yanye Lu. Omni-view: Unlocking how generation facilitates understanding unified 3d model based multiview images. arXiv preprint arXiv:2511.07222, 2025.
Ziqi Huang, Kelvin CK Chan, Yuming Jiang, and Ziwei Liu. Collaborative diffusion for multi-modal face generation and editing. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 6080-6090, 2023.
Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Os- trow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card. arXiv preprint arXiv:2410.21276, 2024.
Tero Karras. style- based generator architecture for generative adversarial networks. arXiv preprint arXiv: tet..0442..499 2019.
Tero Karras, Timo Aila. Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for improved quality, stability, and variation. 2018.
Jihyun Kim, Changjae Oh, Hoseok Do, Soohyun Kim, and Kwanghoon Sohn. Diffusion- driven gan inversion for multi-modal face image generation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 10403-10412, 2024.
Black Forest Labs. httppp///sttt..ccco///aaoaoeaesa
2024.
12

---

Published as a conference paper at ICLR 2026
Bokyeung Lee, Hyunuk Shin, Bonhwa Ku, and Hanseok Ko. Frame level emotion guided dynamic facial expression recognition with emotion grouping. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 5681-5691, 2023.
Jingzhi Li, Changjiang Luo, Ruoyu Chen, Hua Zhang, Wengi Ren, Jianhou Gan, and Xiaochun Cao. Faceinsight: A multimodal large language model for face perception. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 11052-11061 2025.
Shufan Li. Konstantinos Kallidromitis, Akash Gokul, Zichun Liao, Yusuke Kato, Kazuki Kozuka, and Aditya Grover. Omniflow: Any-to-any generation with multi-modal rectified flows. arXiv preprint arXiv:24/2.01 169, 2024a.
Yanwei Li, Yuechen Zhang, Chengyao Wang. Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, and Jiaya Jía. Mini-gemini: Mining the potential of multi-modality vision language models. arXiv preprint arXiv:2403. 188144 2024b.
Yudong Li, Xianxu Hou, Zheng Dezhi, Linlin Shen, and Zhe Zhao. Flip-80m: 80 million visual- linguistic pairs for facial eraguage-image mre-training. In Proceedings of the 32nd ACM Interna- tional Conference on Multimedia, pp. 58-67, 2024c.
Zijie Li, Henry Li, Yichun Shi, Amir Barati Farimani, Yuval Kluger, Linjie Yang, and Peng Wang.
Dual diffusion for unified image generation and understanding. arXiv preprint 2:11:22501.00289, 2024d.
Bin Lin, Zhenyu Tang, Yang Ye, Jiaxi Cui, Bin Zhu, Peng Jin, Jinfa Huang, Junwu Zhang, Yatian Pang, Munan Ning, et al. Moe-llava: Mixture of experts for large vision- lisoon-language models. arXiv preprint MXXv:2441.159444 2024a.
Ji Lin, Hongxu Yin, Wei Ping, Yao Lu, Pavlo Molchanov, Andrew Tao, Huizi Mao, Jan Kautz, Mohammad Shoeybi, and Song Han. Vila: On pre-training for visual language models, 2023.
Yexiong Lin, Yu Yao, Zhaoqing Wang, Xu Shen, Jun Yu, Bo Han, and Tongliang Liu. Improving the instance-dependent transition matrix estimation by exploiting self-supervised learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025.
Zhiqiu Lin, Deepak Pathak, Baiqi Li, Jiayao Li, Xide Xia, Graham Neubig, Pengchuan Zhang, and Deva Ramanan. Evaluating text-to-visual generation with image-to-text generation. In European Conference on Computer Vision, pp. 366-384. Springer, 2024b.
Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao Chengqi Deng, Chenyu Zhang, Chong Ruan, et al. Deepseek-v3 technical report. arXiv preprint arX1v:2412.19437, 2024a.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36:34892-34916, 2023.
Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved basclines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 26296-26306, 2024b.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. Advances in neural information processing systems, 36, 2024c.
Jiaxi Liu. Chatgpt: Perspectives from human-computer interaction and psychology. Frontiers in Artificial Intelligence, 7:1418869, 2024.
Jiumiin Liu, Mengmeng Liu, Siting Zhu, Yunpeng Zhang, Jiangtao Li, Michael Ying Yang, Francesco Nex, Hao Cheng, and Hesheng Wang. Arflow: Auto-regressive optical flow estimation for arbitrary- length videos via progressive next-frame forecasting. In The Fourteenth International Conference on Learning Representations.
Jumming Liu, Weicai Ye, Guangming Wang, Chaokang Jiang, Lei Pan, Jinru Han, Zhe Liu, Guofeng Zhang, and Hesheng Wang. Difflow3d: Hierarchical diffusion models for uncertainty-awee aware 3d scene flow estimation. IEEE Transactions on Pattern Analysis and Machine Intelligence, 48(3):
2837-2854, 2026. doi: 10. 11999PPAM..22553329555
13

---

Published as a conference paper at ICLR 2026
Junkang Liu, Yuanyuan Liu, Fanhua Shang. Hongying Liu, Jin Liu, and Wei Feng. Improving generalization federated learning with highly heterogeneous data via momentum based stochastic controlled weight averaging. In Forty-second second International Conference on Machine Learning, 2025a.
Junkang Liu, Fanhua Shang, Yuxuan Tian, Hongying Liu, and Yuanyuan Liu. Consistency of local and global flatness for federated learning. In Proceedings of the 33rd ACM International Conference on Multimedia, pp. 3875-3883, 2025b.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. arXiv preprint S..iv:2310.168344 2023.
Aaron Lou, Chenlin Meng, and Stefano Ermon. Discrete diffusion modeling by estimating the ratios of the data distribution. In Forty-first International Conference on Machine Learning, 2024.
Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang. Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation. arXiv preprint 1.099.241..07775 2024.
Andrew Melnik, Maksim Miasayedzenkau, Dzianis Makaravets, Dzianis Pirshtuk, Eren Akbulut, Dennis Holzmann, Tarek Renusch, Gustav Reichert, and Helge Ritter. Face generation and editing with stylegan: A survey. IEEE Transactions on pattern analysis and machine intelligence, 46(5): 3557-3576, 2024.
Pietro Melzi, Christian Rathgeb, Ruben Tolosana, Ruben Vera- Pera-Rodriguez, Dominik Lawatsch, Florian Domin, and Maxim Schaubert. Gandiffface: Controllable generation of synthetic datasets for face recognition with realistic variations. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 3086-3095, 2023.
Chenlin Meng. Kristy Choi, Jiaming Song, and Stefano Ermon. Concrete score matching: Generalized score matching for discrete data. Advances in Neural Information Processing Systems, 35:34532- 34545, 2022.
Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual adversarial training: regularization method for supervised and supervied learning. JEEE transactions on pattern analysis and machine intelligence, 11(8):197--199 1993, 2018.
Nithin Gopalakrishnan Nair, Wele Gedara Chaminda Bandara, and Vishal M Patel. Unite and conquer:
Plug & play multi-modal synthesis using diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6070-6079, 2023.
Kartik Narayan, Vibashan VS, and Vishal M Patel. Facexbench: Evaluating multimodal Ilms on face understanding. arXiv preprint arXiv:2501.10330 2025.
Chaojun Ni, Guosheng Zhao, Xiaofeng Wang. Zheng Zhu, Wenkang Qin, Guan Huang, Chen Liu, Yuyin Chen, Yida Wang, Xueyang Zhang, et al. Recondreamer: Crafting world models for driving scene reconstruction via online restoration. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 1559-1569, 2025.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10744 2021.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01952. 2023.
Liao Qu, Huichao Zhang, Yiheng Liu, Xu Wang. Yi Jiang, Yiming Gao, Hu Ye, Daniel K Du, Zehuan Yuan, and Xinglong Wu. Tokenflow: Unified image tokenizer for multimodal understanding and generation. arXiv preprint arXiv:2412.03069) 2024.
14

---

Published as a conference paper at ICLR 2026
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748-8763. PMLR, 2021.
Ahmed Roshdy, Abdullah Karar, Samer Al Kork, Taha Beyrouthy, and Amine Nait-ali. Advancements in eeg emotion recognition: Leveraging multi-modal database integration. Applied Sciences, 14(6):
2487, 2024.
Subham Sekhar Sahoo, Marianne Arriola, Yair Schiff, Aaron Gokaslan, Edgar Marroquin, Justin Chiu, Alexander Rush, and Volodymyr Kuleshov. Simple and effective masked diffusion language models. arXiv preprint arXiv:2406.07522 07524, 2024.
Jiaxin Shi, Kehang Han, Zhe Wang, Arnaud Doucet, and Michalis Titsias. Simplified and generalized masked diffusion for discrete data. Advances in Neural Information Processing Systems, 37: 103131-103167, 2025.
Weijia Shi, Xiaochuang Han, Chunting Zhou, Weixin Liang, Xi Victoria Lin, Luke Zettlemoyer, and Lili Yu. Llamafusion: Adapting pretrained language models for multimodal generation. arXiv preprint arXiv:2412.15188 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learning. pp. 2256-2265. pmlr, 2015.
S Srinivasan, R Raja, C Jehan, S Murugan, C Srinivasan, and M Muthulekshmi. Jot-enabled facial recognition for smart hospitality for contactless guest services and identity verification. In 2024 11th International Conference on Reliability, Infocom Technologies and Optimization (Trends and Future Directions//CRTTTO) pp. 1-6. IEEE, 2024.
Haomiao Sun, Mingjie He, Tianheng Lian, Hu Han, and Shiguang Shan. Face-mllm: large face perception model. arXiv preprint arXiv:2410.20711 2024a.
Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autoregressive model beats diffusion: Llama for scalable image generation. arXiv preprint 066v::2400.06555, 2024b.
Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models. arXiv preprint 0991:2405.00818, 2024.
Kai Tong, Kang Pan, Xiao Zhang, Erli Meng, Run He, Yawen Cui, Nuoyan Guo, and Huiping Zhuang. Any-ssr: How recursive least squares works in continual learning of large language model. In Proceedings of the IEEF/CVF International Conference on Computer Vision (ICCV), pp. 3047-3057, October 2025.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13977 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
Hanyang Wang, Bo Li. Shuang Wu, Siyuan Shen, Feng Liu, Shouhong Ding, and Aimin Zhou. Rethinking the learning paradigm for dynamic facial expression recognition. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 17958-17963 17968, 2023.
Linqing Wang, Ximing Xing, Yiji Cheng, Zhiyuan Zhao, Jiale Tao, Qixun Wang, Ruihuang Li, Xin Li, Mingrui Wu, Xinchi Deng, et al. Promptenhancer: A simple approach to enhance text-to-image models via chain-of-though prompt rewriting. arXiv preprint 250: 250.0445555 2025a.
Peng Wang. Shuai Bai, Sinan Tan, Shijie Wang. Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang. Wenbin Ge, et al. Qwen2-vl: Enhancing rision-langure model's perception of the world at any resolution. arXiv preprint 2409 /2/9/, 2024a.
15

---

Published as a conference paper at ICLR 2026
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot 2024b.
generation in seconds. arXiv preprint arXiv:2401.05519,
Xiaoqin Wang. Xianxu Hou, Meidan Ding, Junliang Chen, Kaijun Deng, Jinheng Xie, and Linlin Shen. Disfacerep: Representation disentanglement for co-occurring facial components in weakly supervised face parsing. Proceedings of the 33rd ACM International Conference on Multimedia, pp. 4020-4029, 2025b.
Xiaoqin Wang. Xusen Ma, Xianxu Hou, Meidan Ding, Yudong Li, Junliang Chen, Wenting Chen, Xiaoyang Peng, and Linlin Shen. Facebench: A multi-view multi-level facial attribute vqa dataset for benchmarking face perception milms. In Proceedings of the Computer Vision and Putlern Recognition Conference, pp. 9p..1544 9154-9164, 2025c.
Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need. arXiv preprint arXiv:2409.18880 2024c.
Junfeng Wu, Yi Jiang, Chuofan Ma, Yuliang Liu, Hengshuang Zhao, Zehuan Yuan, Song Bai, and Xiang Bai. Liquid: Language models are scalable multi-modal generators. arXiv preprint 21t.:24112.4332. 2024a.
Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal 1lm. In Forty-first International Conference on Machine Learning, 2024b.
Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation. arXiv preprint arXiv:2409. 249:02409 2024c.
Weihao Xia, Yujiu Yang, Jing-Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2021.
Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shuting Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 13294- 13304, 2025.
Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijic Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation. arXiv preprint arXiv:2408. /2528, 2024.
Mengwei Xie, Shuang Zeng, Xinyuan Chang, Xinran Liu, Zheng Pan, Mu Xu, and Xing Wei. Seggrowgraph: Learning lane topology as a chain of graph expansions. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 27166-277
2025.
Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, and Heikki Kälviänen. Emo-llama: Enhancing facial emotion understanding with instruction tuning. arXiv preprint arXiv:2408. / /424, 2024.
Yicheng Xu, Yuxin Chen, Jiahao Nie, Yusong Wang, Huiping Zhuang, and Manabu Okumura. Advancing cross-domain discriminability in continual learning of vision-language models. arXiv preprint arXiv:2406. 02824066.88666 18868, 2024.
Yichao Yan, Zanwei Zhou. Zi Wang, Jingnan Gao, and Xiaokang Yang. Dialoguenerf: Towards realistic avatar face-to-face face conversation video generation. Visual Intelligence, 2(1):24, 2024.
Ling Yang, Ye Tian, Bowen Li, Xinchen Zhang, Ke Shen, Yunhai Tong, and Mengdi Wang. Mmada: Multimodal large diffusion language models. arXiv preprint arXiv:2505. 15809, 2025.
Jiang Yong Yu, Sifan Zhou, Dawei Yang, Shuoyu Li, Shuo Wang, Xing Hu, Chen Xu, Zukang Xu, Changyong Shu, and Zhihang Yuan. Mquant: Unleashing the inference potential of multimodal large language models via static quantization. In Proceedings of the 33rd ACM International Conference Multimedia, pp. 1783-1792, 2025.
16

---

Published as a conference paper at ICLR 2026
Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu. TextbV-Text: A large-scale facial text-video dataset. In CVPR, 2023a.
Jianhui Yu, Hao Zhu, Liming Jiang, Chen Change Loy, Weidong Cai, and Wayne Wu. Celebv-text: A large-scale facial text-video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 14805-14814, 2023b.
Lijun Yu, José Lezama, Nitesh B Gundavarapu, Luca Versari, Kihyuk Sohn, David Minnen, Yong Cheng. Vighnesh Birodkar, Agrim Gupta, Xiuye Gu, et al. Language model beats diffusion- tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023c.
Shuang Zeng, Dekang Qi, Xinyuan Chang, Feng Xiong, Shichao Xie, Xiaolong Wu, Shiyi Liang, Mu Xu, and Xing Wei. Janusvin: Decoupling semantics and spatiality with dual implicit memory for vision-language navigation. arXiv preprint arXiv:2509.22.
2025.
Yunan Zeng. Yan Huang, Jinjin Zhang, Zequn Jic, Zhenhua Chai, and Liang Wang. Investigating compositional challenges in vision-language models for visual grounding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14141-14151, 2024.
Fengda Zhang, Qianpei He, Kun Kuang, Jiashuo Liu, Long Chen, Chao Wu, Jun Xiao, and Hanwang Zhang. Distributionally generative augmentation for fair facial attribute classification. In Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22797-22808, 2024.
Guozhen Zhang. Zixiang Zhou, Teng Hu, Ziqiao Peng. Youliang Zhang, Yi Chen, Yuan Zhou, Qinglin Lu, and Limin Wang. Uniavgen: Unified audio and video generation with asymmetric cross-modal interactions. arXiv preprint arXiv:2511.03334, 2025.
Fufangchen Zhao, Ming Li, Linrui Xu, Wenhao Jiang, Jian Gao, and Danfeng Yan. Favchat: Unlocking fine-grained facial video understanding with multimodal large language models. arXiv preprint arXiv:2503.099 099558, 2025.
Zengqun Zhao, Yu Cao, Shaogang Gong, and loannis Patras. Enhancing zero-shot facial expression recognition by Ilm knowledge transfer. arXiv preprint MXXi::245519100) 2024.
Yinglin Zheng, Hao Yang, Ting Zhang, Jianmin Bao, Dongdong Chen, Yangyu Huang, Lu Yuan, Dong Chen, Ming Zeng, and Fang Wen. General facial representation learning in aisual-ligguiste manner. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 18697-18709, 2022.
Chunting Zhou, Lili Yu, Arun Babu, Kushal Tirumala, Michihiro Yasunaga, Leonid Shamis, Jacob Kahn, Xuezhe Ma, Luke Zettlemoyer, and Omer Levy. Transfusion: Predict the next token and diffuse images with one multi-modul model. arXiv preprint 2rXiv:2408.1100 2024.
Yining Zhou A unified reinforcement learning framework for dynamic user profiling and predictive recommendation. Available at SSRN 584/223, 2025.
Yutong Zhou and Nobutaka Shimada. Generative adversarial network for text-to-face synthesis and manipulation with pretrained bert model. In 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pp. 01-08, 2021.
Hao Zhu, Wayne Wu, Wentao Zhu, Liming Jiang, Siwei Tang. Li Zhang, Ziwei Liu, and Chen Change Loy. CelebV-HQ: A large-scale video facial attributes dataset. In ECCV, 2022.
17

---

Published as a conference paper at ICLR 2026
A RELATED WORKS
Unified Multimodal Models. Recent works (Ma et al. 2024 Li et al. 2024d Wu et al. 2024a
Chen et al., 2025c. Wang et al.) 2024c Fang et al., 2022; 2026a b Tong et al. 2025 Xu et al.
2024)/Zeng et al. (2025) in image understanding and generation have primarily focused on unified
multimodal models (UMMs). Early approaches (Li al. 2024b Wu al. 2024b) often integrated external decoders of diffusion models (DMs) with text autoregressive models (ARMs). Inspired by next-token prediction tasks, they proposed using a single Transformer (Vaswani et al. (017) model to unify understanding and generation (Wu et al. (2024c). For instance, Janus-Pro (Chen et al. (202cc) decouples the visual encoder into specialized tokenizers for separate handling of understanding and generation tasks. Chameleon Team 2024) and Emu3 (Wang et al.) (202c)) employ an ARM to simultaneously manage both tasks, highlighting the advantages of autoregressive models in multitask settings. Additionally, Transfusion (Zhou et al. (024) and Show-o (Xie et al., (024) combine a text ARM with visual DM, enabling seamless integration of image understanding and generation. These studies have advanced the fusion of visual and text generation models, enhancing performance on multimodal tasks. However, despite the proliferation of UMMs, their application has largely been limited to generic domain tasks, with limited exploration in fine-grained visual analysis, particularly in the face domain. Unlike previous UMMs that simply combine ARMs and DMs, we pioneer sparse UMMs by introducing both token-level and sequence-level Mixture of Experts (MoEs), significantly improving model performance.
Face Multimodal Models. Face multimodal models are primarily categorized into two types: face understanding models and face generation models. For understanding, carly models were
task-specific and lacked multimodality (Miyato et al. 2018 Zhang et al.) 2024; Wang et al. 2023
Lee al. 2023). Recent works (Chettaoui et al. 2025| [unn et al. 2024a| Xing al. 2024 Zhao
et al.] 2024)) leverage the reasoning capabilitiies of LLMs or MLLMs, often using MLLM --genrrtte
face Q& data to fine-tune or post-train foundation models, incorporating face domain knowledge. For example, EMO-LLaMA (Xing et al., 2024) introduces facial experts to extract facial features, which are aggregated with handcrafted prompts and fed into LLaMA (Touvron et al. (023)) enabling
it to answer facial-related queries. Recent research (Wang et al.) 2025b Li et al. 20255 Zhao cct al. 2025) has increasingly focused on performing tine-grained facial attribute analysis. On the modeling front, Facelnsight (Li et al.) (025) advances facial perception by introducing visual-textual alignment of facial knowledge and segmentation maps. FaVChat (Zhao et al. (2025) extends these fine-grained perceptual capabilities to the domain of video understanding. Complementing these modeling advancements, a new wave of benchmarks has emerged for rigorous evaluation (Narayan) et al. 2025 Wang et al. 2025c). A notable example is FaceXBench (Narayan et al.) 2025). which provides a comprehensive assessment covering 14 tasks across 6 broad categories, including bias and fairness, authentication, recognition, and analysis. Collectively, these synergistic efforts in both model development and evaluation are driving the field of fine- grained face understanding forward. For generation, recent works (Dai et al.) 2025 Wang et al., 2024b. Huang al. 2023 Kim et al. 2024) focus on using diffusion models to personalize face images by conditioning on textual and visual information, such as semantic masks, but avoid directly capturing fine-grained face attributes from text prompts. Despite these advances in understanding and generation separately, developing unified multimodal models (UMMs) remains a significant research challenge. Addressing this gap can enhance cross-modal capabilities and advance progress toward Artificial General Intelligence (AGI).
B DATASET CONSTRUCTION
To overcome the limitations of existing datasets in the realm of multimodal facial modeling, we introduce a high-quality dataset called Unit aceD- which boasts remarkable alignment between facial images and textual descriptions (see Fig. 7). This dataset encompasses nearly 130K facial images, each paired with richly detailed captions. Additionally, it contains approximately 1M visual question answers, significantly enhancing its value for training and evaluating multimodal models. By offering such comprehensive resource, aim to propel advancements in facial image understanding and generation, establishing a solid foundation for a wide range of multimodal learning tasks. The creation of aneF2aC encompassed three key stages. (1) Step-1: Collect high-quality facial
18

---

Published as a conference paper at ICLR 2026
Step-1
Step-2
Examples
L Attribute Classifier
Appecrance: blands
Image Encoder
har bage under eyes Emotion: noutrsl
GPT40
Action
Earril daptions The per the trom
maye kid for condorke with rudy
Celebv-IIQ
Face Encoder
rppaermnce the cheske, Thainhe hoi ligh-
Hurr wesgt erend the heod,
around the his the head ring but
Head
blks grean and exprestiie witt
racpett ai is sipqued corne anfic Shw
ainundly craad,, co cokee arring coriil
cang lonurt her drouff race and edbtis on
traught
Step-3 3
mudo and musts diele wiilh codicco
ecduu There one clickt begt unile
% Attribute
cesuetratio She conuinng like
FFRQ
Clossifier
euppen eight be enpoced in
harking moofmmpen spering
Tra-ggnieed Vivke
derer haireyli
@ GPT4a
? 6PT40
All Corte my the his light bince styled brcid tich wnest amoudd the top
Q2: W octice dhe cattrr
fecued engrging converattio, Q2: Daed upporr tend yoh
bo deserptioe.
Fine- Fine-grained
GPT4
MM-Codeb--IQ
VQAs
bor eneodiis resuir god and frieclyy
Figure 7: Pipeline and examples of UniF construction. Left: A three-stage pipeline for building UniFaaceD-IM IM. 1: High-quality face images are collected. Step-2: Detailed captions are generated by GPT-4o with face attribute model trained to classify fine-grained appearance, action, and emotion. Step-3: Question-ansee answering pairs are created. These stages collectively refine SPT-4o-geenratte captions and produce fine-grand fine-grained descriptions for VQAs generation. Right: A representative example showcasing acelD- 1M's ability to correct (e.g., gender), enhance (e.g.. captions.
bags under eyes), and reason (e.g., talking, slight tiredness) in
images. (2) Step-2:
Generate detailed captions. (3) Step-3: Create question-answeriin pairs. Each stage is outlined in detail below.
(1) Step-1:
Collect High-quality Facial Images. In this step, curated more than 130,000 high-
quality facial images from the following distinguished datasets. Vele (Zhu et al. (022)) is large-scale video dataset featuring 35,666 clips representing 15,653 identities, each clip meticulously annotated with 83 facial attributes. We extracted one key frames from each video to utilize detailed annotations for fine-grained face-text alignment. Flick-FFaces-HG (FFHQ) (Karras 2019) provided 70,000 high-quality PNG images at a resolution of 1024 by 1024, offering substantial diversity Miee-MMMea---bbAAA
in attributes such as age and ethnicity. 2021) contributed 30,000 high-resolution images paired with descriptive captions that have proven invaluable for facial generation and analysis.
(MM-CelebA A-Q)) (Xia al..
(2) Step-2: Generate Detailed Captions. Existing face image datasets often lack detailed descrip- tions of fine-grained attributes like bags under eyes or jewelry. To handle this, we develop two-stage caption generation process.
In Stage I, employed advanced MLLM such as GPT-4o (Hurst al. 2024) produce initial captions. We designed specialized prompt that incorporated brief face descriptions from the MM-CelebA-HOC dataset (Xia al. 2021) help GPT-4o accurately describe key facial attributes, including appearance, emotion, and actions. The detailed descriptions of all prompts are presented later (see Fig. 1
In Stage II. we refined these captions by training face attribute classification models using the CelebV- HQ dataset (Zhu et al.) 2022). Focusing on single-person images, we used the pretrained face model Antelope to extract face embeddings. By combining these with image embeddings from CLIP (Radford et al. 2021), we trained classification heads for appearance, action, and emotion attributes. We selected 29 appearances with accuracies over 93%, 10 actions with accuracies over 87%, and 7 emotions with accuracies over 80% as final preditions for inference. These highly accurate attributes were then predicted for all remaining images in FFHQ and MM-CelebA-HO datasets (Karras 2019 Xia et al. 202T). Finally, a prompt integrating these classification results with the Stage captions was fed into GPT-4o to generate final captions that are both highly accurate and diverse.
19

---

Published as a conference paper at ICLR 2026
(3) Step-3: Create Question-answering Pairs. In this step, we proposed 1M VQAs covering diverse facial appearances, emotions, and character action reasoning for our UneF 1M dataset. These VQAs are designed to enhance MLLMs' ability to understand fine- ine-grained facial attributes through instruction tuning. Inspired by LLaVA (Liu et al. 2024c), we carefully designed prompts to enable GPT-4 (Achiam al., 2023) to generate series of VQAs based on image captions, facilitating fine-grained understanding and reasoning. Most current face-text datasets lack VQAs, while VQÅs in general image-text datasets often focus on people's clothing, location, and behavior, neglecting detailed facial descriptions. In contrast, our proposed VQAs encompass diverse facial details, including hair, nose, eyes, mouth, ears, skin, eyebrows, and adornments. Additionally, since facial attributes can reflect a character's s ongoing actions, our VQAs incorporate detailed reasoning processes to infer and describe these actions. By organizing the VQAs into the same format as the LL dataset (Liu et al.] (024c), we streamlined the process of adapting multimodal face models for post-training This alignment minimizes alteration costs, ensuring efficient integration and enabling the models to leverage both datasets scamlessly for improved performance.
Discussion of ethical risks. Regarding the ethical risks of multimodal models (Gu et al. 2024 2026), we first verified that the licenses for CelebV-HQ, FFHQ, and MM-CelebA--HC permit non- commercial research and educational use. Our contribution, which consists of secondary annotations for these datasets, executed with focus neutrality to prevent bias, and every L..M-generteed annotation underwent manual review to ensure the absence of hallucinations. To prevent future misuse, the released dataset will explicitly state these terms and strictly inherit the licensing restrictions of its source data.
C IMPLEMENTATIONS DETAILS
We train our model on the UniF2acel--IM training dataset part, comprising 120K 256 x 256 face images, each annotated with detailed captions and seven to eight VQAs, about 900K. 2niF2aae utilizes discrete image tokens as input, represented by the pre-trained MAGVIT-v2 (Yu et al. 2023c). For token-level MoE, each group (generation and understanding tasks) includes one shared expert and eight routed experts. selected via a top-2 strategy. The expert structure is single-layer MLP with the gating mechanism (Dai et al., 2024). In sequence-level MoE, the generation group employs two copy experts, one zero expert, and one noise expert. Noise embedding is implemented using sinusoidal embedding, following (Nichol et al. 202T). The noise resampler uses a 4-layer Multi- Attention mechanism to map noise embeddings to the UniFPace hidden space. For the understanding group, there are two copy experts, one CLIP expert, and one face expert. We employ pre-trained self-supervised models (Lin et al. 2025), such CLIP-VIT for image embedding and Antelope V2 for face embedding, with resampler configuration consistent with that of the noise expert. Moreover, training is divided into two stages: Stage I uses only captions for generation and understanding tasks, while Stage II incorporates VQAs into the understanding task. This pipeline transitions the model from general image feature understanding to fine-grained feature capture. Both stages are trained on 8 NVIDIA A 100 (80GB) GPUs, optimized using Adam W with a weight decay of 0.01, 5K warm-up up steps, and an initial learning rate of 5c-5 with cosine scheduling. The total batch size is 600 for Stage I and 480 for Stage II, with 20K steps for Stage I and 40K steps for Stage II. For a fair comparison, we also performed full- parameeeeeter fine-tuning on all competing models using an identical amount of data, leveraging their official fine-tuning scripts where available. In the inference process of Unee following the computation method in (Lin et al. 2024a), we compute the maximum and minimum activation parameters for Une under the Top-2 strategy due to the different number of parameters included between different experts in the sequence-leve MoE. The total number of parameters for UniF2ace ace is 1.84B, the maximum activation parameter is about 1.63B, and the minimum activation parameter is about 1.42B. The average number of activation parameters tested in the 2nce-- M test dataset is 1.47B.
20

---

Published as a conference paper at ICLR 2026
D ABSORBING-STATE CASE WITH INDEPENDENCE BETWEEN TOKENS.
The absorbing state case means that for any single token with possible values in X (I,...,NK the transition matrix is
0
1
Qubsarb
(15)
-1
1
0
The reverse transition rate matrix of the reverse process from state X₁ state Sn is
(+(
(Xt, xe),
8 + Xt.
Qi (x,,
(x()
(16)
Xepix Q: (Xe, ())
& Xt
As Qi (Re, x1) is known, sufficient to estimate the concrete score
q.lR.) by a score network qx(x;)
se (xe, t)
(8..
Score based discrete diffusion model an effective objective to train
(x+)
de
&
the score network (Meng et al. 2022. Lou et al. 2023). Specifically, the score function in
multidimensional discrete space
@t (x] 2 x at [x] ... x
(xi, 1) 80 (x/... x x9 [i, xi]]
(17)
and accordingly,
Q (x.... x² x4 x/... x so Q: (x), 80 (x....x) xf x4, [1.
(18)
E PROOF OF THEOREM 1
To prove Theorem 1 we first introduce two loss formulas and establish useful lemmas.
(1) L1 = Excure (se) D (ęrio (- xo) [[ where Lecore (Xo) is the diffusion weighted denoising score entropy for data point Xg, and 0xx,1807
go xx)
Lacore (80) = /" F (xo)
Σ
Qe (xe, (so (Xe:tly
x#X.
(19)
2r (y xo)
2rjo (y xp))
))
log (x1, t)y K
Ø:jo (x: xx) dt.
qt (Xe xp)
(2) -== - E (xg) [log Pa (Xn
x, ) C. where C is a constant independent of
the model parameters. By (Xie et al. 2024 C C C2, and constants C₁ and C2
21

---

Published as a conference paper at ICLR 2026
shown as:
C1 = [
Σ log q (xt (x-1) +
log (XT)
Note ((xx))=((0))
(xxo))
Σ log (x,, xx-1)
log y (xx)
C₂ (g((o))
log (xt-1 xx)
(x(x)))
ΣΣ a (8o xx-1) log (Ro xx)
&
Σ log X1-1) Σ log (xxx)
E ( logg (Xu xx)]
(20)
C1 + C2 (xxo))
log (xo)
Σ 1=1
log q (xo xx)
(21)
Let L = (Xo)) be the model's log-likelihood for a data point xo, and let K be its variational
lower bound:
Po (Xo:T-1 xT) lxgg((T))
K E (x)))g((1)) xx)))
log
(22)
(XI.T xo)
Then, the following inequality chain holds:
L K = - -22,
(23)
where L1 and L2 are defined as above.
The proof is based on two applications Jensen's incquality to the log-likella log-likelihood.
1. Proving L 2 K
This the standard variational lower bound for diffusion models, derived by applying Jensen's inequality:
log Po (xx)) = log / Po Po(Xo.T) dxx
[oog Ee(xxrrbo))
po (X0:T--- (x))po(XT)
Eg(x))
(24)
q(x1:1 Xp)
log px(X0:T-1 xT)
By(xo:r)
+ log (XT)
= K
g(x1: xo)
Here, we assume po(x7) ss g(XT). This proves the first part of the inequality. Proving K -Li
po ( (xe)
+ log po (x())
K
By(xo-))
log
q (x Xx 1x)
(25)
T
log pa (x (xx)
Σ
E By(xa:r)
+ By(xoor)) [[og)o(X)]]
g(xx 1
22

---

Published as a conference paper at ICLR 2026
From the derivations in (Sohl-Dickstee et al. 2015), the variational lower bound K can be strictly rewritten in terms of KL divergences, which directly correspond to our L1. Specifically.
K Σ dx (Xo, KL (g (Xy-1 Xo) [xp-x11 xx))
(26)
+ Hg (xr xo) Hg (x1 xo) (xT)
Since
[
Hg (XT Xo)
Hp (xx)
q (xr
logg (xr xo) dx
xr Jx0
[ /
(xx1 xx) q (xx) dxn log (xT) dxr
Xx Xo
(27)
q (XT Xo) q (Xo) log 9 (XT xo) dxodxr
(xr)
Exrau (Xo)
[KL (q (XT xo) lpp (XT)]
and Hg (x1 x0) Eyra Ey(ar (xn)
[log Poja (xo
[2]]] then the above formula is equivalent
-Exe (-1x0) [DLL xPxx
x2) IP0 ( xx))]] DKL. (gr o ( xx) l|TT)
(28)
which is equal to L,, according to (Lou
2024).
3. Proving K 2 22
The derivation of L2 involves a second application of Jensen's inequality on K:
Pe (x,-1 xx) log
K = Ex((x.r)
+ log (xr)
q (x+ x(-1)
Σ log Σ q(xx-_1 Xx,, 1o) xx1) ]] + Ci
(6)
T
g(xx_11 xx)
Do (Ro xxi)
(29)
(x(rr))
(80 x(-1) log
+
{(80 xx)
[=1
Σ
Egtx, x 1o xx)] + C1 + C2
In summary, - log po (xn) S L1 S L2 holds.
F IMPLEMENTATION OF THE RESAMPLER
We define resampler S I + RLDD where is the length of the input vector, the length of the sequence and D is the hidden dimension of UniFPace. Specifically, we define learnable hidden latent matrix:
Mo E Lxd
Mo LearnableParameter
(30)
where df the hidden dimension of the resampler. Its process involves:
Project the noise embedding x E R via
H xW. in
B Ild
(31)
23

---

Published as a conference paper at ICLR 2026
2. Iteratively refine the latent matrix through T layers, suen as the I-th layer:
Mj = M, 1 + MHA (MІ-1, Concat (H, Mr-1)
(32)
Mc M FFN(M))
(33)
where MHA denotes the Multi-Head Attention mechanism, FFN denotoes the Feed-Forward Network. In MHA, the query, key, and value denoted as:
Q M M
(34)
Ki HH; Mr W K
(35)
Vi [H; Mr W/
(36)
Project the final latent to the output space:
Y LayerNorm W Won) R
(37)
This enables adaptive fusion of input vector into sequence features through learned latent queries.
G TRAINING COST DETAILS
For analysis of training cost, we define the "simplest baseline" as model trained directly without any of our proposed methods. As shown Table [1 the detailed cost breakdown is as follows: (I) D3DifT: The introduction of D3DifT, novel loss function, results in modest +5.8% increase in training overhead, while achieving substantial improvements in generation performance as measured by VQAscore and VLM-score. (2) Multi-level MoEs: The multi-level MoEs expands parameters from 1.3B to 1.8B, resulting in a 40.7% increase in training cost. However, this is a worthwhile trade-off as the overhead scales nearly linearly with the parameter growth and, importantly, delivers substantial improvements in both generation and understanding capabilities.
Table 11: Ablations of training cost vs. performance.
Generation
Understanding
Method
Params
GPU Hours
VQAscoret
VLM-scoref
Desef
Conv
Baseline
1.3B
117.4
0.855
79.698
4,958
6.037
UniF2ace (w/ D3Diff)
1.3B
124.3
0.878
84,432
4,988
6.031
UniF2ace (w/ MoEs)
1.8B
165.2
0.879
85,993
5.885
6,427
UniF2ace (Full)
1.8B
175.3
0.894
88.049
6.023
6.532
H
PERFORMANCE ON THE NOISY INPUTS
Evaluating model robustness under degraded conditions is paramount for transitioning Multimodal Large Language Models (MLLMs) from idealistic, controlled environments to unpredictable real- world applications. However, given the scarcity of datasets containing authentic, naturally occurring facial noise, we employed synthetic noise simulations to rigorously assess performance stability. Specifically, we injected Gaussian noise into the RGB space with a mean (p) of 0 and standard devi- ation (r) of 25. This controlled degradation effectively emulates prevalent real-world artifacts-sun such as motion blur, environmental shadows, and partial acclusions-wie ensuring that fundamental facial identity and structural features remain discernible for analysis.
We conducted comparative analysis of face understanding performance for the aforementioned noisy inputs. Following the evaluation metrics established in the paper, assessed both Description (Desc) capabilities-i.e. Captioning and Conversation (Conv) capabilities- i..., Visual Question Answering (VQA). The experimental results, shown in Table 12] unequivocally demonstrate that 2 exhibits strong robustness to input noise, achieving the best overall understanding performance across the majority of evaluation metrics degraded face images.
I THE USE OF LARGE LANGUAGE MODELS (LLMS)
LLMs were used solely to aid in writing and polishing the text (e.g., improving clarity and grammar), with all outputs verified by the authors.
24

---

Published as a conference paper at ICLR 2026
Table 12: Performance comparison under synthetic noisy inputs.
Model
Params
Desc-GPT
Conv-GPT f Desc-DS
Conv-DS
Qwen2-VL. Wang et al. 2024a
7B
5.29
4.76
6.03
5.84
Qwen2.5-VI Bai et al. ace ace (Ours)
2025
3B
5.38
5.61
6.21
6.14
1.8B
5.43
5.58
6.37
6.29
SDXL
TokenFlow
OmniFlow
Show-o
UniF2ace(Ours)
"A male with shart, slightly touiled groy hair and distinct Hit foce characterizg heizzed broad forsheed and protinent chaskbanes. The made straight and well-defned, The skin tone with natural Tecttuee and visible lines, indicative age and life experienos."
The image festures male with receding hairline and shart, gray hair. He has mustache and neatly trinned beard. The person wearing thin framed syeglesses that rest on his nose, His expression appears serious or conterplattive, with clesed les and focused eyes: The lighting highlights the contours of his face, emphosizing the forchead and cheekbones. The background is slightly blurred, festuring on American flog. suggesting formal official setting. The overall composition conveys a sence growity and Fecus
M
PI
"The person The image has worm and friendly expression, characrerized gantle snile that conveys happiness They with lang hair partially covered by patterned headscorf in shades green and blux, adding clegance to their appearence The decorative headpiece fectures the forehend. Their wapring lpstick, which enhances ther smile, and
female and have skin,
control red and dongling eluments facial Faatianes. The sepeistiiaaioon, The lighting highlights the stooth texture ther skin, enhancing the natural besuty of their face."
accentustedd ted with dark cysliner and they have orched sysbrows that complement their sarrings that odd touch of
nose, Adá tianally,
Figure 8: More comparison of generated face images with other models, Fine-grained attributes
highlighted in the prompt.
25

---

Published as a conference paper at ICLR 2026
Figure 9: More face images generated by UniF2ace
Q: Break down the elements of the image in detailed manner
The image captures close-up portrait womon, shade of blonde, She is adorned with pair of earnings, subtie accessory that complements her overall look, Her gaze is directed straight at the camera, creating sende of connection with the viewer: The background is stark block, providing striking text or additional objects. The relative pesition of the womon the image minimalist aesthetic The image is study in contrasts the blonde hair ogainst the black background. the white blouse against block background, and
filling
frame.
blonde hoir white
The image does contain discernible frome and the ebsence other elements give
direct gaze against the surrounding dorkness.
score::3
The image features a person with short, blonde hoir styled in neat strolskt monner. The hoir parted slightly off off-center, and the individual weaning white, high collared gorment, The background plain and dark, which helps to highlight the sub subject. The of direct engagement subject's face and hair, without casting harsh shadows. The everall composition simple and focused, emphasizingg the subject's facial features and attire. :score:63
Qwen2- VL:
person looking dinectly
comera, giving
lighting soft, evenly illuminating
wearing white comero, giving confident and professional appearence, The womon appears to be well-dressed and polsed, possibly for formal event or professional setting. Sccore:43
5: image features woman
necklace. is looking directly
InternVL2 5:
The image coptures close-up portrait of woman with light skin and short, straight blond hair. Her hair falls just above shoulders, framing her face soft at the Her lips. wears makeup that
light shade, possibly hazel or blue, and she has slight smile mascara and possibly neutral lipstick, enhancing her natural features without appearing overly done, The woman's s ears are adorned with small, round earrings that catch the light, suggesting they may be made of a metollic or pearl- like material, The background not visible, focusing attention solely the woman's s face. There disceriibe texts, symbols,
objects image. Sscore: 73
OmniFlew: womon with blonde hoir and bongs. score:6:
features woman with short, blande hair styled layered Her ported side, and she has few stronds froming her face, wearing small, round earrings, background is dork, which controsts light-colored and clothing. The woman is wearing white or light-colored top with high collar, which oppears to be made of textured or patterned fabric. Sscore:6>
JonusFlow:
Show-o:
The image features a woman with blonde hain and white collar, weaning a white dress, She looking directly at the camera, her steady and confident. The background is dork, providing stork controst to her bright ottire, The womon's 's expression and pose magazine
impression professional portrait, possibly
professional setting. :5core:5s
(anuus) The image features a female with short, blonde hoir styled in loyered cut with bangs, Her hair appears wavy, she hos a receding hairline, She has a foir complexion and smooth skin texture, Her well groomedd arched eyebrows frome expressive eyes that highlighted long loshes. The person has straight nose and lipstick, contributing to her balanced and symme trical face. Her confident demeanor with direct gaze, weaning carnings, and her natural features beautifully. Sccore:
expression neutral, exuding
makeup enhances
Figure 10: Comparison of captioning results and SeepSeeek-v3-baseee scores. We highlight fine- grained attributes with blue and errors answers with red.
26

---

Published as a conference paper at ICLR 2026
Step1: Prompt for generating initial captions:
User: "In the image there is a person, describe the image in paragraph giving detailed fine-gre inningr attributes of the person face. [Note that the output is mixed with the captions given below: xxxxx]
Step2: Prompt for generating Fine-grained captios:
User:
"Please combine the face caption you just replied to and the following features into paragraph:
Appearance: xxx, xxx, xXx, xxX.....
Action: xxx, xxx, xxX.....
Emotion: xxx
Step3: Prompt for generating fine-grained VQAs
User:
You are an AI visual assistant, and you are seeing face image. What you see are provided with paragraph, describing the same image you are looking at. Answer all questions as you are seeing the image.
Design conversation between you and a person asking about this photo. The answers should be a tone that a visual AI assistant is seeing the image and answering the question.
Ask diverse questions and give corresponding answers.
Questions cover many face attributes as possible, such as hair, nose, eyes, mouth, ears, skin, eyebrows, adornment, and so on. Only include questions that have definite answers:
(1) one can the content in the image that the question asks about and can answer confidently: determine confidently from the image that it is not the image. not ask any question that cannot be answered confidently.
(2)
image, for example, asking whether the person the image has earrings, asking whether is the hair the person the image long or short, etc. Again, do not ask about uncertain details.
Also include dlssed-ended questions that are relevant
content
Also include complex questions that are relevant to the content in the image, for example, asking about the action and emotion of the person in the image, asking to discuss about events happening in the image, etc. Again, do not ask about uncertain details.
Provide detailed answers when answering complex questions. For example, give detailed examples or reasoning steps to make the content more convincing and well-organized. You can include multiple paragraphs if necessary.
Please return the results in the following json format:
Example:
("from": "human", "value": "Can you describe his eyes and eyebrows?"),
("from": "gpt", "value": "Certainly! eyes are deep-set and expressive, and bushy dark eyebrows complement them well, enhancing his expressive appearance. "}, ("from": "human", "value": "What color are her earrings?"), "value": "She wears gold earrings."), ("from": "galu""valuee".
Figure 11: Prompts for building dataset. The first and second prompts are to GPT-4o, while the last prompt GPT-4. In the first prompt, the content in "[]" is used only when the image data includes built-in captions, such as in the MM-CclebA-HO dataset.
27


Reviews:
[{"review_id": "ZtxyLiahdU", "reviewer": "Reviewer_9L5B", "summary": "This paper presents a unified multimodal model that integrates fine-grained face understanding and generation into a single framework. It introduces a dual discrete diffusion loss and multi-level grouped MoE architecture. Which combines token-level and sequence-level expert routing to preserve semantic and identity features during multimodal representation learning. Additionally, the authors introduce a large-scale dataset (130K image caption pairs and 1M VQAs) designed for fine-grained facial attribute learning. Experiments demonstrate state-of-the-art performance on both generation and understanding tasks.", "questions": "Please refer to the Weaknesses.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1.\tThe proposed D3Diff loss provides a principled unification of score-based and masked generative modeling.\n2.\tThis paper proposes a multi-level grouped MoE structure effectively separates semantic and identity embeddings,.", "weaknesses": "1.\tThe authors should more clearly justify the necessity of a unified model for fine-grained face understanding and generation. The current motivation might not be accepted by the community, as expert models specifically designed for face understanding and generation already handle these tasks well.\n2.\tThe study’s exploration of fine-grained face understanding appears limited, primarily focusing on attributes such as hair and beard. This narrow scope raises concerns about the generality and depth of the proposed approach. Furthermore, the paper overlooks several recent works that specifically address face understanding, such as:\n[1] FaceXBench Evaluating Multimodal LLMs on Face Understanding\n[2] FaceInsight: A Multimodal Large Language Model for Face Perception\n[2] FaVChat: Unlocking Fine-Grained Facial Video Understanding with Multimodal Large Language Models\n3.\tThe motivation for sequence-level MoE layers and token-level MoE layers is not very clear. Why is a token-level MoE layer needed, and what role does it play in face understanding or generation tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "znlynMzUJg", "reviewer": "Reviewer_7WgR", "summary": "This paper proposes a unified multimodal large language model framework for facial understanding and generation in an auto-regressive plus diffusion-based style. The paper develops a novel multi-level, grouped mixture-of-expert-based architecture, incorporating the advantages of semantic and identity features from CLIP and facial experts. To facilitate fine-grained facial understanding and generation, a fine-grained facial image-caption dataset, uniF^2aceD-1M, containing large-scale visual question-answering pairs across diverse facial attributes, is simultaneously constructed to train the model. Additionally, a novel dual discrete diffusion (D3Diff) loss is proposed to enhance the quality of generated images.", "questions": "1. Do the authors have any plan to scale up the model size (e.g., larger than 7B) to boost the performance further?\n2. Is there any open-source plan for the training, inference, evaluation codes, and the checkpoints?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. The paper is well-written and easy to follow.\n2. The paper is the first one to propose a unified multimodal framework for both facial understanding and generation, supervised by a compound loss for next-token textual generation and diffusion-based image generation.\n3. A novel architecture, based on a multi-level grouped mixture of experts, incorporating semantic and identity features, is proposed to enhance the model's understanding of fine-grained facial attributes.\n4. A novel dual discrete diffusion (D3Diff) loss with a theoretical guarantee is proposed to improve the generation quality further.\n5. Thorough experimental results demonstrate the effectiveness of the proposed method in both facial understanding and generation benchmarks.", "weaknesses": "1. The evaluation of fine-grained facial understanding should include the result in a challenging benchmark, FaceBench, developed for fine-grained facial understanding across over 210 facial attributes.\n2. Concern about the fairness in evaluation. Were other methods pre-trained on the UniF^2ace training set before evaluation, or were they evaluated in a zero-shot manner? If they are evaluated in a zero-shot manner, their performances are certainly below the upper bound compared to the proposed method, which is pretrained on fine-grained facial understanding and generation data.\n3. It is better and strongly recommended to include the user study for both facial understanding and generation.\n\nFaceBench: A Multi-View Multi-Level Facial Attribute VQA Dataset for Benchmarking Face Perception MLLMs. CVPR 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6pc517gSjI", "reviewer": "Reviewer_PpVm", "summary": "This paper presents UniF2ace, a unified multimodal model (UMM) that integrates fine-grained face analysis and synthesis within a single framework. Additionally, the paper introduces UniF2aceD-1M, a new large-scale dataset featuring 130K high-resolution images and 1M fine-grained Visual Question Answering (VQA) pairs, annotated with an unprecedented average of 17.7 details across 46 distinct attributes.", "questions": "The ablation studies clearly show that a Top-K=2 configuration is optimal for the MoE routing. Is there a deeper, non-empirical reason for this? Can the authors provide a more intuitive or theoretical explanation for why using two experts performs better than using a single expert or dispersing the load among more (e.g., K=4), especially in the context of balancing semantic and identity embeddings?\n\nThe dataset UniF2aceD-1M combines CelebV-HQ, FFHQ, and MM-CelebA-HQ, among others. Could you elaborate on the data licensing and consent aspects, especially for identifiable human faces? Were any filtering, de-identification, or bias auditing steps performed (e.g., balancing across demographics)? Will dataset release include ethical usage guidelines or restricted licensing to prevent misuse (e.g., deepfakes, impersonation)?\n\nThe dataset is described as “fine-grained” but primarily based on high-quality celebrity-style images. How diverse is UniF2aceD-1M in terms of age, ethnicity, lighting, and pose variation? Do you observe any performance degradation when tested on non-celebrity, low-quality, or occluded faces?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper makes a strong theoretical contribution through the proposed Dual Discrete Diffusion (D3Diff) loss, which is formally derived to offer a tighter upper bound to the Negative Log-Likelihood (NLL) compared to conventional masked generative losses, moving the work beyond purely empirical engineering.\n\nArchitecturally, the model is innovative, employing a multi-level grouped Mixture-of-Experts (MoE) design with distinct token-level and sequence-level routing. This structure, which utilizes specialized CLIP and Face experts, provides both interpretable specialization and computational efficiency via sparsity.\n\nA substantial dataset contribution is also provided with UniF2aceD-1M. This resource is highly valuable due to its diversity, high quality, and rich dual supervision (captioning and VQA) across fine-grained face attributes, significantly improving upon predecessors like FFHQ-Text.", "weaknesses": "While the ablation studies (e.g., D3Diff in Tab. 5, MoE in Tabs. 6–8) are technically sound, the paper provides limited practical intuition for the observed improvements. Specifically, the rationale behind optimal hyperparameter choices, such as why a Top-K=2 configuration consistently outperforms others, is presented only as an empirical result. A deeper, theoretical exploration of the hyperparameter sensitivity and architectural dynamics would greatly enhance the contribution.\n\nGiven the model's focus on generating and analyzing human facial data, the ethical discussion is critically minimal. The paper must include a dedicated and robust discussion addressing potential issues related to bias, privacy, and misuse, such as the generation of deepfakes or facial profiling. Furthermore, the dataset sourcing and consent must be clarified. Since UniF2aceD-1M is sourced from web-scraped and public datasets (FFHQ, CelebV-HQ, MM-CelebA-HQ), the authors must provide explicit details on data licensing, subject consent, and any filtering mechanisms used.\n\nThe current empirical evaluation lacks diversity. All benchmarks rely on ideal, celebrity-style, or synthetic datasets. The paper fails to test the model’s robustness and generalizability across real-world conditions, such as faces with occlusion, low-light conditions, or poor image quality. This limits confidence in the model's real-world applicability.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lc5pahoWX6", "reviewer": "Reviewer_WDVe", "summary": "The paper introduces UniF2ace, a model that unifies face understanding (captioning, VQA) and generation (text-to-image) in one framework. It combines a new Dual Discrete Diffusion (D3Diff) loss with a multi-level Mixture-of-Experts design, and introduces a large dataset (UniF2aceD-1M) with detailed captions and 1M VQAs. Experiments show it outperforms state-of-the-art models on both tasks, even against larger systems.", "questions": "1. How balanced and unbiased is the dataset?\n2. How well does the model handle real-world noisy inputs or domain shifts?\n3. What is the computational/training cost compared to simpler baselines?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. First unified model for fine-grained face understanding + generation\n2. Novel D3Diff loss improves detail and fidelity\n3. New large-scale dataset with rich annotations\n4. Strong results across benchmarks, with thorough ablations", "weaknesses": "1. Limited discussion of ethical risks (deepfakes, misuse)\n2. Dataset bias and annotation quality not fully addressed\n3. Training appears complex and resource-heavy", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)