UniF^2ace: Fine-grained Face Understanding and Generation
  with Unified Multimodal Models
Authors: name: Junzhe Li, name: Xuerui Qiu, name: Linrui Xu, name: Liya Guo, name: Delin Qu, name: Tingting Long, name: Chun Fan, name: Ming Li

Unified multimodal models (UMMs) have emerged as a powerful paradigm in
foundational computer vision research, demonstrating significant potential in
both image understanding and generation. However, existing research in the face
domain primarily focuses on coarse facial attribute understanding,
with limited capacity to handle fine-grained facial attributes and
without addressing generation capabilities. To overcome these limitations, we
propose UniF^2ace, the first UMM tailored specifically for
fine-grained face understanding and generation. In general, we train
UniF^2ace on a self-constructed, specialized dataset utilizing two
mutually beneficial diffusion techniques and a two-level mixture-of-experts
architecture. Specifically, we first build a large-scale facial dataset,
UniF^2ace-130K, which contains 130K image-text pairs with one
million question-answering pairs that span a wide range of facial attributes.
Second, we establish a theoretical connection between discrete diffusion score
matching and masked generative models, optimizing both evidence lower bounds
simultaneously, which significantly improves the model's ability to synthesize
facial details. Finally, we introduce both token-level and sequence-level
mixture-of-experts, enabling efficient fine-grained representation learning for
both understanding and generation tasks. Extensive experiments on
UniF^2ace-130K demonstrate that UniF^2ace outperforms
existing UMMs and generative models, achieving superior performance across both
understanding and generation tasks.