OmniPortrait: Fine-Grained Personalized Portrait Synthesis via Pivotal Optimization
Authors: Dongxu Yue
Bo Lin
Yao Tang
Jiajun Liang
Zhihai He
Chun Yuan
Venue: ICLR 2026 Poster (2026)
Keywords: Personalized Portrait Synthesis
ImageGeneration
High-Fidelity Facial Details
Pivotal Optimization
Primary area: generative models
TL;DR: By leveraging pivotal optimization, identity-customized portrait synthesis is achieved with highly faithful fine facial details while preserving editability.

Abstract: Image identity customization aims to synthesize realistic and diverse portraits of a specified identity, given a reference image and a text prompt. This task presents two key challenges: (1) generating realistic portraits that preserve fine-grained facial details of the reference identity, and (2) maintaining identity consistency while achieving strong alignment with the text prompt. Our findings suggest that existing single-stream methods fail to  capture and guide fine-grained identity details.
To address these challenges, we introduce \textit{OmniPortrait}, a novel diffusion-based framework for fine-grained identity fidelity and high editability in portrait synthesis. Our core idea is pivotal optimization, which leverages dual-stream identity guidance in a coarse-to-fine manner. First, a Pivot ID Encoder is proposed and trained with a face localization loss while avoiding the degradation of editability typically caused by fine-tuning the denoiser. Although this encoder primarily guides coarse-level identity synthesis, it provides a good initialization that serves as the identity pivot for optimization during inference.
Second, we propose Reference-Based Guidance, which performs on-the-fly feature matching and optimization over diffusion intermediate features conditioned on the identity pivot. In addition, our approach is able to generalize naturally to multi-identity customized image generation scenarios. Extensive experiments demonstrate significant improvements in both identity preservation and text alignment, establishing a new benchmark for image identity customization.

Published as a conference paper at ICLR 2026
OMNIPORTRAIT: FINE-GRAINED PERSONALIZED PORTRAIT SYNTHESIS VIA PIVOTAL OPTIMIZATION
Dongxu Yue', Bo Lin', Yao Tang², Jiajun Liang?, Zhihai He Chun Yuan'*
'Tsinghua University, China JJIOO Technology, China Southern University of Science and Technology. China ydx25emails ts tsinghua. edu.cn
ID-Reference image
A photorealistic image of a woman wearing headphones with auburn hair in a living room.
Instant-11)
PakteMaker
Ours
(a) Comparison on Facial Detail Generation
(b) Seentit--onsisston Portrait Generation Prompt
Ours (single)
Ours (multiple)
Prompt
woman'man
A woman and a man are wulking near lake
riding horse in front of medieval castle
OMM
woman/man wearing a jeun jacket spraying paint wall
woman man have dinner together
We demonstrate (a) the challenges faced by existing methods in customizing fine-grained facial features, and (b) that OmniPortrait able to generate images with exceptional identity simi- larity, strong text-image alignment and can be extended to multi-identity customization.
Figure I:
ABSTRACT
Image identity customization aims to synthesize realistic and diverse portraits of a specified identity, given a reference image and text prompt. This task presents two key challenges: (1) generating realistic portraits that preserve fine- fine-grained fa- cial details of the reference identity, and (2) maintaining identity consistency while achieving strong alignment with text prompt. Our findings suggest that exist- ing single-stream methods fail to capture and guide fine-grained identity details. To address these challenges, we introduce OmniPortrait, a novel diffusion beafusioo-bbsedd framework for fine-graines identity fidelity and high editability in portrait syn- thesis. Our core idea is pivotal optimization, which leverages dual-stream iden- tity guidance in coarse-to-fin manner. First, Pivot ID Encoder is proposed and trained with a face localization loss while avoiding the degradation of ed- itability typically caused fine-tuning the denoiser. Although this encoder pri- marily guides coarse-level identity synthesis, it provides a good initialization that serves as the identity pivot for optimization during inference. Second, we propose Reference- Based Guidance, which performs on-the-fly -ly feature matching and opti- mization over diffusion intermediate features conditioned on the identity pivot. addition, our approach is able to generalize naturally to multi-identity customized
"Corresoonding authors.

---

Published as a conference paper at ICLR 2026
image generation scenarios. Extensive experiments demonstrate significant im- provements in both identity preservation and text alignment, establishing a new benchmark for image identity customization.
INTRODUCTION
Recent text-to-image generation models Xu et (0118) Dhariwal & Nichol (202T): Rombach al. (2022) Nichol et al. (2022);
3 Yu et al. (2025b); Saharia et al. (2022); Li ct al. (2024a). especially dif-
fusion models Ho et al. (2020); Song et al. (202T): Yue et al. (2023); Chen et al. (202a)) Esser
et al. 2024); Guo & Lin 2204), have made significant developments. Stemming from T21 diffu-
sion models, personalized portrait synthesis task is achieved by injecting identity conditions into the conditional space of the base model, enabling the synthesis of realistic and diverse portraits for specified identity. Recent advances in customized generation have explored two main directions:
test-time fine-tuning Ruiz et al. (2023); Gal et al. (2022): Kumari et al. (2023), and design ad- ditional encoders in the conditional space of diffusion models Gal et al. (0033)
g Wei et al. (2023);
Arar et al. 2023) Valevski al. (2023); Chen et al. (2023b); Ma et al. (2023a): Yue et al. (2024).
Despite significant advancements in customized generation, two major challenges remain for syn- thesizing Seenttty-personalizee portraits: (1) generating realistic portraits that faithfully preserve the fine-grained facial details of the reference identity, and (2) maintaining strong identity consistency while ensuring precise alignment with the text prompt. As illustrated in Fig. (a), existing state-of- the-art methods Wang et al. (2024); Li et al. (2024b) struggle to faithfully preserve the facial details of the reference identity image. The loss of critical facial details, such as beauty marks, may give the impression of an overly retouched or even fake photograph. thereby hindering their practical appli- cability. Recent works, such as FastComposer Xiao al. (2023), attempt to enforce the presence of the reference portrait by full fine-tuning. However, it damages the rich priors of pretrained diffusion models and often leads to imbalanced visual fidelity and editing flexibility, as illustrated in Fig. 2
Inspired by PTI Roich et al. (2022), our core idea differs from existing single-stream conditioning methods by introducing pivotal optimization, which leverages dual-stream identity guidance a coarse-to-fine manner. We start by introducing Pivot ID Encoder, vision encoder that takes the reference identity image as input and injects identity features into the conditional space of the denoising network, thereby establishing pivot for portrait generation. To train the Pivot ID Encoder, we design a face localization loss that encourages the identity embedding of the reference image to focus on the facial region. This not only enhances identity similarity but also enables more accurate localization face during inference. Throughout training, the parameters of the base model remain frozen to ensure text alignment capability is preserved without degradation.
Although the Pivot ID Encoder alone provides only coarse-grained identity consistency, it estab- lishes strong initialization that serves as the identity pivot for optimization. Building on this, we introduce Settrence-BBseee Guidance (RB-Guidance), which performs feature-space matching be- tween the diffusion intermediate features of the reference identity image and those of the generated image, thereby enabling pivotal optimization guided by identity features. Thanks to the Pivot ID En- coder design, the rich priors and text alignment capabilities of the base model are preserved, while identity and facial details from the reference image are guaranteed through RB RB-Guidance. More- over, OmniPortrait plug-and-plas and is readily compatible with many other methods or plugins, while naturally extending to multi-identig customization scenarios, shown in Fig. (bb.
Equipped with these techniques, as shown in Fig. [ OmniPortrait demonstrates extraordinary cus- tomization capabilities with just single reference image and achieves state-of-the-ann results, show- ing a significant quality advantage over prior works in similar settings. In summary, our contribu- tions follows:
We propose OmniPortrait, a pivotal optimization based framework for detail-preserving identity-customized portrait synthesis. It incorporates Pivot ID Encoder and Reference- Based Guidance to enable coarse-to to-fee personalized portrait generation, enhancing iden- tity similarity with minimal impact on generative capability.
We design Reference- Based Guidance (RB- (RB-Guidance), novel test-time optimization framework based on local diffusion feature matching, which enables training-free preser- vation of fine-grained identity details.
2

---

Published as a conference paper at ICLR 2026
Input
Prompt
0.5
a 0.6
R 0.7
0.8
A min riding
horse
Figure 2: The trade-off between identity fidelity and text editability in FastComposer 2023
Xiao et al.
controlled by its delay condition parameter
We construct large-scale human face dataset with detailed annotations, containing mil- lion data pairs, which can facilitate training for identity-customize image generation tasks.
Extensive experiments demonstrate the excellent performance our framework particu- larly in preserving fine-grained facial details, Results show that our method achieves supe- rior identity fidelity without compromising editability.
2 RELATED WORK
Text-to-Image Diffusion Models In recent years, diffusion models Ho et al. (2020); Nichol &
Dhariwal 202T) have rapidly advanced image generation, with models like Stable Diffusion Rom- bach et al. (2022). DALL-E2, and Imagen Rombach et al. (2022); Saharia al. (2022) producing
high-quality images. More recent models Peebles & Xie (2023): : Yu et al. (2025a); Chen et al.
(023a); Guo al. 2025), Esser et al. (2024) adopt Transformer-basee denoising networks and
leverage larger datasets, resulting in stronger generalization and enhanced image generation.
Few-shot Finetuning Customized Generation Subject-driven image generation enables models to create images with personalized content. Some methods Ruiz et al. (2023); Gal et al. (2022); Kumari al. (2023) fine-tune models multiple reference images capture specific subject or face. However, these approaches are constrained by the requirement for several images of the same subject and the high computational costs Wang et al. (2025a):
al. (2025).
Encoder-Basee Customized Generation Recent works Gal et al. ): Wei et al. (2023): Ma et al. y Arar et (2023) Ye al. (2023); Xiao et (2023): Guo et al.) (2024);
(203b)) Shi al. (2023
Qian et al. (2025); Mou et al. (2025) Jiang et al. (2025); Huang et al. (2024) use visual encoders for efficient image synthesis but struggle with balaneing fidelity and editing flexibility. Building on reference image injection paradigm, subsequent studies have explored enhancements in multiple directions, including improved preservation of facial identity Li et al. (2024b) and extended support for multiple reference subjects Gu et al. (2023); ; Kwon & Ye (222).. InstandD Wang et al. (2024) consistent generation through dedicated face encoder and identity network.
enhances
Training-free free Consistent Generation Training-fire approaches have recently attracted significant attention owing to their computational efficiency. For instance, MasaCtrl Cao et al. (2023) proposed mutual self-attention, in which the keys and values of self-attention are substituted with those derived from the reference image. Similarly, ConsiStory Tewel et al. (2024) introduced subject driven self- attention, allowing each frame reference multiple subjects from different images in batch. They further incorporated DIFT [Tag et al. (202)) for feature injection in self-attention to improve detail consistency, Despite these advances, existing methods still struggle preserve the fine facial details of reference images, which remains a challenging problem worthy of further investigation.
3 METHOD
Given an identity reference image Øref and target prompt P, that specifies the desired scene, the objective is to generate an image aligned with the target prompt while preserving the fine-grained identity details of the reference face. Ideally, customized portrait generation should achieve high fidelity in identity preservation while simultaneously maintaining satisfactory alignment with the target prompt P.. To this end, OmniPortrait is introduced, which is built upon latent diffusion model and extended from energy- energy-based based diffusion guidance. Through Pivot ID Encoder and GuiGuidaecee, customized portraits generated in coarse-to-fine manner, the overview is shown in Fig.

---

Published as a conference paper at ICLR 2026
Pivot Encoder Training
A woman
Cross Attectionn
Text Enoader
in snoie
wearing doctoral cap Input
Lineaz
Mixed pivot embeddings
Pivor ID Encoder
"Pivot LD Encoder
Ding
Referenice Image
Zr-
Diffusion Diflusion Sampling
Reforence -aased Guidance
U-Net Denoiser
Furs
Decode
RB-Guidance
Facial Fenture Alignment
Pivotal Optimization
Add noise
U-Net Danoiser
RB-Guidance Compunents
Pivot ID Encoder Components
Trainable
*
Frovzn
Concot
Figure 3: Overviews of the proposed OmniPortrait, Pivot ID Encoder training pipeline and reference-Basets Guidance. We train the proposed Pivot ID Encoder with a face localization loss. During inference, the Pivot ID) Encoder kept frozen and injects identity information from the reference image into the denoiser obtain the identity pivot, In addition, 1B-GGudannce is introduced to optimize an energy function, thereby enhancing the preservation of facial identity features.
3.1 PRELIMINARIES
Latent Diffusion Model Sampling with Classifier-free Guidance The latent diffusion model is type of generative model that reduces images to low-dimensional latent representation using an autoencoder Kingma & Welling (2013); Wang et al. (2025b) and then iteratively denoise in the latent space. DDIM sampling is deterministic and non-Markovian process that generates image latents by progressively denoising in the latent space. Formally, the DDIM sampling procedure can be divided into two components: a denoising step based on Tweedie's formula and subsequent re-noising step. Given noisy latent z E h and text condition y, the update rule of DDIM can be expressed as:
21-1 = Vāc-1 (Ea t,y] + vI 11-168 (2,, t.)
(1)
Iee (Ze, t)) = (z: vI- āreg (z,, t.)) /Va,,
(2)
where denoiser is used predict noise eg (, y) from the noisy latent 2, specified time step i, given text condition y, where 0ç and a; are predefined functions of that determine the diffusion process. In addition, a₂ = 1 3r and ā; = II: in which B, E (0, is variance schedule.
Classifier-free guidance employed enhance the controllability of diffusion models. Accordingly, during training, the model is required to randomly drop the condition in ep (Zz, (,) with a certain probability. During inference, the predicted noise are formulated as follows:
Êy (z,) Eg (2,, Ø) + (Eg (z),t,y) Eø (x,,t 2)).
(3)
where w denotes the guidance scale, and ey (z,,4, 1, Ø) is the unconditional prediction.
Energy Diffusion Guidance From the continuous perspective of score-based diffusion, condition can be combined by a score function Vz, log (ze y) and can be further decomposed as:
q (y
2x) (xt) ((y)
2, log (Zt yy) log
Vz, log (zz) Ve, log 2t),
where Vz, log q (21) corresponding cœ (zt, t, Ø) and q (y 21) E (zittty) E is an energy function and it usually requires an additional model to estimate the conditional probability between the noisy latent z, and the condition y.
4

---

Published as a conference paper at ICLR 2026
3.2 PERSONALIZED IDENTITY PIVOT
Pivot ID Encoder As shown in Figure 3 Our goal is to associate the identity contained in the reference image with specific text token, such woman man, that most effectively represents the desired identity concept. To achieve this. A Pivot ID Encoder is deployed to extract visual features lref from the reference image Zref- To bridge the gap between visual embedding fref and text embedding Stzt, a linear projection layer is adopted as a feature alignment network. Specifically, we concatenate Cref and text embedding Ctrt before feeding them into the linear projection layer, and then we get Cmir as mixed pivot embeddings.
Training with Face Localization Loss Previous practices have attempted achieve high- high-fidelity face preservation by fine-tuning the denoising network. However, our observations indicate that such training compromises the priors embedded in the pre-trained model, which turn leads to abnormal scene composition and poor editability, illustrated in Fig. 2 Therefore, in this work, only the Pivot ID Encoder and a linear projection layer are trained, which not only enables efficient fine-tuning but also makes the Pivot ID Encoder plug-and-play.
Specifically, we employ an LLM Bai et al. 2023) to extract person-related nouns from the image captions. A face detection model and face parsing model are used to obtain cropped face images and corresponding face masks M. These elements are then combined to construct triplets consisting of caption noun, cropped face image and face mask. More details are presented in Section 4.1
To obtain an accurate facial region for subsequent pivotal optimization, is necessary to ensure that the facial mask of the generated portrait is available at the start of sampling. A straightforward way is to derive this mask from the cross-attention map of emia. However, our findings suggest that when the Pivot ID Encoder is trained merely with diffusion loss, the cross-attention of emiz tends to spread across the entire person rather than being restricted to the facial region. To address this issue, we propose a face localization loss to constrain the attention region of emiz- Specifically, let A, E [0, ()xxx)>* be the cross-attention map in sampling timestep t. where Ali. k] denotes the information flow from the k-th conditional token to the (i.j) latent pixel. Let m be the index of Emias and (m(m) = AlE, i, m] E [0, ().1))))) be the cross-attention map of the mixed pivot embeddings. We supervise the cross-attention map Am be close to the resized segmentation mask M, and the overall training objective can be formulated as:
L = -diff + SLoe
= -B-.u [Ike co (2., +
7
(Ae(i,)) M(i,5)) (5)
hw ==
where 17 = 2e-3 is hyperparameter, and we apply fioe at A.(m) E j0, ()16x10)
3.3 REFERENCE BASED GUIDANCE
Regional Mask Extraction By training the Pivot ID Encoder with the proposed facial localization loss, the cross-attention map Am(m) corresponding to emiz can be leveraged during inference to obtain the facial mask of the generated image. However, directly applying a threshold to A (m) can produce coarse-grained mask. We choose incorporate self-attention map rich in structural information, and design novel iterative refinement procedure, which can be formally expressed as:
Áç(m) Ag (m) norm (norm (..- ((.m)).....)
times
Here, norm refers to the min-max normalization to adjust the map value fall within the range (o, 1]. After y = 3 iterations with = 2, threshold 3 = 0.5 is then applied obtain the mask Myen.
Diffusion Feature Correspondence The human face in a portrait contains rich fine-grained details, which serve as the implicit basis for human perception of identity. Therefore, goal is to devise an energy guidance that explicitly focuses on the fine mne-grained facial features. Recent works Tang et al. 2023 have shown strong local correspondence between intermediate features in diffusion models which can be used for dense matching between different images. During the sampling process, the diffusion intermediate feature Doee enables the noisy latent code 2, timestep /. be directly
5

---

Published as a conference paper at ICLR 2026
mapped through the denoising network. Similarly, Fref by adding to steps noise, and we set tu = 671 in practice. Before the sampling process, a face parsing model is employed to extract the facial mask rf from the reference image 2ref- We first identify, within the feature spaces D and D the set of best-matching points Pgen within the mask M that correspond to the points Pref located inside the reference mask M
extract D from the reference image
Pgen argmind PEM
(D% [Prey]. D [pl)
where d(;;-) donates Euclidean distance. We then measuring correspondence by:
cos (Dgen [ggenl Diof (reel)) +
Saift
Σ
(8)
ErefEMref
Background Gradient Masking We aim to make the face details Iref closely resemble the one in the generated image. Thus, optimization objective is to maximize the similarity Sais (2e, T M9en, "_e,MM,ese]] The gradient has the same dimensionality as 2, allowing pixel- wise optimization. Nevertheless, this causes both the foreground and background simultaneously. so regions unrelated to the face are also influenced by the gradient, leading to im- age blurring. To address this issue, we apply mask M gen on Equation. [ to filter out the gradients corresponding to the background before performing the guidance step:
be modified
Doee [Pgenl. Dro (Presl) +1
cos
Sayfe =
Σ
o M
PrefEM
On the Fly Pivotal Optimization During inference, the trained Pivot ID Encoder takes single reference image as input. Although it is difficult to fully preserve the fine-grained facial details of the reference, it provides strong initialization and accurately localizes the facial region of the generated portrait. In practice, during the early denoising stages, the generated image only contains coarse-level concepts. This makes it challenging to establish precise matches between the reference image and the generated image, which in turn causes the guidance gradients of RB-Guidance to diverge. To address this issue, pivotal optimization introduced starting from timestep i uT and = 0.6, which achieves the best trade-off between guidance effectiveness and stability, as illustrated in Fig. Accordingly, We further formulate diffusion feature correspondence as an energy function:
p
(10)
+ Saift t Tref, Mgen, Mrrf)²
where denotes the hyperparameter that controls the strength of pivotal optimization and the pivotal optimization can be formulated as follows:
co (2,, t,, 2) (co (z,, ca (2,, Ø)) Ea (z,, t, 2) w (Ea (z,,ty) Eg (z,,t 3.E
ift i
Ey (zx)
(II)
<=
4 EXPERIMENT
4.1 DATASET CONSTRUCTION
Although customized portrait generation has been extensively studied, previous efforts typically re- lied on private datasets for training Wang et al. 2024)) Li et al. (2024b). Moreover, most existing multimodal face datasets only guarantee the existence of human faces within the images, while lack- ing precise annotations of individual body positions and facial regions. Motivated by these limita- tions, we construct and release OmniPortrait- IM, a large-scale high-quality multimodal face dataset with detailed annotations, which can support variety of personalized portrait generation tasks. We collect raw data from Pexels, COYO-7001 Byeon et al. (2022) and LAION-2B (Schuhmann et al. (2022), and further filtered by resolution and aesthetic scores. To annotate faces, YOLOv7-Face is used to detect all faces, after which images containing no faces or multiple faces are discarded. In addition, YOLOX Ge et al. (202T) is employed to generate bounding boxes for each person, and BLIP-2 Li et al. 2023) is used to produce image caption. Details are in the Appendix.
https: thub. com httbbbbco///rrrogg///// / cv face

---

Published as a conference paper at ICLR 2026
ID-Reference
Prompt
DreamBooth
I1
FastComposer
InstantID
IP-Adapter
PhotoMaker
Ours
"A woman posing like tourist in fromt of the Eiffel Tower (pose)
"A wearing police uniform" (clothing)
beach" (bockground)
"A man holding piece of cake" (action)
"A wearing
Christmas hot bright scene" (lighting)
"A painting of woman the style of Vincent Van Gogh " (style)
Figure 4: Qualitative comparisons. We compare our method with baseline approaches in terms of text alignment across several aspects, including pose, clothing, background, action, lighting, and style. OmniPortrait achieves the best text editability while effectively preserves the identity.
4.2 EXPERIMENTAL SETUP
Implementation Details The pretrained backbones utilized in our experiments include SD Rombach ct al. (2022) and SDXL Podell et al. (2023) based community models. For the Pivot ID Encoder, OpenCLIP- VTT-L/14 Cherti et al. (2023) is adopted. We train of the Pivot ID Encoder and the linear projection layer using the AdamW optimizer with a learning rate of 2.-5. SD-based Realistic Vision optimized for 80k steps on four NVIDIA A A100& 80GB GPUs. For SDXI.-based ReaI., images are filtered to 1024 resolution, followed by 60k training steps. To enable classifier-free guidance, randomly drop the text and ID condition with 10% probability. During inference, w is set to 7.5 and T = 1000 with 50 sampling steps, and the strengths of RB-Guidance is set to 8.5. We conduct the subsequent experiments using SDXL.
Evaluation Settings For evaluation, 50 reference images are sampled from the CelebA-Ho HHQ Karras al. (2017) and FFHQ Karras al.] (2019) datasets, and text prompts are constructed for each identity. To comprehensively assess the effectiveness of the proposed OmniPoratrait, three cate- gories of metrics are considered: (1)Tex Editability. CLIPP-T Hessel et al. (202)) and BL.I scores are employed to measure the consistency between the input text prompts and the generated results. (2)ldentity Fidelity. We report CLIP-I scores between the reference face image and the generated image. In addition, adopt RetinaFace Deng et al. (222)) the detection model, SIM represents the ID cosine similarity, with ID embeddings extracted by FaceNet Schroff et al. (2015). (3)lmage Quality. CLIP-IQA and FID Heusel et al. (2017) are adopted to evaluate perceptual quality.
4.3 QUALITATIVE COMPARISON
We compare with finetuning-base based methods, including DreahBooth Ruiz et al. (2023) and Textual Inversion Gal et al. (2022), as well as encoder-based methods, including Fastcomposer Xiao et al. 2023). InstantID Wang et al. (2024). Photomaker Li et al. (022b) and IP-Adapter Ye et al. (2023). Note that use the IP- Saapteer-Faeel-PPPe prepared model on a set challenging prompts. Fig. 4 shows the generated samples from our method and the baseline methods. It demonstrates that our OmniPortrait exhibits higher text-image
version for fair comparison. We test each well-
7

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative comparisons. "Plug and Play" refers to whether the method supports control conditions beyond text, such as pose keypoints and segmentation maps. Our method achieves opti- mal performance in both text alignment and identity preservation.
Text Editability
Identity Fidelity
Image Quality
Method
Plug and Play
BLIPt
CLIP-TT
CLIP-If
SIM
IQA1
FID
DreamBooth
X
78.42
23.32
66.45
60.07
85.88
219.40
Textual Inversion
X
76.36
23.89
58.32
58.32
71.73
235.57
FastComposer
78.87
22.98
69.83
64.97
82.04
223.94
PhotoMaker
X
80.51
23.97
67.64
62.68
70.51
236.93
InstantID
78.21
22.26
73.03
68.35
84.17
221.62
IP-Adapter
79.33
23.93
68.23
64.19
88.15
211.15
OmniPortrait (Ours)
80.24
24.25
73.08
69.13
86.80
213,48
alignment while capturing the face details of the identity in the reference face. IP-Adapter and FastComposer achieve relatively high facial fidelity but at the cost of poor text alignment. Their results often resemble copy-and -aste of the reference image, as shown in the last row of Fig.
4.4 QUANTITATIVE COMPARISON
Metric Evaluation Despite our method requiring only single reference image, to prevent catas- trophic overfitting of our baselines, we provide images for each ID to optimization-base meth- ods like DreamBooth. As shown in Table [ OmniPortrait outperforms all baselines in both iden- tity preservation and text-alignment metrics, indicating better text control capability while pre- serving the identity of the reference image. To further compare the performance of method and the baselines terms text alignment and identity fidelity, we conduct five sets of experi- ments for each method within their tunable parameter ranges, where each uses the same ref- erence identity image and prompt. As shown in Fig. 5 baseline methods, which rely on fine- tuning or single-stream identity encoding, struggle strike a balance between prompt consis- tency and identity preservation. In contrast, OmniPortrait achieves the best overall performance.
User Study We conducted user study to compare the proposed approach against base- line methods. Thirty participants from diverse backgrounds were recruited to evaluate 20 sets of generated images. Each set consisted of one reference identity image and five textual prompts, with every baseline method produc-
FampComposer
InstandID
OmmiPortain(Uurs)
Photomaker
PP-Adapter
0.30
0.28
Prompt Consistency
ing five corresponding predictions. Participants were asked to select the top-2 best-performing method according to four aspects: text align- ment, identity fidelity, facial detail preserva- tion, and the overall naturalness of the gen- erated results. We present the results the Appendix, the proposed OmniPortrait is con- sistently preferred over the baseline methods across all evaluation criteria. The advantage is particularly pronounced in facial detail preser- vation, where more than 50% of participants judged the generated portraits to exhibit a closer resemblance to the reference identity image. These results highlight the effectiveness of OmniPortrai in achieving superior balance between semantic editability and faithful identity preservation.
0.24
0.22
0.20
0.50
0.35
0.60
0.65
0.70
Identity Preservation
Figure 5:
Comparisons on prompt consistency and identity pereservation.
4.5 ABLATION STUDY
Pivot ID Encoder with Face Localization Loss We first perform an ablation study on the face
localization loss Lioe used for training the Pivot ID Encoder (PIE). As illustrated in Fig removing the face localization loss results in inaccurate inject regions of the mixed pivot embeddings Emir during inference. This misalignment further causes partial mismatches between the diffusion
8

---

Published as a conference paper at ICLR 2026
features of the reference face and those of the generated subject, which leads degraded identity fidelity and corruption of regions unrelated to the face as shown Tab.
Our core idea is to employ the pivot embeddings obtained by encoding the reference face image through the Pivot ID Encoder, which serve as a reliable initialization for guidance during inference. Without such embeddings, the optimization based the reference image cannot properly localize the target region or establish feature correspondences. Consequently, the guidance gradients are propagated non-pivot areas, resulting no identity enhancement shown in Fig 6 and Tab. 2
ID-Reference
Prompt
w/o feer
wio PIE
wio BGM
w/o RB-Guidance
Ours
woman
sitting at the table with cup of hands
Figure 6: Qualitative ablation study on OmniPortrait.
Background Gradient Masking As shown in Fig when the guidance gradients are applied glob- ally across the image, the effect of aligning the matched feature points between the reference and generated faces is weakened, making it challenge to achieve high-fidelity preservation of facial de- tails. We attribute this phenomenon gradient leakage, where the diffusion features of the reference face become implicitly coupled with the non-facial regions of the generated image.
0.9 0.8 0.7
0.6
M CLS
0L44
0.3
Figure 7: Qualitative ablation study starting time parameter introduced by RB-Guidance.
Pivotal Optimization An ablation study was first conducted on the starting time parameter u7 introduced by RB-Guidance. shown in Fig 7 during the early denoising stages, the generated image contains only coarse-level concepts, making difffcult establish precise correspondences between the diffusion features of the reference image and those of the generated image, which causes the guidance gradients of RB-Guidance to diverge. Conversely, applying the guidance a later stage reduces its effectiveness. Therefore, u 0.6 was selected our experiments. As illustrated in Fig in the absence of RB Guidance, the generation process is conditioned solely the mixed pivot embeddings encoded by the Pivot ID Encoder, and the resulting portraits exhibit low identity similarity and insufficient preservation of facial details.
Table 2: Quantitative ablation study on OmniPortrait.
Diog
PIE
BGM
RB-Guidance
BLIP
CLIP-T1
CLIP-I1
SIM
IQAT
FID,
77.32
22.11
46.54
35.01
68.10
476.22
/
75.78
23.19
21.83
14.15
63.98
383.17
/
66.80
19.34
37.30
33.42
52.44
483.93
X
81.05
24.88
66.10
63.87
210.45
80.24
24.25
73.08
69.13
86.80
213.48
4.6 EXTENDING TO MULTI-IDENTITY PERSONALIZED PORTRAIT SYNTHESIS
For multi-identity personalized portrait synthesis, instantiate separate Pivot ID Encoder per reference identity to obtain its pivot embedding. Each embedding is concatenated with the corre- sponding person-related text tokens such woman man, and RB Guidance is then applied in the respective face spatial regions. This design naturally extends our pipeline to multi-identity cus- tomized generation, shown in Fig. More corresponding results provided in the Appendix.

---

Published as a conference paper at ICLR 2026
5 CONCLUSION
We presented OnniPortrait, a diffusion-based framework for identity-preserved portrait synthesis that addresses the core limitations of prior single-stream upproaches-namely insufficient fine- grained identity fidelity and degraded text alignment. Central to our design is pivotal optimization: a Pivot ID Encoder, trained with a face-localization loss, establishes a robust identity pivot without fine-tuning the base model, while Reference- Based Guidance (KB-Guidance) performs test- rest-time, coarse-to-fine optimization via local matching of diffusion intermediate features. This dual-stream strategy preserves the rich priors and editability of the pretrained backbone, enabling high-quality, Adentity-faithuul. and prompt-consistem generation. To further facilitate research on personalized portrait synthesis, we will open-source the large-scale human-face dataset constructed in this work. Extensive experiments demonstrate that OmniPortrai achieves state-of-the-at performance on iden- tity similarity and text-image alignment, with clear qualitative gains in fine facial details. The framework is plug-and aang--n--pay,,. readily compatible with community models and naturally extensible to multi-identity scenarios, offering practical value the broader community.
ACKNOWLEDGEMENTS
This work was supported by the National Key R&D Program of China (Grant No. 0222YBB40014004044400) the National Natural Science Foundation of China (Grant No. 62331014), and the SSTIC Grant (Grant Nos. KJZD2023092111510602, KI/ZD2023092311411 CJH12002021881330400888
REFERENCES
Moab Arar, Rinon Gal, Yuval Atzmon, Gal Chechik, Daniel Cohen-Or, Ariel Shamir, and Amit H Bermano, Domain-agnostic tuning-encoder for fast personalization of texl-to-image models. arXiv preprint arXhv:2307.06925. 2023.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309. 190,:999..669, 2023.
Romain Beaumont. 2mg2dataset: Fasily turn large sets of image urls to an image dataset.
https:
com http://cccnooooommmmmmmaaaaaaaaa
2021.
Minwoo Byeon, Beomhee Park, Haecheon Kim, Sungjun Lee, Woonhyuk Baek, and Saehoon
Kim. Coyo-700m: Image-text pair dataset. https: http:///gtttbbboo/o.o kawkkkkkaakkaaaakkaa
coyo-dataset] 2022.
Mingdeng Cao, Xintao Wang, Zhongang Qi, Ying Shan, Xiaohu Qie, and Yinqiang Zheng. Mas- actrl: Tuning-free mutual self-attention control for consistent image synthesis and editing. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 22560-22570, 2023.
Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Pixart-alpha: Fast training of diffusion transformer for photorealistic text-to-image synthesis. arXiv preprint arXiv:2310.00426 2023a.
Zhuowei Chen, Shancheng Fang, Wei Liu, Qian He, Mengqi Huang, Yongdong Zhang, and Zhen- dong Mao. Dreamidentity: Improved editability for efficient face-identity preserved image gen- eration. arXiv preprint Xxxx:2377000000, 2023b.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gor- don, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive language-iman image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818-2823 2023.
Jiankang Deng, Jia Guo. Evangelos Ververas, Irene Kotsia, and Stefanos Zafeiriou. Retinaface: Single-shot multi-level face localisation in the wild. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 5203-5212, 2020.
10

---

Published as a conference paper at ICLR 2026
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans image synthesis. Advances in Neural Information Processing Systems, 34:8780-8799 2021.
Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Forty-first international conference machine learning, 2024.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618 2022.
Rinon Gal, Moab Arar, Yuval Atzmon, Amit H Bermano, Gal Chechik, and Daniel Cohen- Or. Designing encoder for fast personalization of text-to-image models. arXiv preprint arXiv:2302.12228, 2023.
Zheng Ge, Songtao Liu, Feng Wang. Zeming Li, and Jian Sun. Yolox: Exceeding yolo series in 2021. arXiv preprint arXiv:2107 0iti::1007004400, 2021.
Yuchao Gu, Xintao Wang, Jay Zhangjic Wu, Yujun Shi, Yunpeng Chen, Zihan Fan, Wuyou Xiao, Rui Zhao, Shuning Chang, Weijia Wu, et al. Mix-of-show: Decentralized low-rank adaptation for multi-concept customization of diffusion models. arXiv preprint arXiv:2305. 18292, 2023.
Qin Guo and Tianwei Lin. Focus on your instruction: Fine-grained and multi-instruction image editing by attention modulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6986-6998 2024.
Qin Guo, Ailing Zeng, Dongxu Yue, Ceyuan Yang, Yang Cao, Hanzhong Guo, Fei Shen, Wei Liu, Xihui Liu, and Dan Xu. Unime: Taming diffusion transformer for unified keypoint- guided multi- class image generation. arXiv preprint arXiv:2507.02770 02711,, 2025.
serpoint-guieed
Zinan Guo, Yanze Wu, Chen Zhuowei, Peng Zhang, Qian He, et al. Pulid: Pure and lightning id customization via contrastive alignment. Advances in neural information processing systems, 37: 36777-36804, 2024.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. arXiv preprini arXiv:2104.08718 2021.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge to local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in Neurul Information Processing Systems, 68556, 2020.
Jiehui Huang, Xiao Dong, Wenhui Song, Hanhui Li, Jun Zhou, Yuhao Cheng, Shutao Liao, Long Chen, Yiqiang Yan, Shengeai Liao, et Consistentid: Portrait generation with multimodal fine- grained identity preserving. arXiv preprint 2rXiv:2404.16771, 2024. 2024.
Liming Jiang, Qing Yan, Yumin Jia, Zichuan Liu, Hao Kang, and Xin Lu. Infiniteyou: Flexible photo recrafting while preserving your identity. arXiv preprint arXiv:2503.16418. 2025.
Kyungmin Jo, Jooyeol Yun, and Jaegul Choo. Devil in the detail: Towards injecting fine details of image prompt in image generation via conflict free guidance and stratified attention. In Pro- ceedings of the Computer Vision and Pattern Recognition Conference, pp. 23595-23603, 2025.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv:1710.10196, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410 2019.

---

Published as a conference paper at ICLR 2026
Diederik P Kingma and Max Welling. arXiv:1312.61 2013.
Auto-encoding variational bayes.
arXiv preprint
Nupur Kumari, Bingliang Zhang, Richard Zhang, Eli Shechtman, and Jun- Yan Zhu. Molti-concept customization of text-to-image diffusion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 1931-1941, 2023.
Gihyun Kwon and Jong Chul Ye. Tweediemix: Improving multi-concept fusion for diffusion-based image/video generation. arXiv preprint arti:tm....222200555555 2024.
Black Forest Labs. Flux, https://githiie http:/ppp//gtthaa...//oaaac---ee
http://llcck--osstoooeeo forest- Lansstt-abb/E1is
2024.
Dongxu Li, Junnan Li, and Steven CH Hoi. Blip-diffusion: Pre-trained subject representation for controllable text-to-image generation and editing. arXiv preprint arXiv:2305.1472) 2023.
Maomao Li, Yu Li, Tianyu Yang, Yunfei Liu, Dongxu Yue, Zhihui Lin, and Dong Xu. A video is worth 256 bases: Spatial-temporal ipectation-maxiiizaaion inversion for zero-shot video edit- ing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 71277752877777 2024a.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Pho- tomaker: Customizing realistic human photos via stacked id embedding. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 8640-864360 8650, 2024b.
Jian Ma, Junhao Liang, Chen Chen, and Haonan Lu. Subject-diffusion: Open domain personalized text-to-image generation without test-time fine-tuning. arXiv preprint arXiv:2307.11410, 2023a.
Yiyang Ma, Huan Yang, Wenjing Wang. Jianlong Fu, and Jiaying Liu. Unified multi-modal latent diffusion for joint subject and text conditional image generation. arXiv preprint arXiv:2303.09319, 2023b.
Chong Mou, Yanze Wu, Wenxu Wu, Zinan Guo, Pengze Zhang, Yufeng Cheng, Yiming Luo, Fei Ding, Shiwen Zhang, Xinghui Li, et al. Dreamo: A unified framework for image customization. arXiv preprint ar0iv:2504. 16915, 2025.
Alexander Quinn Nichol and Prafulla Dhariwal. Improved denoising diffusson probabilistic models. In International Conference on Machine Learning, pp. 8162-8171 PMLR, 2021.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text- -exieg diffusion models, In International Conference on Machine Learning, pp- 16784-16804, PMLR, 2022.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF international conference computer vision, pp.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann. Tim Dockhor, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01955, 2023.
Guocheng Qian, Kuan-Chich Wang, Or Patashnik, Negin Heravi, Daniil Ostashev, Sergey Tulyakov, Daniel Cohen-Or, and Kfir Aberman. Omni-id: Holistic identity representation designed for generative tasks. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp- 8786-8795, 2025.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on gruphics (TOG), 42(1):1-13, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference Computer Vision and Pattern Recognition, pp. 10684-10695, 2022.
12

---

Published as a conference paper at ICLR 2026
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuming text-to- eext-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 22500- 22510, 2023.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in Neural Informa- fion Processing Systems, 35:36479-36494, 2022.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings the IEEE conference on computer vision and pattern recognition, pp. 815-823, 2015.
2015.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint 0rXiv:2210.08402. 2022.
Jing Shi, Wei Xiong, Zhe Lin, and Hyun Joon Jung. Instantbooth: Personalized text-to-image generation without test-time finetuning. arXiv preprint arXiv:2304. 00441, 2023.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In Interna- tional Conference on Learning Representations, 2021.
Luming Tang, Menglin Jia, Qianqian Wang. Cheng Perng Phoo, and Bharath Hariharan. Emer- gent correspondence from image diffusion. In Thirty-seventhh Conference on Neural Information Processing Systems, 2023. URL
Yoad Tewel, Omri Kaduri, Rinon Gal, Yoni Kasten, Lior Wolf, Gal Chechik, and Yuval Atzmon. Training free consistent text-to-image generation. ACM Transactions on Graphics (TOG), 43(4): 1-18, 2024.
Dani Valevski, Danny Wasserman, Yossi Matias, and Yaniv Leviathan. FaceO: Instantaneously conditioning text-to-image model on face. arXiv preprint arXiv:2306.00 2666028806 2023.
Jinpeng Wang, Tianci Luo, Yaohua Zha, Yan Feng, Ruisheng Luo, Bin Chen, Tao Dai, Long Chen, Yaowei Wang, and Shu-Taoo Xia. Embracing collaboration over competition: Condensing multiple prompts for visual in-context learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, 2025a.
Qixun Wang. Xu Bai, Haofan Wang. Zekui Qin, Anthony Chen, Huaxia Li, Xu Tang, and Yao Hu. Instantid: Zero-shot adentity-preservve generation in seconds. arXiv preprint u4X.v2240.005119 2024.
Zile Wang, Hao Yu, Jiabo Zhan, and Chun Yuan. Alphavae: Unified end-to-end rgba image re- construction and generation with alpha-aware representation learning, 2025b. URL https:
//arsiv.crr//bb//2555
0930
Yuxiang Wei, Yabo Zhang, Zhilong Ji. Jinfeng Bai, Lei Zhang, and Wangmeng Zuo. Elite: Encoding visual concepts into textual embeddings for customized text-to-image generation. arXiv preprint 23845 2023.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Frédo Durand, and Song Han. Fastcom- poser: Tuning-free multi-subject image generation with localized attention. arXiv preprint 2023.
arXiv:2305. 10430 10431,
Tao Xu, Pengchuan Zhang, Qiuyuan Huang, Han Zhang, Zhe Gan, Xiaolei Huang, and Xiaodong He. Attngan: Fine-grained text to image generation with attentional generative adversarial net- works. In Proceedings of the IEEE conference computer vision and pattern recognition, pp- 1316-1324, 2018
Hu Ye, Jun Zhang, Sibo Liu. Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
13

---

Published as a conference paper at ICLR 2026
Hao Yu, Tangyu Jiang, Shuning Jia, Shannan Yan, Shunning Liu, Haolong Qian, Guang- hao Li, Shuting Dong, and Chun Yuan. Comrope: Scalable and robust rotary po-
sition embedding parameterized by trainable commuting angle matrices. 2025a. URL
Hao Yu, Jiabo Zhan, Zile Wang. Jinglin Wang, Huaisong Zhang, Hongyu Li, Xinrui Chen, Yongxian
Wei, and Chun Yuan. Omnialpha: A sequence-to- io-sequence framework for unified multi-task rgha generation. arXiv preprint arXiv: 25/1.20211, 2025b.
Dongxu Yue, Qin Guo, Munan Ning, Jiaxi Cui, Yuesheng Zhu, and Li Yuan. Chatface: Chat-guided real face editing via diffusion latent space manipulation. arXiv preprint arXiv:2305.14744 2023.
Dongxu Yue, Maomao Li, Yunfei Liu, Qin Guo, Ailing Zeng, Tianyu Yang, and Yu Li. Addme: Zero-shot group-photo synthesis by inserting people into scenes. In European Conference on Computer Vision, pp. 222-239. Springer, 2024.
14

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 DETAILS OF OMNIPORTRAIT-IN IM DATASET
In this section, we describe how the raw images for OmniPortrait-IN are collected and then report
dataset statistics.
Data Selection and Standardization We collected 2.8B public raw images from Pexels, COYO-
700M, and LAION-2B. We first filtered images using the criteria 'aesthetic > 5 & pwatermark < 0.5 width 512 & height > 512". In addition, we use YOLOv7-Face to detect faces and retain only single-face images, discarding those with or multiple faces. We found that relying solely on face detection leads to false positives, including portrait-like patterns and animal faces; therefore, additional safeguards based person detection and keep samples only if a person and a face are simultaneously detected and the face bounding box is spatially enclosed by the person bounding box.
required. To mitigate false positives, we incorporate YOLOX-
The raw image files are renamed using a seven-digit number to avoid duplication. Processing was carried out with img2dataset Beaumont (2021). The standardized outputs form our raw dataset, yielding a collection of 1.16M high-quality portrait images.
Statisties of OmniPortrait-IN 1M We analyze the probability distribution of the face-to-image area ratio to ensure diversity in face sizes; higher prevalence of smaller faces promotes learning more stable distribution, as shown the left part of Fig.
g
We also examine the distribution of face
confidence scores, as depicted in the right part of Fig.
Face Area Ratio
Face Confidence
16
Deiatty
Detat
6-
18
8.00
6.02
0:34
0.56
0.88
0.10
0.78
0.50
0.50
0.54
5.50
0.58
0.50
0.93
Trace zoea rato
Faoe conficence
Figure 8: Distributions of face-to-image area ratios and face confidence scores in OmniPortrait- IM.
A.2
ADDITIONAL COMPARISONS
In this section, first present the results of our user study to reflect participants' assessments of our method and the baselines across multiple dimensions. We then provide additional qualitative comparisons against customization methods built on the FLUX backbone.
User Study As discussed in the main paper, a user study was conducted to more comprehensively assess OmniPortrait and baseline methods. The results are presented in Fig. 9 Our proposed Om- niPortrait is preferred by the largest number of participants across four aspects: text alignment, identity fidelity, facial details, and naturalness. In particular, for facial details, over 50% of respon- dents selected our method as the best at preserving fine fine-grained facial features, demonstrating the effectiveness of the proposed dual-stream pivotal optimization.
Comparison with FLUX-based Methods Recent works have explored FLUX-based Labs (2024)
customization by injecting identity information into the DiT backbone, either via additional con- ditioning encoders or token concatenation. We compare against the strongest FLUXX meth- ods to date, including IP-Adapter-FLUX Ye et al. (2023). PuLID-FLUX Guo et al. (2024). In- finite Jiang et al.) (2025), and DreamO Mou et al. (2025) (see Fig. 10 As illustrated, except for DreamO and our OmniPortrait, the first three methods tend to produce overly smooth faces, los- ing substantial high- feghhfrey detail. While DreamO maintains identity fidelity, it relies facial parsing of the reference image; this dependency leads to disharmonious blending when background defocus is applied, shown in the last row of Fig. 10
15

---

Published as a conference paper at ICLR 2026
Text Alignment
Identity Fidelity
37.5%
36.7%
3.1%
DreamBosths 3.12%
Dreamdesth:: 3.27%
4.2%
Sestual mveesos 6.01%
Seciual Inverson:
FastComposer
fastComposer 16 26:23%
456
6.8%
PhotoMaken: 27%
16 2%
PhotoMaker 5.80%
39%%
33.3%
8 7%
Face Detail
Natural
52 0%
CreamBoath: 5.18%
Breamflathh:
Teatual invesiin:
Recual inverson: 09%
PastCompozer 8.50%
RystComposor 68%
6%
MuttaMiken: 5.02%
PhotoMaker 1.09%
Irstanti: 12.50%
Insantid: za 59%
0%
IMAdapter: 18.225
P-Adspter: 16 45%
Ours 51 59%
Durs: de 87%
28 6%
Figure 9: User study results on text alignment, identity fidelity, facial details, and naturalness.
Extended Comparisons with Closed-Source Image Editing Models To enable an apples-to-apples comparison, extend Fig. [0 evaluate closed-source image editing models under the same reference image and prompt including GPT-4o, JiMeng AI, and Nano Banana. The results are presented in Fig. 11 GPT-4o exhibits suboptimal identity fidelity; JiMeng-Al tends copy-paste the reference face during personalization (see the second and third rows of Fig. [T]; and Nano Banana often preserves more background content, reflecting a bias toward text alignment. Overall, these observations highlight the challenge of balancing identity fidelity and prompt consistency in closed-soures systems.
A.3 MORE APPLICATIONS
Multi-Identit, Personalized Portrait Synthesis In multi-identity personalization. OmniPortrait can be adapted with minimal changes. By performing independent pivotal optimization for each identity, simultaneously preserve per-identity consistency while applying RBB-uidancee within the corresponding facial regions, thereby avoiding identity confusion. As shown in Fig. [2] we evaluate multi-identity generation under diverse prompts; results indicate that our approach achieves strong text editability while maintaining clear, non-confounding identity consistency across multiple identities.
A.4 ADDITIONAL ABLATIONS
Hyperparameter Ablations We present the hyperparameter search results in Fig. 14 regarding the face localization loss weight 7) in Equation. 5 a larger 7) leads to higher face similarity but compro- mises text alignment. The parameter p in Equation. 10, which controls the strength of RB-Guidance and u. in Equation. [T which governs the guidance starting time, exhibit similar properties, and we identify sweet spot by balancing text alignment and face similarity. Regarding the timestep to for the reference image, we observe that text alignment is insensitive to this parameter, whereas face similarity peaks when to is approximately 670. A smaller to results in more rigid feature matching. thereby impairing the structural similarity of the face. Conversely, as to increases, the accuracy of feature matching declines rapidly, which is attributed the intrinsic characteristics of DIFT features.
16

---

Published as a conference paper at ICLR 2026
ID-Reference
IP-Adaptc-FLUUX
PalID-FLUX
InfiniteYou
DreamO
OmuiPortrrat
on
man playong guitar "
"A woman wearing red evening dreas"
"A man busy street"
woman sitting chair"
Figure 10: Qualitative comparison of OmniPortrait against FLUX-based customization methods.
ID-Reference
GPT-40
JiMeng Al
Nano Banana
OmniPortrait (Ours)
on
on
"A playing guitar show"
"A woman wearing red evening dress" "
"A man in busy city street"
"A woman sitting on chair"
Figure 11: Qualitative comparison of OmniPortrait with closed-source image editing models.
17

---

Published as a conference paper at ICLR 2026
man and a woman walking logether street, holding coffee
"A man and woman stiting on a bench the purk
man and a woman sitring on a bench the park
ID-Reference
mus and a woman stonding moustaiin peok
mun and a womonn shopping in fisshion store"
"A mun and woman practicing stadio
Figure 12: Multi-identity personalization with OmniPortrait:
consistent identities and high
text-image alignment.
ID-Reference Image
OnmiPortrait(Ours))
ID:Reference image
OmmiPortrait(Ours)
woman wearing red lipstick and police uniform
A woman in the living room, smiling sideways
OmmiPortrait(Uur))
IP-Roforonce image
Infinite You-FLUX PULID-FLUX
mun sitting computer desk, looking very ungry
A man wearing casual attire, sitting on a bcach chair and smiling
Figure 13: Facial attribute editing via text prompts using OmniPortrait, and a qualitative comparison of facial detail fidelity against other methods.
18

---

Published as a conference paper at ICLR 2026
Effect
Effect
24.30
24.09
54.00
24.00
00.00
OLI-TT
00.10
66.00
06.00
24.20
2415
(5g
Effect of
Effect tg
CUPT
25.30
71.30
24.26
24.25
9600
CUPTT
OUP-T
24.24
24.22
23.00
0,075
0.525
5.875
61 [Eq. 11)
Figure 14: Hyperparameter ablation results. CLIP-T measures text alignment, while SIM measures face similarity.
"A man and a man siting a living room sofa, holding controllers and playing games"
"A man und man are walling the street, holding coffee in their right hands"
"A man and mas wearing hat, dressed in hiking gear on mountain, looking joyful"
ID-Reference
"A woman and sitting living room wofa, holding controliers and playing games"
"A woman and walking on the street, holding coffee in their right hands"
"A woman and wearing hat, dressed in hiking gear on mountain, looking joyfuu"
Figure 15: Additional results for multi-identity personalization with OmniPortrait.
19

---

Published as a conference paper at ICLR 2026
Face loss
ID-Referenee
"A woman wearing jucket stands outiloors."
Figure 16: Qualitative examples showing the impact of face localization loss on learned cross- attention maps.
Table 3:
Full list evaluation prompts for subject customization (5 prompts per category).
Category
Prompt
woman/man posing like a tourist front of the Eiffel Tower
A woman/man performing yoga tree pose
Pose
A woman/man sitting cross-legged the floor, smiling
woman/man jumping in the air with joy
woman/man looking back over their shoulder
woman/man wearing a police uniform
woman/man wearing a formal business suit
Clothing
A woman/man wearing a traditional kimono
woman/man in superhero costume, crying disappointedly
woman/man wearing a chef's white uniform
woman/man on the beach
woman/man standing in a snowy forest, looking very angry
Background
A woman/man floating in outer space
A woman/man standing in crowded city street
woman/man inside an ancient library
woman/man holding piece of cake
woman/man playing an acoustic guitar
Action
woman/man riding bicycle, with sweat on face.
A woman/man drinking cup of coffee
woman/man taking selfie with a phone
A woman/man wearing Christmas hat in bright scene
woman/man wearing glasses illuminated by einematic neon lights
Lighting
woman/man standing in the golden hour sunlight
A woman/man in a dark room with dramatic shadows
A woman/man under the moonlight
A painting of a woman/man in the style of Vincent Van Gogh
An oil painting of woman/man
Style
pencil sketch
woman/man
A 3D render of woman/man in Pixar style
A pop art style portrait of woman/man
20


Reviews:
[{"review_id": "g8zrjLUjPI", "reviewer": "Reviewer_ju3a", "summary": "This paper introduces fine-grained personalized portrait synthesis system upon diffusion models. To mitigate the shortcoming of existing models, it propose pivot-based optimization strategies, which ensures the identity of reference image and editability of input prompt.", "questions": "- Without pivotal optimization, is there any degradation when adopting existing manner like cross-attention or attention map swapping for reference path?\n- Before validating the Pivot ID Encoder, how do off-the-shelf identity extractors work compared to the proposed one?\n- It is recommended to discuss the diversity on prompt-based portrait synthesis in terms of facial expression and appearance. The results presented in the manuscript appear to have a tendency to just copy-and-paste the overall facial structure of the reference image, including the open mouth and smile, while PhotoMaker changed those attributes naturally.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- High fidelity with editability: The proposed system demonstrate pretty plausible outcomes while preserving input fidelity with prompt-based editability with reduced trade-off problem between them.\n- Compatibility and extendability: This framework is basically plug-and-play approach with high compatibility. Moreover, some additional results including multi-identity injection showed the significant extendability of the proposed method.", "weaknesses": "- Advantage of the proposed pivotal optimization: For reference-based editing, there are several well-known technics in diffusion models such as cross-attention, attention map swapping, style injection via AdaIN and so on. Within the proposed RB-Guidance, there is no comparison about aforementioned approaches before proposing the pivotal-optimization. It is imperative to provide and discuss some limitations of existing guidance manner on reference path and how the proposed one is useful.\n- Effects of Pivot ID Encoder: The authors propose a new trainable encoder to extract facial identity using CLIP-based model. However, it is common to adopt identity embedding models such as ArcFace, InsightFace to provide identity vectors within denoising process using cross attention module, etc. It is wondered the proposed Pivot ID Encoder offers more useful information in entire process, compared to off-the-shelf identity extractor.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1EYPGFO0E5", "reviewer": "Reviewer_iUtW", "summary": "This paper introduces a personalized portrait generation framework based on pivotal optimization, i.e., combining the facial representation with a specific text token and optimizing it to preserve the identity. Specifically, the proposed method consists of two parts. First, an efficient pivot ID encoder is designed to generate the combined representation, which is learned by a face localization loss calculated between cross-attention maps and facial segmentation masks. Second, during the inference, a pivotal optimization objective is proposed to capture the feature correspondence between the reference image and the generated image. Moreover, a large-scale dataset is collected to train the proposed ID encoder. The experimental results on 50 reference images sampled from the CelebA-HQ and FFHQ datasets show that the proposed method outperforms a few baselines.", "questions": "Q1: Among the baselines, are they trained on the proposed OmniPortrait-1M dataset as well (at least the key components of one baseline)? If not, the comparisons may be unfair. \n\nQ2: Will the proposed dataset be released?\n\nQ3: How does the proposed method perform in the wild (e.g., non-celebrity)? \n\nMiscs.:\n+ \"PulID-FLUX\" in Figure 10 should be \"PuLID-FLUX\".", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "+ The proposed pivot ID encoder is simple (feature concatenation followed by a linear layer) and efficient. And from the quantitative results, it improves the base model significantly.\n\n+ The designed facial localization loss and the reference-based guidance loss are reasonable and can be combined with other models seamlessly. \n\n+ Compared with the baselines, the proposed method balances text following and identity preservation better.\n\n+ This paper is written well and conveys its idea clearly.", "weaknesses": "1. There are many hyperparameters (Eq. (3), (5), (6), (10), and (11)) in the proposed method, of which the settings seem to overfit the dataset (e.g., $t_0 = 671$) and require in-depth analysis. \n\n2. According to the ablation study (Table 2), the proposed face localization loss contributes most to the performance gains of the proposed method. However, it also lacks a comprehensive analysis, e.g., the qualitative examples of learned cross-attention maps, comparisons with other localization losses, and the selection of denoising layers for learning.\n\n3. The base models of the proposed method and the baselines are out-of-date (e.g., SDXL). As a few qualitative results of some FLUX-based methods are reported in the Appendix, why not implement the proposed method based on FLUX for comparisons directly in the main paper?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1L0KUukAaW", "reviewer": "Reviewer_V3R8", "summary": "This work proposes method called OmniPortrait to generate photo-realistic image while preserve identity consistency and achieve high alignment with text prompt by introducing pivotal optimization and reference-based guidance (RB-Guidance). It also supports multi-identity customization.", "questions": "1. Author mentioned they utilized SD and SDXL in Sec. 4.2. However, which base model are used in each experiment?\n2. For measuring ID-preservation, why authors didn’t measure cosine distance between reference and generated faces images? It will be more convinced if measuring recognition accuracy among different face recognition models.\nOther questions see in weakness.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Only Pivot ID Encoder and linear protection layer are needed for training which makes OmniPortrait more efficient than finetuning UNet model.\n2. The proposed RB-Guidance enable OmniPortrait to have better ID-preserving generation.\n3. Authors have conducted various experiments to verify the effectiveness of the method.", "weaknesses": "1. The size of evaluation dataset is too small, only 50 reference images are used. Moreover, the content of 30 text prompts used are not clear. Based some samples in Figure 4, it seems the prompts mostly descript general object rather than descript facial attribute itself.\n\n2. Authors claim that OmniPortrait have better \nfine-grained identity details. However, it seems no relevant experiments support this. For example, authors should measure how facial attributes changes in generated face images according to text prompts that descript the wanted facial attributes. \n\n3. Author use RB-Guidance to optimize the energy function. However, how energy function associated with ID-preservation generation is under explained.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "CgqIDuve99", "reviewer": "Reviewer_Lh8m", "summary": "This paper explores the field of personalized image generation. It begins by introducing the Pivot ID Encoder to achieve coarse-grained identity consistency, thus avoiding the degradation of prompt alignment caused by directly fine-tuning the base model. The authors then propose RB-Guidance to enhance fidelity during inference. This approach leverages the correlation between the denoised latent features of the generated image and the reference image, improving the fidelity of facial features. Additionally, the paper presents the construction and curation of the large-scale dataset OmniPortrait-1M, which will contribute to further advancements in the community.", "questions": "- This method does not propose a solution to the multi-person identity confusion problem but instead relies on the semantics of the prompt for distinction, e.g., “a man and a woman.” How would the method perform if the prompt were “two men” or “a man and a man”? Would it still function properly?\n- On line 377, the authors state that previous methods often produce results resembling a “copy-and-paste” of the reference image, which is expected, since during training, the reference face and the ground truth face are always aligned. However, this method does not appear to make any substantial improvements to this issue during training. Why does this method avoid this problem in the qualitative experiments?\n- The authors claim to have constructed and released OmniPortrait-1M, a large-scale, high-quality multimodal face dataset. However, I don't find access provided in the manuscript or appendix. Will the dataset be released upon acceptance?", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The proposed method can be extended to multi-ID personalization without requiring multi-ID training data.\n- The paper is well-written and easy to follow.\n- The ablation study is thorough and effectively demonstrates the validity of each module.\n- The introduction of the OmniPortrait-1M dataset is a valuable contribution.", "weaknesses": "- The paper claims that current state-of-the-art methods suffer from two main issues: low facial fidelity and failure to maintain prompt alignment. However, the methods demonstrated (InstantID, PhotoMaker, FastComposer) are outdated. In fact, more advanced approaches such as ConsistentID, Pulid, and InfiniteYou have already addressed these issues, making the motivation less academically compelling.\n- 1）The operation of associating visual embeddings with text embeddings in the Pivot ID Encoder has already been widely adopted, for example, in PhotoMaker. Similarly, the idea of face localization loss is also present in FastComposer.   2）The insight behind RB-Guidance—utilizing diffusion feature matching to improve facial fidelity—has been extensively explored by other methods, such as Visual Persona and Consistory.\n- The base model used in the paper, based on the U-Net architecture, is outdated. Currently, competitive base models are using the DiT architecture, and the effectiveness of the proposed method on DiT remains untested.\n- The methods compared in the paper are outdated, while results comparing to more advanced methods are relegated to the appendix. Furthermore, only qualitative results are provided, without any quantitative experiments.\n-  When using InstantID, the reference image should not be cropped. Instead, the original reference image should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)