InstantPortrait: One-Step Portrait Editing via Diffusion Multi-Objective Distillation
Authors: Zhixin Lai
Keqiang Sun
Fu-Yun Wang
Dhritiman Sagar
Erli Ding
Venue: ICLR 2025 Poster (2025)
Keywords: Portrait Editing
Diffusion Multi-Objective Distillation
On Step Inference
Identity Preservation
Text Fidelity
Primary area: generative models
TL;DR: We train IPNet via Diffusion Multi-Objective Distillation and achieve one-step instruction-based portrait image editing, with high-precision identity preservation, precise instruction-based image editing, and rapid model inference concurrently.

Abstract: Real-time instruction-based portrait image editing is crucial in various applications, including filters, augmented reality, and video communications, etc. However, real-time portrait editing presents three significant challenges: identity preservation, fidelity to editing instructions, and fast model inference. Given that these aspects often present a trade-off, concurrently addressing them poses an even greater challenge. While diffusion-based image editing methods have shown promising capabilities in personalized image editing in recent years, they lack a dedicated focus on portrait editing and thus suffer from the aforementioned problems as well. To address the gap, this paper introduces an Instant-Portrait Network (IPNet), the first one-step diffusion-based model for portrait editing. We train the network in two stages. We first employ an annealing identity loss to train an Identity Enhancement Network (IDE-Net), to ensure robust identity preservation. We then train the IPNet using a novel diffusion Multi-Objective Distillation approach that integrates adversarial loss, identity distillation loss, and a novel Facial-Style Enhancing loss. The Diffusion Multi-Objective Distillation approach efficiently reduces inference steps, ensures identity consistency, and enhances the precision of instruction-based editing. Extensive comparison with prior models demonstrates IPNet as a superior model in terms of identity preservation, text fidelity, and inference speed.

Published as a conference paper at ICLR 2025
INSTANTPORTRATT: ONE-STEP PORTRAIT EDITING VIA DIFFUSION MULTI-OBJECTIVE DISTILLATION
Zhixin Lai'>
Keqiang Sun
FuYun Wang
Dhritiman Sagar'
Erli Ding'
Snap Inc
The Chincse University of Hong Kong
(laizhixin16, f erlidingg@gmail.oom com,
(kquun, fyaang) lnk... cuhk. edu. edu.hk, .k, athritiman.sagar Cdnntimaa.sggar/aammppaa.ccmmm @snapchat.com
Input
Output images conditioned by the input images and editing instructions
Figure 1: The output of Instant-Portrait model (IPNet). IPNet portrait image editing model trained through Diffusion Multi-Objective Distillation. IPNet excels in identity preservation and portrait editing, while also achieves one-step model inference. (Prompts in Appendix T
ABSTRACT
Real-time instruction-based portrait image editing is crucial in various applications, including filters, augmented reality, and video communications, etc. However, real-time portrait editing presents three significant challenges: identity preservation, fidelity to editing instructions, and fast model inference. Given that these aspects often present tradc-off, concurrently addressing them poses an even greater challenge. While inffusiio-basee image editing methods have shown promising capabilities in personalized image editing in recent years, they lack dedicated focus on portrait editing and thus suffer from the aforementioned problems as well. To address the gap, this paper introduces an Instant-Portrait Network (IPNet), the first one-step diffusion-based model for portrait editing. We train the network in two stages. We first employ annealing identity loss to train an Tdentity Enhancement Network (IDE-Net), to ensure robust identity preservation. We then train the IPNet using novel diffusion Multi-Objecke Distillation approach that integrates adversarial loss, identity distillation loss, and a novel Facial-Style Enhancing loss. The Diffusion Multi-Objectio Distillation approach efficiently reduces inference steps, ensures identity consistency, and enhances the precision of instruction-based editing. Extensive comparison with prior models demonstrates IPNet as a superior model in terms identity preservation, text fidelity, and inference speed.
"Equal contribution. Work done while Snap Inc.

---

Published as a conference paper at ICLR 2025
GAME
plossrosliic piun mostaoke reaning sket, passe ant foofiet roliet, Mer buskground
Input
Magic Brush
Inerrctt
IP-Contrel
InsamilD
Ours
Magic
Instract
IP-Control
InstenlDD
Ours
PizPPix
XL
Brush
PicPPs
-XL
Figure 2: Bad cases of SOTA methods. Image editing models like MagicBrush Zhang ct al. [2024) and InstructPix2Pix Brooks et al. (2023) frequently cause significant identity distortion and artifacts in complex edits. In identity preservation modes like IP-Adapter Ye ett al. (2023) with ControlNet Zhang et al. (2023) SD-XL (IP-Control-XL) and InstantID Wang al. 2024ap, identity is better maintained but style effects and text fidelity are bad. Simultancously excelling in both identity preservation and text fidelity remains a challenge.
INTRODUCTION
Instruction- based portrait image editing is defined maintaining the face identity while allowing modifications to all other aspects such as background, clothing, and facial features. Instruction-basee portrait image editing is critical for applications such digital lens filters, augmented reality, and vidco communications.
For cvaluation of the portrait image editing models, people mainly care about three aspects: the identily consistency Tewari et al. (2000); Sun et al. (2023), the editing fidelity Gu el al. (2019): Xia et. al. (202T), and the speed Portenier al. (2018); Kim et al. (2021): Bai ltt al. (2024h These aspects are typically trade-off Fitzsimmons et al. (2018); Alaluf et al.
(2022
et al. (2023a), which poses significant challenge to improve all of these aspects simul tancously.
Recent image oditing models such as InstructPix2Pix Brooks al. (2023) and MagicBrush Zhang et al. (2024) focus on text fidelity, enabling detailed edits based on user instructions
but often struggle with maintaining consistent identity, resulting in significant discrepancies and distortions, particularly in complex portrait edits (see Figure 2b. A primary reason for these issues is the lack datasets that are both robust. identity preservation and rich in complex portrait edits, along with insufficient supervision of identity during training. On the other hand, personalized image synthesis models like 1P-Adapter Ye et al. (2023) and InstantID Wang et al. (2024a) enhance identity preservation compared to image editing models but at the expense of style, particularly in facial effects, as shown in Figure 2 This trade-off results from using identity embeddings as an input condition, which limits the ability to edit facial style elfects. Additionally, using identity embeddings as the sole input condition without. identity-specific supervision has been proved to be an inadequale method for preserving identity, with noticeable discrepancies compared to the original identity as illustrated in Figure 2 Furthermore, the models require 30 50 sampling steps in inference.
To address these challenges, we first construct a dataset specifically designed for instruction- based portrait image editing. Using the datasct, we train our Identity Enhanccment Network (IDE-Net) and the Instant-Portrait Network (IPNet) through a two-stage approach, optimiz- ing for identity preservation, precise image editing, and inference speed:
(1) In the first stage, train IDE-Net with novel Annealing Identity Loss, which uses the input face embedding as supervision and employs an annealing strategy to dynamically balance the identity loss and stable diffusion loss. This approach enables our IDE-Net to generate high-quality images that maintain precise identity, even better than the training datasct.
(2) In the second stage, we distill IDE-Net into IPNet with our novel Diffusion Multi-Objective Distillation technique. During the distillation, incorporate Adversarial Loss to enhance the image quality, integrating Identity Distillation Loss to preserve the identity, and also utilizing Face-Style Eahancing Triplet Loss to further enhance the facial style. IPNet.
2

---

Published as a conference paper at ICLR 2025
achieves faster inference, higher image quality, and better facial style, than the teacher model IDE-Net.
In our evaluation, IPNet is compared with top sfate-of-theare models in portrait image editing across metrics inchuding identity preservation, text fidelity, and image quality, as well model size, and inference steps. IP outperforms these models by substantial margin, demonstrating the effectivencss of our Diffusion Multi-Objoctive Distillation approach.
The core contributions three folded:
(1) To our knowledge, we are the first to achieve one-step instruction based portrait image editing, with high-precision identity preservation, precise instruction-based image editing, and rapid model inference concurrently.
(2) We introduce the Constant Identity Loss to significantly improve identity preservation. Building on this, we propose the Annealing Identily Loss, an enhancement of the Constant Identily Loss that dynamically adjusts to better balance identity preservation and text alignment during IDENet training.
(3) We introduce the Diffusion Multi- Objective Distillation process to distill IDENet into 1PNet with Adversarial Loss, Identity Distillation Loss, and Face-Style Enhancing Triplet Loss, achieving onc-step instruction-based portrait image editing and balancing multiple concerned objectives.
2 PRELIMINARY
Stable Diffusion Kingma Welling (2013) utilizes a pre-trained variational autoencoder with encoder & and decoder D. For image the encoder produces an initial latent 2 = {x), and during the diffusion process, noise is incrementally added, creating a noisy latent 2ạ at each timestep t T. The transformation for each step is mathematically described as follows:
= Vazz + VI- āce, e~ 1)
(1)
where ā₂ is fixed noise scaling factor, and is noise sampled from standard normal distribution.
Building the diffusion model, Instruct Brooks et al. (0233 adapts controlled diffusion framework for image editing. employs a U-Net Eo to predict the noise in the
latent 2t, guided by image conditions ej and text instructions er. InstructPix2Pix modifies the U-Net's first convolutional layer to include additional channels that concatenate ey with 2,. The diffusion model loss (L)am of InstructPix2Pix is as follows:
Ldm E( ||e col2,, ((()),
(2)
For the instruction-driven portrait image editing, DustrrctPix2225 requires precise identity alignment between input and target images for model training, otherwise it would result in identity distortion, as shown in Figure] 2
METHOD
Our objective is to swiftly edit portrait images that preserve identity and comply with specific instructions. We first construct a tailored dataset for the instruction-based portrait editing, described in Appendix B and then develop two-stage training approach for our diffusion models, Identity Enhancement Network (IDE-Net) and Instant-Portrait Network (IPNet). Initially, outlined in Figure] 3 the IDE-Net is trained using a novel Annealing Identity Loss Laid (Section 3.1.1 to ensure robust identity preservation, achieving performance beyond the dataset's 's baseline. Subsequently, cloning IDE-Net and augmenting it with our Diffusion Multi-Objectin Distillation technique. This technique integrates Adversarial Loss Lady (Section 3.2.1 for enhancing image quality,
outlined in Figure 4 initialize the IPNet by
3

---

Published as a conference paper at ICLR 2025
Lidd
weight
Input c
Cosine
Input ct
Output &g
Linear
* FaceNet
0
999
Target
Zt
IDE-Nct 0
Output R
Annealing Weight
Lam
IDE-Net Training
Annealing Identity Loss Lidd
Figure& 3: We train our Identity Enhancement Network (IDE-Net) with novel Annealing Identity Loss excel in identity preservation.
Identity Distillation Loss Caistill (Section 3.2.2 for maintaining identity, and Faco-Style Enhancing Triplet Loss Striplet (Section 3.2.3 for facial style improvement. As result, IPNet achieves faster inference, superior image quality, and enhanced facial style than IDE-Net.
3.1 IDE-NET TRAINING
For image editing conditioned by instruction text, InstructPix2Pix relies precise identity alignment between input and target images in its training data, as discussed in Section 2 To address this constraint, we propose IDE-Net, built InstructPix2Pix, incorporating Anncaling Identity Loss Caid that preserves identity more effectively, oven with suboptimal datasets, shown in Figure
3.1.1 ANNEALING IDENTITY LOSS
To ensure identity preservation, utilize an Constant Identity Loss Leid- This involves cropping facial regions from both the input image er and the output image šo, which are then converted to grayscale F, to emphasize structural features over color and texture. Face embeddings are extracted using FaceNet Ferop |Scrroff et al. (2015), and the Leia is computed based on the L2 distance between these embeddings, as detailed in Equation (3.
Ceid = Eeg,2
- (F
(3)
Prior works, such as Ju et al. 2023 Li et al. (024), mention the importance of preserving face identity and pose structure in early denoising stages and enhancing style later. Therefore, to balance diffusion loss Lam and identity loss Lid, we introduce the Annealing Identity Loss Laid- This strategy involves gradually reducing the weight of the identity loss across the denoising steps, enabling smooth transition from identity preservation
alignment.
Laid = Wa(t, Tmax) * Leid
(4)
where Tmaz represents the maximun timesteps T and usually sets 999 for slable diffusion models. denotes the timesteps, decreasing from T Tmax to 0. The function W, (t, T maz),, ) an anncaling algorithm, decreases as reduces, lcading to a progressive reduction in the Annealing Identity Loss. Among various options, we opt for linear decay T.me. with
comparative analysis in Appendix
The total loss for IDE-Net is defined in Equation ( Initially, LIDE-Net effectively preserves the identity of the input image using Caid- As the denoising process progresses, LIDE-Net gradually shifts focus towards emphasizing style using Lam, ensuring uniform enhancements throughout the denoising steps.
LIDE-Net Lam + Auid Laid
where Xaid the balancing weight for the Annealing Identity Loss.
4

---

Published as a conference paper at ICLR 2025
Jino
Fdmno
Style +
Frripat
Input oy
Output Ri
Target x
d
Inpur 9
d2
2
E-mmat (amuadd22200 (2--d--m d.-d.
T Targer
IPNer
Output Pp
IDE-Net 9
Outpen Sy
Imput o
Ourput 8
Target
Casent
Push Away
Rads
IPNet Training with Diffusion Multi-Objeerive Distillation
Face-Style Enhunced Triplet Loss Pppplet
Figure 4: Our Instant-Portrait Network is distilled by IDE-Net with novel Diffusion Multi- Objective Distillation technique. During distillation, we freeze the IDE-Net and incorporate Adversarial Loss Ladv to improve image quality, an Identity Distillation Loss Cdistill to maintain identity, and a Faco-Style Enhancing Triplct Loss Liriplet to further refine facial style.
Training with Caid results in a consistency identity, but the style described by the text prompt is limited in the outputs of IDE-Net, which is then tackled by the introduced IPNet with Diffusion Multi-Objective Distillation.
3.2 IPNET TRAINING WITH DIFFUSION MULTI-OBJECTIO DISTILLATION
As depicted Figure TPNet tg, cloning from IDF-Net cg. is trained with the Diffusion Multi Multi-Obeective Distillation method, inspired by the Score Distillation Sampling (SDS) Poole eet al. 22022). The distillation process integrates an Adversarial Loss Ladv for enhancing image quality (Section 3.2.1. an Identity Distillation Loss Laistill to improve identity prescrvation (Soction 3.2.2 and a Faco-Style Enhancing Triplct Loss Liriplet to boost facial style editing capabilities (Section 3.2.3)
3.2.1 ADVERSARIAL LOsS
The Adversarial Loss Ladv accelerates generation by enabling the model to learn complex transformations between marginal distributions Xiao et. al. (2021), which can effectively reduce the number of denoising steps and can even achieve one-shot generation. Therefore, we incorporate a discriminator D, as suggested by Sauer et al. (2023a), which uses Cadv to distinguish between the output image of IDE-Net Es (labeled as fake) and the target image (labeled as real).
In contrast the text conditional discriminators referenced in Sauer et al. (2023a) Kang et al. (2023), utilize an unconditional discriminator. This choice is supported by our findings that random cropping significantly enhances generative performance. However, such cropping strip away crucial image semantics and adversely affect text conditions. To resolve this issue, we optimize the text aligmment through knowledge distillation from the teacher model described in Section 3.2.2 rather than in the Ladv- Therefore, the Ladv is formulated as:
Ladv = Eg ma -D(x)) + Eeg (0a + D(++))
3.2.2 IDENTITY DISTILLATION Loss
Score Distillation Sampling (SDS) Poole et al. (022)) is critical for Stable Diffusion model distillation Sauer al. (202bb) as defined Equation (10) and Equation
Luds = Ez || Libosstop_grade 2001
5

---

Published as a conference paper at ICLR 2025
2t Va vI -
(8)
where &g is the output of IPNet with &g being its corresponding predicted latent code. Zt. the input of IDF-Net, is calculated by applying the stochastic noise v - to Var (s(stop_ (stop__rrd(())))) is the output of IDE-Net with a stop-gradient operation. The loss is defined as the L2 distance between the image output of IPNet and IDE-Net.
However, SDS often produces over-smoothed and low-detail images due to its reliance on stochastic noise sampling Wang et al. (2024b). To address this limitation, we integrate DDIM inversion Song
(2020) the time steps lower than for fine-grained distillation. And for the time steps higher than which captures structure information, like pose and identity, we maintain the cearse-grained distillation with stochastic sampling, formulated as:
Varis + vI arEa t) Vonis + vI Oxftt
SST 1>T
(9)
Here, the time-step threshold т is a checking the spalial aligument accuracy and image quality. When the time step 1 т, we apply stochastic noise vI- aye Oyt; var&s. When t S т, DDIM inversion directly predicts 2t by IDE-Net eg, omitting stochastic noise. We call the new loss with the refined Zy as Ldistill-
manually determined to 200 by
Ldistill = EE [|10ololop_ [[(o((lop_gruu(t))) - 20l13
(10)
3.2.3 FACE-STYLE ENHANCING TRIPLET Loss
Preserving identity while updating facial style poses significant challenge due to their interdependence. To address this, we introduce the Face-Style Enhancing Triplet Loss Ltriplet which balances identity preservation and facial style variation by comparing the relations of input, output, and target images, as shown in Figure 4 The loss is outlined below.
Cuiple = Mar(d2 - d1 - m,
(11)
where d1 Faiuo [Fdiio(FFrr)(()) - Fdino PFiiooFerrppoe Pcro((fx))) d2 Faimo Ferop(x)) Feino [ (Ferop (f+))
(12)
Here, Cr is the input, is the output from IPNet, and x the target image. Fooo denotes the facial crop, and Fano refers to the mapping of image to DINOv2 embedding Oquab) eet al. 2023).. dj represents the distance between the output and input embedding, and d2 represents the L2 distance between the output and target embedding. The margin m, tailored to datasets, serves as a dynamic mechanism to balance identity preservation with facial style. Typically, set based the observed distances in validation set to ensure appropriate thresholds.
Overall, the total loss for IPNet defined as:
LIPNet = Ladv + Adistill Ldistill + Atriplet * Striplet
(13)
where Adistill and Atripiet balances the loss functions, varving across different time-step sampling stages during distillation, as detailed in Section 4.1
3.2.4 STYLE BOOST VIA ITERATIVE INFERENCE
To further improve the facial style, we apply the iterative inference by using the output of the first step in IPNet inference instead of the original image the input for the second step. By feeding the output back into the model, subsequent steps further concentrate on stylistic details, amplifying the intended visual characteristics while preserving the overall structure.

---

Published as a conference paper at ICLR 2025
Table 1: Quantitative comparison against Srate-of-the-an models. The best results are highlighted in bold, and the second-best results are underlined.
Image
Model
Inference Step.
Pack-Siiilrryyt
Foxt-Fideliyy
Image- Lunge-Quality
Mathod
Resolution
Sizel
FaceNet
lusight Face
CLIP-Vit-g
CLIP-Vit-H
HPS
Q-Align-Q
Q-Aligu-A
MagicBrush
768x512
S59M
0.161
0.401
0.219
0.262
0.209
1.192
3.148
Isstroctt5s2PPps
1026x576
659M
30
0.263
0.196
0.247
0.287
0.216
3,653
2.755
IP-Control-11 1.5
IP- Consrro--XL
1024x576
3739M
0.662
0.548
0.228
0.273
0.228
4,061
2,890
Instant
1024x:576
4150M
50
0.751
0.680
0.257
0.296
01.663
4.0/2
3.087
IPNet (Ours)
1021x376
EXPERINMENTS
We outline the experiment setup in Section 4.1
compare IPNet with state-of-the-art models
(SOTA) in Section 4.2 and conduct ablation studies in Section 4.3 to evaluate components'
effectiveness.
4.1 EXPERIMENT SETUP
Dataset For training, we utilize our dataset outlined in Appendix B For evaluation, we select 100 images from FFHQ Karras el al. (2019) and 40 prompts from our dataset, generating 4000 validation data pairs.
Baseline Our baseline model, InstructPix2Pix, utilizes Stable Diffusion V1.5 and is fine-tuned on our dataset.
Evaluation metrics We evaluate models along four dimensions: Speed, Face-Similarity, Text-Fidelity, and Image-Quality. Speed metries include model size, and inference steps. Face- Sine--imilarity measures similarity between the output and the input image using FaceNet Schroff et al. (2015) and Face Guoo Deng (2019). Text-Fidelity involves CLIP-T scores (CLIP- and CLIP-Vit-H) [Hessel et al. between the output image and the condition text. Lastly, Image-Quality is assessed through Human Preference Score (HPS) Wu et al. (202bb), Q-Align-Quality (Q-Align-Q)) and Q- A-Aligu-Aesthetic (Q-Align-A) Wu et al. (2023a). tocusing on image quality and aesthetics.
2021), 1,, which evaluate semantic similarity
Wu
Implementation IDE-Net is trained with time steps sampled in the range [0, 999], using Annealing Identity Loss Caid weight of 0.7 to balance identity preservation and image quality, Building on IDE-Net, IPNet distillation is divided into three stages: High Time Step (I400, s00J) focuses on the structure and pose alignment with Identity Distillation Loss Cdistill set to 1 to stabilize early training; Middle Time Step ([200, 400)) reduces Identity Distillation Loss Liistill weight to 0.3, emphasizing Adversarial Loss Ladv to improve image quality and mitigate artifacts and further keeping structure and pose alignment; and Low Time Step (1500, 200J) refines details and style with DDIM inversion- Eaistill and Face-Stae Style Enhancing Triplct Loss Luriplet, utilizing a large batch size of 2048 to enhance style consistency and stabilize training. More training details are summarized in Appendix
Identity Distillation Loss
4.2 BASELINE COMPARISON
To demonstrate the effectiveness of our method, we compare our IPNet with statc-of-the art (SOTA) models on portrait image editing, including MagicBrush Zhang et al. (2024), InstantID Wang et al. 2022a), IP-Adapter Ye el al. (2023) with ControlNet Zhang et al.
(2023) SDI.5 (IP-Control- IP-Attrter with ControlNet SDXL (IP-Control-XL), and IustructPx22P2 SDXL Brooks et al. (2023). The integration of IP-Atteter with ControlNet aims to preserve the pose the input image.
Qualitative comparison As shown in Figure we assessed our IPNet models using diverse images and instruction prompts. The results confirm that IPNet surpasses SOTA modols in Faco-Similarity, Text-Fidelity, and muage-uuaiity.
Quantitative comparison As reported in Table D. IPNet excels across Face-Similarity, Texi-Fidelity, and Image-Quality metrics with fewer inference steps compared to other
7

---

Published as a conference paper at ICLR 2025
The Wolf God, with tribal face paint. woll toeth and für and leather and stands forest by wofves
Cyberpunk churacter with unruly hair, wearing gritt riot gear. in a cyberpank megucity sumsel
A 96r computer geek yearbook, photo feuttrrig glasses. and shirt with pocket protector. blue-colored drape buckground
A 90r antme fan photo, with bright colored hair. eyes makeup. wearing 90's imspreed anime character castume
nccnpunk Crisp with magente highlights, purple shadows
Weuring 1920h Rapper look, beauled dress, carl huir with feuthered hendbund featuring metallic details
Victorian Queen elaborate royal clothing depicted in a Rococo painting filled with romantic and playful scenes of royal life
Input
MagieBrush
Instruct Pix2Pix
IP-Control -1.5
IP-Control
InstantID
IPNet step)
IPNet
-XL
(2 stop)
Figure 5: Qualitative comparison with SOTA methods

---

Published as a conference paper at ICLR 2025
Table 2: Quantitative Results of Progressive Improvement over model training and distillation.
"4" indicates the inclusion of the specified loss function in the training process.
Inference Step1
Face-Similarity!
Text- Foxt-Fideityy
Unage-Quality"
Method
FaceNet
InsightFace
CLIP-Vit-g
CLIP-Vit-H
HPS
Q-Align-Q
Q-Aligu-A
Baseline
30
0.331
0.184
0.278
0.319
0.270
4.852
3.635
+ Coid
20
0.958
0.848
0.221
0.264
0.232
4.780
3.216
Caid (IDE-Net)
20
0.791
0.251
0.291
0.203
1.814
3.463
1 Lutv
1
0.048
0.029
0.197
0.236
0.189
2,436
2.167
0.247
3.514
1 Laav't Laistlli
0.873
0.789
0.270
0.306
0.269
4.929
3.628
+ Extor Lateen Leciptee (IPNet)
(a) V' Identity
(b) Eae Artifatt] Stylet
Eate Edd InferStep. Idenitty
(c) Ea -Emai Stylet Quality
(a.Eatt -Eaom -Zongter (I Step) Stylef Quality!
(g) aas Aenit
" (2 Step) Style1
enstrmment tallnons
Input
IDE-Nei
Disiil
IPNe:
Figure Qualitative results of progressive improvement over model training and distillation
state-of-the-art (SOTA) models. IPNet scores notably high in FaceNet at 0.867, surpassing IP-Control-1.5ss 0.794, and leads in Insight with 0.782, well ahead of InstantlD's 0.689. Additionally, IPNet achieves CLIP- Vit-g score of 0.271, outperforming InstantID's 0.257, and records a Q-Align-Q score of 4.960, exceeding Instant 4.546.
4.3 ABLATION STUDY
Table 2 and Figure B demonstrate the progressive improvement over model training and distillation. Figure 7 are qualitative comparisons of extra ablation experiments. For fair comparison across ablation studies, keep the same training steps and time-step sampling strategy outlined in Section 4.1 while changing the loss function.
4.3.1 ANNEALING IDENTITY Loss
We evaluate IDE-Net using Constant Identity Loss Leid and Annealing Identity Loss Laia- As shown in Table [ both versions significantly improve Face-Similarity scores compared to the baseline. Laid maintains better balance between Identity-Similariit, Text-Fidelity, and Image-Quality, than Leid- Furthermore, Figure 0 illustrates that while both Leid and Laid preserve identity, Laid achieve better Text-Fidelity with stronger facial style, consistent with the quantitative results Table [ This observation confirms the effectiveness of annealing algorithms in balancing identity preservation and facial style. Further discussion the selection of anncaling algorithms is provided in Appendix
4.3.2 IDENTITY DISTILLATION LOSS
We validate the elfectiveness of Tdentity Distillation Loss Caistill for identity preservation through two key abservations. First, Table demonstrates that + Ldisttll significantly enhances FaceNet and InsightFace scores over Ladv, with increases from 0.048 to 0.873 and 0.029 to 0.789, respectively. Second, Figure shows that (e) Ladv Caistill substantially improves identity prescrvation over (c) Ladv alone.
We further evaluate DDIM Inversion Laistall comparing to the conventional SDS Loss Leds- First, Table 2
demonstrates that Laisuill surpasses Lads in both Text-Fidelity and Image-Quaal Quality. Second, in Figure 0 (e) Laistil output is sharper and more aligned with instruction prompt. than (d) Luts: with minimal difference identity preservation.

---

Published as a conference paper at ICLR 2025
moisy Sampling:
IDE-Net Net outpun
Sampling:
Eripior
Dil
Good IDE-Net
Input
DIIM Inversio:: mony lntert
DOIM IDE-Ne
with Pepptet
latant space
P Ablation:
cippit
Distillation with
Distillation on Different Space
vs Stuchastic Sampling
Ablation
Figure 7: Qualitative comparison of ablation experiments
In Figure 7 we delve deeper into why Ldistill with DDIM Inversion generales superior images over Ceds. Specifically, Figure 7 (a) illustrates that the DDIM Inversion's deterministic sampling generates clearer and more accurate noisy latent codes than Cads 's stochastic sampling. This clarity leads to improved output quality of IDE-Net, as depicted Figure (b). Consequently, this enhanced guidance from IDE-Net facilitates the distillation of more effective student model, IPNct.
4.3.3 FACE-STYLE ENHANCING TRIPLET Loss
To evaluate the effect of Face-Style Enhancing Triplet Loss Liripiet facial style, we compare models trained with and without Ltriplet- According to Table Liriplet improves Text-Fidelity and Image- Image-Quality and only slightly diminishes Simelaiiml.rity. Figure (c) shows that Ctriplet effectively accentuates instructed facial features, such as glasses, while maintaining identity. The balance between style enhancement and identity preservation is fine-tuned by the margin setting which varies by dataset. A margin of 0.15 is optimal in our dataset, enhancing style with minimal effect on identity similarity.
4.3.4 STYLE BOOST VIA ITERATIVE INFERENCE
We assess the effectiveness of iterative inference by comparing images (f) and (g) in Figure
4.3.5 MULTI-OBJECT DISTILLATION
We validate the effectiveness of multi-object distillation through two key observations. First, Table 2 demonstrates that IPNet not only enhances Text-Fidelity and Image- Quality over IDE-Net but also maintains high Pace-Similarity Second, Figure 6 illustrates that images (f) and (g) generated by IPNet have superior facial style over (b) generated by IDE-Net. multi-object. distiilation using diffferntt IDE-Nets. As depicted in Figure 7 (d) and (e), a well-performing IDE-Net teacher model retains high face similarity, significantly boosting the identity preservation capabilities of the student model IPNet during model distillation. This underscores the importance of strong identity preservation in the teacher model for effective distillation. Furthermore, in Figure 7 (f), observe that distillation in pixel space achieves slightly better facial style over latent space with the same IDE-Nct.
Additionally, perform experiments
CONCLUSIONS
This study establishes exceptional identity preservation, precise image editing, and rapid model inference. We introduce the Annealing Identily Loss significantly enhance identity preservation within IDENet and implement the Diffusion Multi- Aulti-OOjecctiv Distillation process to effectively distill IDENet into IPNet. This approach utilizes Adversarial Loss, Identity Distillation Loss, and Face-Style Enhancing Triplet Loss to address multiple critical objectives simultancously.
benchmark in instruction-basel portrait editing, achieving
10

---

Published as a conference paper at ICLR 2025
ETHICS STATEMENT
Our approach enhances the quality of portrait editing and enables users to create facial portraits in the styles they prefer. We encourage the responsible use of this technology, avoiding the generation of output that could be deemed inappropriate or harmful.
REPRODUCIBILITY STATEMENT
We have made significant efforts to ensure the reproducibility of the results presented in this paper. Detailed instructions for dataset generation are provided in Appendix B Model training procedures are outlined in Section 3 while the main experiments and evaluation are discussed in Section 4 Additional experimental results can be found in Appendix E Moreover, a demo video is also available for download the Supplementary Materials. We hope these resources will facilitate the replication of our findings and encourage further research building on our work.
REFERENCES
Yuval Alaluf, Omer Tov, Ron Mokady, Rinon Gal, and Amit Bermano. Hyperstyle: Stylegan inversion with hypernetworks for real image editing. In Proceedings of the TEFE/ /CVF conference on computer Vision and pattern recognition, pp. 115111 2022.
Qingyan Bai, Yinghao Xu, Zifan Shi, Hao Ouyang, Qiuyu Wang, Ceyuan Yang, Xuan Wang, Gordon Wetzstein, Yujum Shen, and Qifeng Chen. Real-time 3d-aware portrait editing from a single image. Xiv preprint 2rx: 1 14000, 2024.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instruclpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/ IEEE/CVF /CVF Conference on Computer Vision and Pattern Recognition, pp. 18392-18402, 2023.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780 8794, 2021.
William J Fitzsimmons, Robert J Woods, John T McCrone, Andrew Woodman, Jamie Arnold, Madhumita Yennawar, Richard Evans, Craig E Cameron, and Adam S Lauring. A speed-fidelity trado-off determines the mutation rate and virulence of an ma virus. PLoS biology, 16(6):e2000)))) 22006459,,, 2018
Shuyang Gu, Jianmin Bao, Hao Yang, Dong Chen, Fang Wen, and Lu Yuan. Mask-guided portrait editing with conditional gans. In Proceedings of the DEEE/CVP conference on computer vision and pattern recognition, pp. 3436-3445, 2019.
Jia Guo and Jiankang Deng. Insightface: 2d and 3d face analysis project, 2019.
Amir Hertz, Ron Mokady, Jay Tenenbaum, Kfr Aberman, Yael Pritch, and Daniel Cohen-
Or. Prompt-tee toonppt-o-prompt image editing with cross attention control. ar preprint orxi::2008.0 01626, 2022.
Jack Hessel, Ari Holtzman, Maxwell Forbes, Ronan Le Bras, and Yejin Choi. Clipscore: A reference-free evaluation metric for image captioning. Xiv preprint ar 2i:i::2004 000010014111 2021.
Xuan Ju, Ailing Zeng, Chenchen Zhao, Jianan Wang, Lei Zhang, and Qiang Xu. Humansd: A native skeleton-guided diffusion model for human image generation. In Proceedings of the IEEE/CU /CVF International Conference on Compuler Vision (ICCV), pp. 15988 15988-15998, October 2023.
Minguk Kang, Jun-Yan Zhu, Richard Zhang, Jacsik Park, Eli Shechtman, Sylvain Paris, and Taesung Park. Scaling up gans for text-to-image synthesis. In Proceedings of the IEEE/ CVF Conference on Computer Vision and Pattern Recognition, pp. 10124-10134, 2023.
11

---

Published as a conference paper at ICLR 2025
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE/CW /CFF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Hyunsu Kim, Yunjey Choi, Junho Kim, Sungjoo Yoo, and Youngjung Th. Exploiting spatial dimensions of latent in gan for reme-time image editing. In Proceedings of the IEEE/CV CVF Conference on Computer Vision and Pattern Recognition, pp. 852- 861, 2021.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXio:1312.6114, 2013.
Bingchuan Li, Tianxiang Ma, Peng Zhang, Miao Hua, Wei Liu, Qian He, and Zili Yi. Reganie: rectifying gan inversion errors for accurate real image editing. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1269 1277, 2023a.
Sicheng Li, Keqiang Sun, Zhixin Lai, Xiaoshi Wu, Feng Qiu, Haoran Xie, Kazunori Miyata, and Hongsheng Li. Ecnet: Effective controllable text-to-image diffusion models. arv preprint or 0rXiv:2408.184. 2024.
Zhen Li, Mingdeng Cao, Xintao Wang, Zhongang Qi, Ming-Ming Cheng, and Ying Shan. Photomaker: Customizing realistic human photos via stacked id embedding, arXio preprial ar Xix:222212.00 2.4461 2023b.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems,
2022.
Chenlin Meng, Yutong He, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Guided image synthesis and editing with stochastic differential equations. arXiv preprint Xix::21008 210.0008,, 2021.
Chenlin Meng, Robin Rombach, Ruiqi Gao, Diedcrik Kingma, Stefano Ermon, Jonathan Ho, and Tim Salimans. On distillation of guided diffusion models. In Proceedings of the IEEE/ CCV Conference on Computer Vision and Pattern Recognition, pp. 14297 14306, 2023.
Maxime Oquab, Timothée Darcet, Théo Moutakamni, Huy Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304 07193, 2023.
Jingtan Piso, Keqiang Sun, Quan Wang, Kwan- Yee Lin, and Hongsheng Li. Inverting generative adversarial renderer for face reconstruction. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 15619 15628, 2021.
Ben Poole, Ajay Jain, Jonathan T Barron, and Ben Mildenhall. Dreamfusion: Text-to-3d using 2d diffusion. arXiv preprint oxxi:22209.14000 14988, 2022.
Tiziano Portenier, Qiyang Hu, Attila Szabo, Siavash Arjomand Bigdeli, Paolo Favaro, and Matthias Zwicker. Faceshop: Deep sketch-based face image editing. arXiv preprint or Xiv: 08972, 2018.
Tim Salimans and Jonathan Ho. Progressive distillation for fast sampling of diffusion models. or Xio preprind ar 2ix:2220 20012, 2022.
Axel Sauer, Tero Karras, Samuli Laine, Andreas Geiger, and Timo Aila. Stylegan-t: Unlocking the power of gans for fast large-scale text-lo-image syuthesis. In International conference on machine leurning, pp. 30105-30118, PMLR, 2023a.
Axel Sauer, Dominik Lorenz, Andreas Blattmann, and Robin Rombach. Adversarial diffusion distillation. Xrxiv preprint ar Niv:2311 17042, 2023b.
12

---

Published as a conference paper at ICLR 2025
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recogration, pp. 815-823 2015.
Shelly
Sheynin, Adam Polyak, Uriel Singer, Yuval Kirstain, Amit Zohar, Oron Ashual, Devi Parikh, and Yaniv Taigman. Emu edit: Precise image editing via recognition and generation tasks. arXiv preprinl orXiv:2311 10089, 2023.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.0255 2020.
Keqiang Sun, Shangzhe Wu, Zhaoyang Huang, Ning Zhang, Quan Wang, and HongSheng Li. Controllable 3d face synthesis with conditional generative occupancy fields. Advances in Neural Information Processing Systems, 75:16331-16343, 2022.
Keqiang Sun, Shangzhe Wu, Ning Zhang, Zhaoyang Huang, Quan Wang, and Hongsheng Li. Cgof++: Controllable 3d face synthesis with conditional generative occupancy fields. IEEE Transactions Pattern Analysis and Machine Intelligence, 2023.
Ayush Tewari, Mohamed Elgharib, Florian Bernard, Hlans-Pele Seidel, Patrick Pérez, Michael Zollhöfer, and Christian Theobalt, Pie: Portrait image embedding for semantic control. ACM Transactions Graphics (TOG), 39(6):1-144 2020.
Haochen Wang, Xiaodan Du, Jiahao Li, Raymond A Ych, and Greg Shakhnarovich. Score jacobian chaining: Lifting pretrained 2d diffusion models for 3d generalion. In Proceedings of the IEEE/CVI Conference on Compuler Vision and Pullern Recognition, pp. 12619 12629, 2023.
Qixun Wang, Xu Bai, Haofan Wang, Zekui Qin, and Anthony Chen. Instantid: Zero-shot dlentity-preservinn generation in seconds. arXiv preprint arXiv:2401.07511, 2024a.
Zhendong Wang, Huangjic Zheng, Pengcheng He, Weizhu Chen, and Mingyuan Zhou. Diffusion-gan: Training gans with diffusion. arXiv preprint XxX::2206.0226,, 2022.
Zhengyi Wang, Cheng Lu, Yikai Wang, Fan Bao, Chongxuan Li, Hang Su, and Jun Zhu. Prolifiddremme: High-fidelity and diverse text-to-3d generation with variational score distillation. Advances in Newral Information Processing Systems, 36, 2024b.
Haoning Wu, Zicheng Zhang, Weixia Zhang, Chaoleng Chen, Liang Liao, Chunyi Li, Yixuan Gao, Annan Wang, Erli Zhang, Wenxiu Sun, et al. Q-align: Teaching Imms for visual scoring via discrete text-defined levels. arXiv preprint ar Xrv:2233 70900 2023a.
Xiaoshi Wu, Yiming Hao, Keqiang Sun, Yixiong Chen, Feng Zhu, Rui Zhao, and Hongsheng Li. Human preference score v2: A solid benchmark for evaluating human preferences of lext-to-image synthesis. arXiv preprinl. ar w.Xiv:2306.093// 2023b.
Xiaoshi Wu, Yiming Hao, Manyuan Zhang, Kegiang Sun, Zhaoyang Huang, Guanglu Song, Yu Liu, and Hongsheng Li. Deep reward supervisions for tuning text-to-image diffussin models. arXiv preprint Xrxv::2405. 00760, 2024.
Weihao Xia, Yujiu Yang, Jing- Hao Xue, and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 2256-2265, 2021.
Guangxuan Xiao, Tianwei Yin, William T Freeman, Fródo Durand, and Song Han. Fastcom- poser: Tuning-fre free multi-subject image generation with localized attention. arXiv preprint 23X55:2305.1 10431, 2023.
Zhisheng Xiao, Karsten Kreis, and Arash Vahdat. Tackling the generative learning trilemma with denoising diffusion gans. arXiv preprint arXic:2112. 07804, 2021.
Shaoan Xie, Zhifei Zhang, Zhe Lin, Tobias Hinz, and Kun Zhang, Smartbrush: Text and shape guided object inpainting with diffusion model. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 22428 22437, 2023.
13

---

Published as a conference paper at ICLR 2025
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint (tt::2208.06722 2023.
Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually anmotated dataset for instruction-guided image editing. Advances in Neural Information Processing Systems, 36, 2024.
Lymin Zhang, Anyi Rao, and Mancesh Agrawala. Adding conditional control to text-to-image diffusion models. Proceedings of the IEEE/CV CCF International Conference on Computer Vision, pp. 3836 3847, 2023.
Qinsheng Zhang and Yongxin Chen. Fast sampling of diffusion models with exponential integrator. ar preprint arXw:22044 13902, 2022.
Jingyuan Zhu, Huimin Ma, Jiansheng Chen, and Jian Yuan. Few-shot image generation via masked discrimination. orXiv preprint arvi:22200.10 2022.
14

---

Published as a conference paper at ICLR 2025
SUPPLEMENTARY MATERIAL
A RELATED WORK
A.1 TEXT-BASED IMAGE EDITING
Recent advancements in image editing leverage pre-trained text-to- text-to-image diffusion models. The Prompt-to-Prompp (P2P) Hertz et al. (2022) utilizes cross-attention maps to enable targeted edits based on text prompts. In contrast, SmartBrush Xie et al. (2023) enhances impainting by integrating text and shape guidance. SDEdit [Mnng et al. (2021) b applies noise followed by denoising via a stochastic differential ial equation, without requiring specialized training. Further, large datasets have been generated to refine user interfaces and facilitate instruction- besteection-based editing. For instance, InstructPix2P15 Brooks et al. (2023) uses a synthetic dataset to train models for guided image editing. Similarly, MagicBrush Zhang et al. (2024) and Emu Edit (Sheynin et al. (2023) develop datasets and employ multi-task training to expand the capabilities and intuitiveness of image editing tools. Existing methods prioritize instruction accuracy but neglect the specific needs of portrait editing, such as identity preservation. Our approach uniquely focuses on portrait image editing, ensuring identity integrity while accurately following instructions.
A.2 IDENTITY-
IMAGE GENERATION
ID -D-preserving image generation techniques, such as IGAR Piao et al. (2021) and CGOF series Sun et al. 2202 2023), focus retaining key facial attributes essential for identity recognition in practical applications. PhotoMaker Li ct al. (023b) enhances these features by adjusting Transformer layers and integrating class and image embeddings. IP-Adapter Ye et al. (2023) introduces a specialized cross-attention mechanism, leveraging reference images visual prompts to improve (ext-visual data integration. Instan(ID [ang et. al.) (2024a) incorporates Icentity to preserve detailed attributes from reference portraits. FastComposer Xiao al. (2023) employs localized cross-attention to prevent identity blending issues common in text-to-image models, ensuring more accurate feature representation. All methods share a common issuc: adding identity weakens the image's style due to inadequate exploration of balancing style and identity. Morcover, using only identity emboddings as imput without targeted supervision fails to effectively preserve identity.
A.3 STABLE DIFFUSION MODEL DISTILLATION
Diffusion models' iterative denoising steps hinder real-time application Li et al. (224):
:: Wul
ct al. (2024). To enhance their speed, approaches such as DPM-Solver Lu et al. (2022).
DDIM Song et al. (2020), and DEIS Zhang & Chen (2022) have been developed to accelerate
the sampling process. Other advancements include Progressive Distillation [Salimans & Ho (2022) and Guided Distillation Meng et al. (2023). which reduce the number of sampling steps from thousands to as few as 4-8. Additionally, recent innovations in model distillation, such Score Distillation have extended capabilities to 3D synthesis Poole et al. (022))
Wang et al. (202)) Moreover, the adversarial diffusson models Sauer et al. (2023b 1 Xiao
let al. [2021] further enhance performance by integrating GANs and adversarial training, marking significant strides in diffusion model efficiency and application scope. Although distillation methods enhance the speed of diffusion models, they inevitably lead to reduction in performance.
B DATASET
Due to a lack of publicly available datasets that adequately preserve identity for portrait image editing, we create specialized dataset. Each entry (cr,e7, x)) in this dataset includes an input image éı, an instruction prompt cr, and a target image x. Figure & outlines the framework for dataset generation, which includes prompt generation, image pairs generation, and image post-processio Table 3 includes the basic information of the dataset, including the size, image resolution, and face similarity score.
15

---

Published as a conference paper at ICLR 2025
Identity Prompt
age gender nationality skin color
ControlNct
Instruction Prompt
GPT-4
Canny Edge
style makeup hair cloth
Input Image
accessories background
Target
Image
Ideotity
x IP-Adapter
Prompt
Target Prompt
young Chinese woman, with long hair, wearing pale red lipstick, in Egyptian style, in front of a museum.
SDXL
Target Image
Prompt Generation
Image Generation
Figure 8: Dataset Generation Pipeline
Table 3: Basic Information of the synthetie training datasel
Face-Stmilacity
Data Pair Number
Image Resolution
PaceNet InsightFace
Dataset
10,000,000
1024x576
0.749
0.576
B. PROMPT GENERATION
From Figure we design identity prompts to define human attributes such as age, gender, nationality, and skin color for generating imput images, resulting in 10 million unique prompts. Instruction prompts, crafted using GPT-4, direct the transformation of these input images into various styles, makeups, hairstyles, and backgrounds. We generate in total of million instruction prompts. We then combine these identity and instruction prompts and sample 10 million target prompts, which are used for generating target images.
B.2 IMAGE PAIR GENERATION
We deviate from traditional methods that first generate the input image and then create the target image using instruction prompts. Our approach starts by creating the target image using the SDXL model without control inputs adapters, ensuring the generated image closely aligns with the instruction prompt, thereby improving text alignment in our model. The input image is subsequently produced under controlled conditions based the target image, as illustrated in Figure To improve identity and pose consistency between the images, utilize IP- IP-Adapter Ye et al. (2023), fine-tuned the target images, and ControlNet Zhhang et al. (2023), leveraging canny edge. However, identity preservation between the input image and the target. image is still insufficient. This issue is effectively addressed by introducing a novel identity loss in Section 3.1.1] Using the prompt sets described in Section B.1 we generate 10 million image pairs within this framework.
C QUALITATIVE COMPARISON BETWEEN IDE-NET AND IPNET
As illustrated in Figure the first row presents the outputs of IDE-Net, while the second row showcases the outputs of IPNet. While IDE-Net demonstrates slightly superior identity preservation, IPNet excels in overall image quality and style attributes, such makeup, glasses, and masks. These qualitative observations are consistent with the quantitative results reported in Table 2
16

---

Published as a conference paper at ICLR 2025
Input
The first row the output of IDE-Net and the second row the output of IPNet
Figure 9: Qualitative Comparison between IDE-Net and IPNet
D IMAGE DIVERSITY COMPARISON BETWEEN IDE-NET AND IPNET
Intra-LPIPS (Intra- Perceptual Image Patch Similarity) is metric for evaluating the diversity of generated images Zhu et al. (2022). To compute Intra-LPIPS for model, we generate multiple images from a single portrait input using different instruction prompts and calculate the pairwise LPIPS among the generated images. For multiple portrait inputs, the mean Intra-LPIPS computed by averaging the Intra-LPIPS values across all inputs. The mean Intra-LPIPS scores are shown in Table g
The mean Intra-LPIPS score of the student model IPNet higher than the teacher model IDE-Net, because we applied the proposed loss functions during the distillation process. According to Table 2 IPNet outperforms IDE-Net in text fidelity (including facial style) and image quality, although it shows slight regression in face similarity. The improved style fidclity but weaker identity preservation indicates that, for the same input image and different text prompts, the variations among the images generated by the IPNet greater than those produced by the IDE-Net. As a result, the student model IPNet achieves a higher Intra LPIPS score.
Table 4: Intra Intra-LPIPS of IDE-Net and IPNet
Metrics
IDE-Net
IPNet
Intra-LPIPS
0.571
0.589
E ANNEALING ALGORITHM SELECTION FOR ANNEALING IDENTITY Loss
We tried linear annealing Thag algorithms for Annealing Identity Loss. As demonstrated in Table 5 linear annealing algorithms outperform cosine anncaling in Text-Fidelity and Image- Quality but show lower Simiiariil. Given the already high scores in Pace-Similarity which render the visual distinction between the oulput and input identities negligible, we prioritize superior text fidelity and image quality by adopting Tiaer
ort +b and cosine annealing 0.5* cloinee(pi ( + Timas
Table Quantitative results of the ablation experiment for Anncaling Identity Loss. The best results are in bold.
Face Similarity
Text- Mext-Fidelity
Inage Quality
Aunealing
FaceNet
InsightFace
CLtp-Vit-g
CL-P-Vit-H
HPS
Q-Align-Q
Q-Align-A
0.890
0.791
0.254
0.291
0.263
4.814
3.463
0.903
0.802
0.234
0.272
0.255
4.781
3.327
0.5 (1 cosine )
0.895
0.799
0.233
0.273
0.261
4.783
3.422
0.45 x (1 cosine(pi pi x T.l..) 0.1
0.911
0.809
0.226
0.269
0.249
4.781
3.309
17

---

Published as a conference paper at ICLR 2025
Table 6: Training Parameters
Model
Time step
Loss function
Training batch size Training steps
IDE-Net
[0. 999]
Cam + 0.7 Lxid
256
40k
IPNet (High time step)
4400 800]
Luav 1 Laiam + 0 Cuistes
256
25k
IPNet (Middle time step)
200, 400
Ludv + 0.3 Laiatil Cariptet
256
15k
IPNet (Low time step)
150, 200
Lody 0.3 Laiatul Larplet
2048
1k
F NECESSITY OF TWO-STAGE TRAINING
Directly incorporating the Annealing Identity Loss (Section 3.1.1p Adversarial Loss (Sec-
tion 3.2.1 h, Identity Distillation Loss (Section 3.2.2). and Face-Style Enhancing Triplet Loss
(Section 3.2.3 in stage would not allow it to match IPNet's performance due to the
following reasons:
First, model distillation is necessary for one-step inference. Without distillation from teacher model, training a one-step inference diffusion model IPNet with adversarial loss is similar to training GAN. While GAN is faster than the diffusion model, GAN suffers from issues like mode collapse and reduced diversity. This is because GANs rely on single-step inference, whereas diffusion models involve multiple steps of noise addition and denoising, which naturally enhance diversity. DDariwal & Nichol (2021) By using a diffusion model as teacher to distill GAN-like model with distillation loss, we can combine the strengths of both approaches: achieving the good diversity of diffusion models and the fast, one-step inference capability of GAN-like models. WWang et al. (2022)
Second, (wo-stage training is crucial for balancing identity preservation and facial style. Training diffusion model with all losses combined in a single stage does not achieve a proper balance between identity preservation and facial style. We experimented extensively with various loss weight combinations during single-stage training and observed that the model failed to converge effectively. Specifically, the Anncaling Identity Loss and the Faco-Style Enhancing Triplct Loss oscillated during training. This instability arises because identity preservation and facial style enhancement are inherently conflicting tasks, making it difficult for a model to learn both effectively in a single stage.
G TRAINING DETAILS
The model training parameters are provided in the Table 6 including the denoising sampling step, loss weights, training batch size, and training step.
G.1 IDE-NET TRAINING
For the teacher model training, the time steps are sampled within the range I0, 999]. We selected 0.7 as Anncaling Identity Loss weight to achieve a good balance between identity preservation and image quality. In addition, compared different annealing algorithms with different weights in Appendix
G.2 IPNET TRAINING
The training of the student model IPNet can be counted as three stages based on the sampled time steps:
High Time Step In this stage, the teacher model samples time steps within the range [400, 800j. The loss function in this stage combines Adversarial Loss and Identity Distillation Loss (stochastic noise sampling). The primary goal is to distill the overall structure and pose aligument. The weight of Identity Distillation Loss is set to 1 because the student model is just starting training, the discriminator is weak giving noisy feedback, and thus requires stronger supervision from the teacher. the weight is set to too small, like 0, the training of the student model is likely to collapse at early stage, shown in Figure i image (c).
18

---

Published as a conference paper at ICLR 2025
Middle Time Step The time steps in this stage are sampled within the range of [200, 400] The denoising target the teacher model in the high sampling step introduces the artifacts, which can be removed in the lower sampling time step in this stage. The pose alignment is further strengthened in this stage. Therefore, the time-step threshold is determined to be 200 by checking the accuracy of spatial alignment and image quality during training. The Identity Distillation Loss (stochastic noise sampling) weight is reduced to 0.3, to weigh more on adversarial loss and improve the image quality of few-step generation.
Low Time Step In this stage, the time step is sampled within the range of [150, 200] to refine details and enhance style. The loss function is expanded to include: Adversarial Loss, Identity Distillation Loss (DDIM inversion), and Face-Style Enhancing Triplet Loss. Building upon the solid pose alignment for the student model achieved in the high-time-step and middle- time-step stages, this stage is mainly for refining details and enhancing image quality and style: First, the Identity Distillation Loss (DDIM inversion) allows fino-grained distillation, which improves the text fidelity and image quality, compared to Identity Distillation Loss (stochastic noise sampling) in the previous stages; Second, the Face-Style Enhancing Triplet Loss added in this stage strengthens style consistency while preserving identity. A large batch size of 2048 used to incorporate a greater variety of facial styles within each batch, to stabilize training with the Face-Style Enhancing Triplet Loss. Therefore, the training steps of IPNet are reduced accordingly due to the large batch size.
H LIMITATION OF FACIAL EXPRESSION EDITING
Our approach effectively edits static features like makeup, accessories, style, clothes, and background. However, dynamic attributes like facial expressions are less explored. Facial expression editing was not the primary focus during dataset creation, resulting in a limited mumber of expression-related image pairs in dataset. Consequently, less effectively on expression editing (as shown in Figure 10) compared to other features including background, clothing, jewelry, hair, and makcup edits. However, it still surpasses state-of-the-arl methods in this domain, as demonstrated in Figure I1 with the examples of prompts including "smiling".
method performs
To address this limitation, we plan to expand our dataset: in future iterations by adding more diverse facial expression image pairs, such as "crying", "laughing", and "angry". This will significantly enhance the model's ability to handle facial expression editing. This improvement thcorctically fcasible, as the identity preservation loss, Anncaling Identity Loss (introduced in Section 3.1.1 is calculated at the embedding level rather than the pixel level, ensuring identily preservation while allowing adjustments to facial expressions.
19

---

Published as a conference paper at ICLR 2025
Input
IPNet Output
Figure 10: The output of IPNet with the instruction prompts including "smiling"
Horns forehcad, smiling, wearing tattered cloak adorned with symbols, fiery wasteland.
Circus clown, smiling, with red clown nose and red lips, bright makeup in yellow top hat.
Wearing an unusually designed dress made of silver metal, smiling, ncon city in the background.
Smiling, bright clothes, pop art style
Input
MagicBrush
Instruct Pix2Pix
IP-Control
IP-Control
InstantID
IPNet (1 step)
-1.5
-XL
Figure 11: Qualitative comparison with SOTA methods for the prompts including "smiling"
20

---

Published as a conference paper at ICLR 2025
I PROMPT EXAMPLES
The prompts used to generated images in Figure from left to right are below:
In the first row:
(1) Adorable figure crafted from paper mache
(2) A playful doodle with uneven lines, unusual proportions, and a basic grasp of perspective, drawn with crayons on rough paper
(3) A fusion of organic and mechanical elements in biomechanical style, highly detailed and intricate, with futuristic and cybernetic aesthotic
(4) Greek frescoes, showcasing ancient mythology with harmonious composition and meticulous detailing
(5) A graphic poster featuring beautiful face with plump lips and orange sunglasses, surrounded by multicolored flowers and vibrant hucs, radiating a retro 70's vibe
(6) A joyful clown with cheerful face, dressed in vibrant, colorful costume, set in lively circus environment with bold color contrasts and the circus the background
(7) Realistic Gothie fantasy scene with necromancer in ragged, bone-decorated garments, surrounded by a graveyard and animated skeletons
(8) A demon with intense red eyes and sharp, angular features, dressed in armor marked by glowing red runcs, standing in blazing infernal landscape with molten lava, smoky haze, and glowing ember effects
In the second row:
(1) A watercolor painting in the style of the 1920s Gatsby era
2) Valentinc's Day card inspired by 1950s clegance, showcasing a pink silk shirt, a heart and flower-filldd background, and a blend of artistic modern aesthetics
(3) Lowbrow art featuring a 2D rainbow, infused with bright, surreal elements and modern surrealism, bursting with rainbow colors
(4) Pharaoh of Ancient Egypt, wearing classic Egyptian makeup, embodying the culture and style of the ancient civilization
(5) Wearing lip gloss, a pink strapless tank top, a denim skirt, and a pink handbag slung over the shoulder, with pink sunglasses and street scene in the background
(6) An alchemist dressed in a hooded robe adorned with mystical symbols, set in a mysterious stone laboratory filled with vials, bubbling potions, flasks, and glowing gemstones
(7) Wearing luxurions Venetian masquerade mask with intricate designs, in an ornate palace garden with fountains
(8) Wearing rave makeup and a vibrant rave outfit, with a laser show lighting up the background at the beach
21

---

Published as a conference paper at ICLR 2025
MORE GENERATED IMAGES BY IPNET
Htn
BIGIS
EMEERS
LT
Figure 12: More Generated Images by IPNet.
22

---

Published as a conference paper at ICLR 2025
BMH
FEEE
6tF
F
OS
Figure 13: More Generated Images by IPNet.
23

---

Published as a conference paper at ICLR 2025
K RESOURCES AND PRE-TRAINED MODELS
We provide the details of the resources and pre-trained models used this work, along with their respective URLs in Table
Model Reesuure
Details and URL
IP-Control-XL
IPAdapter (ip-adapter-facent) sdul. sdul:bin):
http:://huggiingfacccc.//////
-P-Adapter-Facel)
http:://bagiigffae..c/ddfferss controlnet-ccamy--theee soannysdxx-1.0
ControlNel-al (connyy:
IP-Control-1.5
IPAdapter I (ip- sd15.bin). https://hugginngaccccc///////
IP- UP-Adapter-Facell
ControlNet-1.5 (control sd15 canny.pth):
https://hugiingtacc.co
IllyasvieelCoatrrolllttt
Instruct-Pix2Pix
jiffusers/ddd-iinttucchhhppppp- 768:
http://haggingfcc...offfuusss
sddl-Insttructppii/p111118
MagicBrush
Instant
Table Resources and Pre-trained Models with URLs.
24


Reviews:
[{"review_id": "Dd612bTGaR", "reviewer": "Reviewer_15Pc", "summary": "This paper tries to tackle the task of portrait editing with identity preservation and style consistency.  It first trains a full-step diffusion model to produce images with robust identity preservation. Then it trains a one-step model that is distilled from the full-step model to further enhance the style, and identity of the edited images.", "questions": "Please list the details of all the comparison methods, including which codebase you are using, which model checkpoint, etc.", "limitations": "", "rating": 8, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The visual results are impressive. Many details are created, while the identity is well-preserved. \n2. The approach design is reasonable. It first generates the identity-preserved image, then distills the model for efficient inference. \n3. The experiments are convincing.", "weaknesses": "Several limitations can be improved to make this paper better.\n1. The dataset creation process is reasonable. However, it is still not very clear to me how this dataset is used. The authors mention that when creating the dataset, they use the AID loss to preserve identity. Then what data is used to train the first stage? Is the data from another model that also uses the AID loss? \n2. There are many losses included in the training. How are they tuned? Is there a guideline for the tuning of the hyper-parameters?\n3. Using the distilled version of the model usually results in a reduced diversity of styles and texture details. If the diversity can be further evaluated, this work can be more convincing.  \n4. For the facial expression control, any thoughts on why the current model cannot achieve it and any future plan on how to achieve it?\n5. Could you list the source code link you use for IP-Control, which also achieves impressive results?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AB4HpZDPqk", "reviewer": "Reviewer_QoMT", "summary": "This paper introduces the Instant-Portrait Network (IPNet), a one-step diffusion-based real-time portrait image editing model. The proposed IPNet can address challenges such as identity preservation and editing fidelity. The training process has two stages. The first stage is to train an Identity Enhancement Network for robust identity preservation. The second stage uses a novel Diffusion Multi-Objective Distillation approach to train the IPNet with multiple losses. Extensive comparisons demonstrate that IPNet outperforms existing models in identity consistency, instruction fidelity, and speed.", "questions": "1. This paper introduces a dataset for portrait editing. Will it be open-sourced? Making it publicly available would greatly benefit the development of this research area.\n2. In constructing the dataset, the authors used IPAdapter combined with ControlNet. Similarly, Table 1 compares the performance of IPAdapter with ControlNet. Could you clarify whether the base models for IPAdapter and the choices for ControlNet in these two instances are the same? Different base model selections will undoubtedly lead to varying performance outcomes.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper constructs a paired dataset for portrait image editing, consisting of 10,000,000 pairs of images at a resolution of 1024x576. This extensive dataset is invaluable for both this paper and future research in the field of portrait editing. Additionally, the supplementary materials provide a detailed description of the dataset construction process.\n2. This paper presents significant strengths in its approach to portrait image editing. It achieves one-step instruction-based editing with high precision in identity preservation, accurate execution of editing instructions, and fast inference.", "weaknesses": "1. The description of the methodology in this paper lacks clarity and omits important details, which may confuse readers. For instance, in section 3.2.2 on Identity Distillation Loss, there is insufficient explanation regarding how the teacher and student models sample time steps during the training process, as well as how the IPNet is distilled to achieve one-step inference. Providing these details would significantly enhance the readers’ understanding of the proposed method.\n2. The paper does not provide a comprehensive explanation of the model and experimental details, lacking important information such as the weights assigned to the different loss functions and specific training details for the two stages. Including this information would greatly enhance the clarity and completeness of the work.\n3. This paper confines the task to portrait editing under fixed poses, only altering attributes such as clothing, makeup, and style without the capability to generate new poses or expressions. However, the comparison models, IPAdapter and InstantID, offer more versatile functionalities, including the ability to generate new poses and expressions. While IPNet demonstrates superior performance within the scope of this paper, it’s possible that IPAdapter and InstantID, when fine-tuned on the dataset presented in this paper, could achieve comparable results to IPNet.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiYbervdJ7", "reviewer": "Reviewer_B5oG", "summary": "This paper proposes a one-step text-based portrait image editing method, which can achieve the balance between identity preservation and text alignment. They firstly train an Identity Enhancement Network (IDE-Net) to ensure robust identity preservation. Then, they propose a diffusion Multi-Objective Distillation process to distill IDENet into IPNet, achieving one-step instruction-based portrait image editing.", "questions": "1. Is IDE-Net purely a reconstruction model? Does it take text as input, and could you provide some sample results?\n2. If we directly train IDE-Net with the adversarial loss from 3.2.1, the triplet loss from 3.2.3, and the original IDENet losses, would the results surpass those of IPNet?\n3. When performing distillation, since IPNet only involves a single step of sampling, how is the number of noise-adding and denoising steps for IDE-Net determined? \n4. Given that the proposed dataset is paired, why wasn’t diffusion loss, some photometric loss, or the losses used in IDENet, applied when training IPNet? Can using those losses provide a more accurate supervision signal than IDENet’s distillation loss?\n\n\nI look forward to the author’s response, and I am willing to raise my score if some of my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. They propose to distill IDENet into IPNet with the diffusion Multi-Objective Distillation process, achieving one-step instruction-based portrait image editing.\n2. They propose an Annealing Identity Loss that balances identity preservation and text alignment.\n3. They design a dataset generation pipeline to generate a large-scale paired dataset, where each pair contains: 1) an input image; 2) a text prompt; and 3) a target image generated by SDXL with IPAdapter and ControlNet.\n4. Experiment results show that they can generate high-quality image editing results with better identity preservation.", "weaknesses": "1. This paper primarily allows editing only the style of portraits; it is not suitable for modifying expressions, poses, or the subject’s position within the image, which imposes certain limitations.\n2. The necessity of IDENet remains unclear. Can you provide a more detailed justification for training IDENet rather than using existing methods like IPAdapter or InstantID, and discuss how this choice impacts the overall performance of IPNet?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)