InstantPortrait: One-Step Portrait Editing via Diffusion Multi-Objective Distillation

Authors: Zhixin Lai, Keqiang Sun, Fu-Yun Wang, Dhritiman Sagar, Erli Ding
Primary area: generative models
Keywords: Portrait Editing, Diffusion Multi-Objective Distillation, On Step Inference, Identity Preservation, Text Fidelity

Abstract:
Real-time instruction-based portrait image editing is crucial in various applications, including filters, augmented reality, and video communications, etc. However, real-time portrait editing presents three significant challenges: identity preservation, fidelity to editing instructions, and fast model inference. Given that these aspects often present a trade-off, concurrently addressing them poses an even greater challenge. While diffusion-based image editing methods have shown promising capabilities in personalized image editing in recent years, they lack a dedicated focus on portrait editing and thus suffer from the aforementioned problems as well. To address the gap, this paper introduces an Instant-Portrait Network (IPNet), the first one-step diffusion-based model for portrait editing. We train the network in two stages. We first employ an annealing identity loss to train an Identity Enhancement Network (IDE-Net), to ensure robust identity preservation. We then train the IPNet using a novel diffusion Multi-Objective Distillation approach that integrates adversarial loss, identity distillation loss, and a novel Facial-Style Enhancing loss. The Diffusion Multi-Objective Distillation approach efficiently reduces inference steps, ensures identity consistency, and enhances the precision of instruction-based editing. Extensive comparison with prior models demonstrates IPNet as a superior model in terms of identity preservation, text fidelity, and inference speed.

Decision: accept (poster)
Reviewer scores: 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730026469949

thread_type:
official_review

<thread_object>
creator: reviewer_b5og

<content>
summary:
This paper proposes a one-step text-based portrait image editing method, which can achieve the balance between identity preservation and text alignment. They firstly train an Identity Enhancement Network (IDE-Net) to ensure robust identity preservation. Then, they propose a diffusion Multi-Objective Distillation process to distill IDENet into IPNet, achieving one-step instruction-based portrait image editing.

strengths:
1. They propose to distill IDENet into IPNet with the diffusion Multi-Objective Distillation process, achieving one-step instruction-based portrait image editing.
2. They propose an Annealing Identity Loss that balances identity preservation and text alignment.
3. They design a dataset generation pipeline to generate a large-scale paired dataset, where each pair contains: 1) an input image; 2) a text prompt; and 3) a target image generated by SDXL with IPAdapter and ControlNet.
4. Experiment results show that they can generate high-quality image editing results with better identity preservation.

weaknesses:
1. This paper primarily allows editing only the style of portraits; it is not suitable for modifying expressions, poses, or the subject’s position within the image, which imposes certain limitations.
2. The necessity of IDENet remains unclear. Can you provide a more detailed justification for training IDENet rather than using existing methods like IPAdapter or InstantID, and discuss how this choice impacts the overall performance of IPNet?

questions:
1. Is IDE-Net purely a reconstruction model? Does it take text as input, and could you provide some sample results?
2. If we directly train IDE-Net with the adversarial loss from 3.2.1, the triplet loss from 3.2.3, and the original IDENet losses, would the results surpass those of IPNet?
3. When performing distillation, since IPNet only involves a single step of sampling, how is the number of noise-adding and denoising steps for IDE-Net determined? 
4. Given that the proposed dataset is paired, why wasn’t diffusion loss, some photometric loss, or the losses used in IDENet, applied when training IPNet? Can using those losses provide a more accurate supervision signal than IDENet’s distillation loss?


I look forward to the author’s response, and I am willing to raise my score if some of my concerns are addressed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730260567485

thread_type:
official_review

<thread_object>
creator: reviewer_qomt

<content>
summary:
This paper introduces the Instant-Portrait Network (IPNet), a one-step diffusion-based real-time portrait image editing model. The proposed IPNet can address challenges such as identity preservation and editing fidelity. The training process has two stages. The first stage is to train an Identity Enhancement Network for robust identity preservation. The second stage uses a novel Diffusion Multi-Objective Distillation approach to train the IPNet with multiple losses. Extensive comparisons demonstrate that IPNet outperforms existing models in identity consistency, instruction fidelity, and speed.

strengths:
1. This paper constructs a paired dataset for portrait image editing, consisting of 10,000,000 pairs of images at a resolution of 1024x576. This extensive dataset is invaluable for both this paper and future research in the field of portrait editing. Additionally, the supplementary materials provide a detailed description of the dataset construction process.
2. This paper presents significant strengths in its approach to portrait image editing. It achieves one-step instruction-based editing with high precision in identity preservation, accurate execution of editing instructions, and fast inference.

weaknesses:
1. The description of the methodology in this paper lacks clarity and omits important details, which may confuse readers. For instance, in section 3.2.2 on Identity Distillation Loss, there is insufficient explanation regarding how the teacher and student models sample time steps during the training process, as well as how the IPNet is distilled to achieve one-step inference. Providing these details would significantly enhance the readers’ understanding of the proposed method.
2. The paper does not provide a comprehensive explanation of the model and experimental details, lacking important information such as the weights assigned to the different loss functions and specific training details for the two stages. Including this information would greatly enhance the clarity and completeness of the work.
3. This paper confines the task to portrait editing under fixed poses, only altering attributes such as clothing, makeup, and style without the capability to generate new poses or expressions. However, the comparison models, IPAdapter and InstantID, offer more versatile functionalities, including the ability to generate new poses and expressions. While IPNet demonstrates superior performance within the scope of this paper, it’s possible that IPAdapter and InstantID, when fine-tuned on the dataset presented in this paper, could achieve comparable results to IPNet.

questions:
1. This paper introduces a dataset for portrait editing. Will it be open-sourced? Making it publicly available would greatly benefit the development of this research area.
2. In constructing the dataset, the authors used IPAdapter combined with ControlNet. Similarly, Table 1 compares the performance of IPAdapter with ControlNet. Could you clarify whether the base models for IPAdapter and the choices for ControlNet in these two instances are the same? Different base model selections will undoubtedly lead to varying performance outcomes.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730677568637

thread_type:
official_review

<thread_object>
creator: reviewer_15pc

<content>
summary:
This paper tries to tackle the task of portrait editing with identity preservation and style consistency.  It first trains a full-step diffusion model to produce images with robust identity preservation. Then it trains a one-step model that is distilled from the full-step model to further enhance the style, and identity of the edited images.

strengths:
1. The visual results are impressive. Many details are created, while the identity is well-preserved. 
2. The approach design is reasonable. It first generates the identity-preserved image, then distills the model for efficient inference. 
3. The experiments are convincing.

weaknesses:
Several limitations can be improved to make this paper better.
1. The dataset creation process is reasonable. However, it is still not very clear to me how this dataset is used. The authors mention that when creating the dataset, they use the AID loss to preserve identity. Then what data is used to train the first stage? Is the data from another model that also uses the AID loss? 
2. There are many losses included in the training. How are they tuned? Is there a guideline for the tuning of the hyper-parameters?
3. Using the distilled version of the model usually results in a reduced diversity of styles and texture details. If the diversity can be further evaluated, this work can be more convincing.  
4. For the facial expression control, any thoughts on why the current model cannot achieve it and any future plan on how to achieve it?
5. Could you list the source code link you use for IP-Control, which also achieves impressive results?

questions:
Please list the details of all the comparison methods, including which codebase you are using, which model checkpoint, etc.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces a method for editing portrait images that keeps the person's identity clear while making the editing process quicker. It uses a two-step approach with a large dataset of 10 million image pairs to ensure the edits are accurate and preserve identity.

***Strengths:***
- The edited images maintain identity well and look impressive.
- The creation of a new dataset may contribute to the domain
- Techniques like Multi-Objective Distillation and Annealing Identity Loss are new and effective.
- The method allows quick editing without losing much quality.

***Weaknesses:***
- Some parts of the method are not explained clearly, which makes it hard to fully understand. This is critical and should be resolved in the final version.
- The paper doesn't thoroughly compare its results with other models that have more editing features.
- The system can only edit images in fixed poses and doesn't allow for changes in expression or pose.

Reviewers praised the paper and supported acceptance after their concerns were addressed in the rebuttal. The paper adds value to research on image editing with its methods and dataset, and it merits acceptance despite some limitations in scope and method clarity. After careful consideration and discussion, we are pleased to accept this submission.