DisControlFace: Adding Disentangled Control to Diffusion Autoencoder for One-shot Explicit Facial Image Editing

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-04 04:36:31

Abstract: In this work, we focus on exploring explicit fine-grained control of generative facial image editing, all while generating faithful facial appearances and consistent semantic details, which however, is quite challenging and has not been extensively explored, especially under an one-shot scenario.
We identify the key challenge as the exploration of disentangled conditional control between high-level semantics and explicit parameters (e.g., 3DMM) in the generation process, and accordingly propose a novel diffusion-based editing framework, named DisControlFace.
Specifically, we leverage a Diffusion Autoencoder (Diff-AE) as the semantic reconstruction backbone. 
To enable explicit face editing, we construct an Exp-FaceNet that is compatible with Diff-AE to generate spatial-wise explicit control conditions based on estimated 3DMM parameters.
Different from current diffusion-based editing methods that train the whole conditional generative model from scratch, we freeze the pre-trained weights of the Diff-AE to maintain its semantically deterministic conditioning capability and accordingly propose a random semantic masking (RSM) strategy to effectively achieve an independent training of Exp-FaceNet.
This setting endows the model with disentangled face control meanwhile reducing semantic information shift in editing.
Our model can be trained using 2D in-the-wild portrait images without requiring 3D or video data and perform robust editing on any new facial image through a simple one-shot fine-tuning.
Comprehensive experiments demonstrate that DisControlFace can generate realistic facial images with better editing accuracy and identity preservation over state-of-the-art methods.

Main paper: 
 introduction: 
 Facial image editing has long been a hot research topic in the fields of computer vision and computer graphics, where the key challenge is to effectively achieve fine-grained controllable generation of realistic facial images while preserving semantic face priors.3D Morphable Models (3DMMs) [3, 16] have been widely employed to represent variations in facial shape and texture [4, 16, 19, 29, [56] [57] [58] [59] , whereas their ability to capture personalized facial features is limited and the performance highly depends on the quality and diversity of the 3D face training data. On top of this, subsequent learning-based explicit face models [7, 11, 13, 28, 50, [56] [57] [58] [59] 63] achieve controllable generations of dynamic and expressive facial animations by capturing the nuances of facial features under different expressions, poses, and lighting conditions, nevertheless, can neither generate realistic facial appearances that correspond to the animated 3D face geometries nor model refined geometric details in non-facial regions, e.g., hair, eyes, and mouth. Follow-up efforts integrate explicit facial modeling with implicit 3D-aware representations like Neural Radiance Fields (NeRFs) to reconstruct animated realistic head avatars [14, 23, 30, 33, 34, 39-42, 51, 52, 66] , which however, heavily rely on 3D consistent data such as monocular portrait videos and tend to exhibit limited generalization.In contrast, generative face models enable single image reconstruction and editing due to the superior capability in learning rich face priors from in-the-wild portraits. Recent GAN-based approaches [2, 5, 6, 10, 18, 22, 37, 55, 62] incorporate explicit 3D facial priors and implicit neural representations to achieve directed generations of high-resolution, realistic, and view-consistent facial images without the need of 3D face scans or portrait videos. However, those methods mainly provide implicit or limited explicit controls of face generations.More recently, the diffusion-based framework [21] has emerged as the predominant choice for various generation tasks, owing to its impressive performance and diverse conditioning options. Some approaches have also shown promising enhancements in face reconstruction and various face editing tasks, such as face relighting [43] , semantic attributes manipulation [44] , and explicit appearance control [12] . Unfortunately, it can be seen that when editing and modifying some specific facial attributes, other facial attributes or editing-irrelevant details often occur unexpected and uncontrollable changes, leading to an incoherent and identity-altered generated face. This prevalent issue can be attributed to that these generative face models struggle to effectively perform disentangled control in the generation process.In this work, we propose a novel diffusion-based generative framework, namely DisControlFace to achieve one-shot editing of facial images. To generate a photo-realistic, high-fidelity facial appearance corresponding to specific explicit parameters (e.g., 3DMM) while faithfully preserving high-level semantic priors, we particularly focus on enhancing the diffusion model with disentangled conditional control. Specifically, we adopt a Diffusion Autoencoder (Diff-AE) [44] as the generative backbone, which can enable a deterministic image reconstruction by conditioning Denoising Diffusion Implicit Model (DDIM) [53] on the semantic information of the input image. We then specially construct an Exp-FaceNet compatible with the Diff-AE backbone, which further provides multi-scale, spatial-aware DDIM conditioning corresponding to the facial parameters of shape, pose, expression, and lighting. Moreover, we claim that training different DDIM conditoning together, as with existing methods is not conductive to learning disentangled face control. We therefore freeze the pre-trained weights of Diff-AE and accordingly design a random semantic masking (RSM) strategy to enable the training of Exp-FaceNet, by means of which the model can learn explicit parameteric face control independently without affecting semantically deterministic DDIM conditioning. Also benefiting from this disentangled setting, instead of relying on 3D or video data, we can utilize 2D in-the-wild portrait dataset such as FFHQ [26] to effectively train Exp-FaceNet to learn a robust and generalized capability in explicit face editing. Considering there exists domain gap between the pre-trained face data and the target new face image, which tends to prevent Diff-AE from performing near-exact semantic reconstrcution, we finally introduce an one-shot fine-tuning to Diff-AE so as to restore personal identity and editing-irrelevant details of the input portrait under a subjectagnostic scenario. Our approach not only achieves state-of-the-art (SOTA) qualitative and quantitative results for one-shot explicit facial image editing, but also supports generating realistic and faithful facial appearance of specific individuals in image inpainting, semantic attributes manipulations, and cross-identity face driving (shown in Figure 1 ).Our contributions can be summarized as follows:• We propose a novel diffusion-based generation framework, consisting of a Diffusion Autoencoder (Diff-AE) backbone and an explicit face control network (Exp-FaceNet) for synthesizing photo-realistic, high-fidelity portrait images corresponding to the editing of explicit facial properties only trained with 2D in-the-wild images. • To the best of our knowledge, we are the first to introduce a weight-frozen pre-trained Diff-AE to explicit face editing pipeline to provide deterministic semantic conditioning, meanwhile designing an effective training strategy to enable the Exp-FaceNet with a disentangled explicit parameteric (e.g., 3DMM) conditioning. • Our method achieves SOTA generation performance in explicit facial image editing, and also supports various one-shot face editing tasks. 
related work: 
 Generative Face Modeling. Various GAN-based models [6, 26, 27, 35, 38, 49] have been proposed to synthesize realistic facial images by learning the underlying data priors from large-scale in-the-wild images [6, 38, 49] . However, when it comes to precise control and interpretability, those generative face models fall short compared to explicit parametric models. Given this, several approaches [5, 54, 55, 62] go a step further by integrating explicit parameters and GANs, which can simultaneously generate highly realistic and coherent portraits and achieve fine-grained control of facial appearance. Recently, diffusion models [21, 53] have gained recognition for their superior ability to learn data distributions compared to GANs, and thus have been widely adopted to generate realistic and diverse images in various generation tasks, including facial image synthesis [44, 44] . DiffusionRig [12] , a closely related method, introduces pixel-aligned physical properties rendered from explicit parameters estimated by DECA [13] to denoising diffusion process to generate photo-realistic facial images corresponding to target pose, expression, and lighting conditions, which however, highly relies on a personalized fine-tuning (around 20 images) to preserve the facial appearance priors of a specific person . Similar problems widely exist in most conditional diffusion face models, as can be observed where identity shifts and unexpected attributes alterations may occur during face reconstruction and editing. This can be attributed to the lack of disentangled control capabilities when conditioning diffusion models with both facial semantics and physical information. DisControlFace overcomes this challenge by leveraging a weight-frozen pre-trained Diff-AE to provide semantically deterministic DDIM conditioning and independently training a separate Exp-FaceNet to learn disentangled face control based on 3DMM parameters. Conditional Diffusion Model. Conditional DDIM enables the Denoising Diffusion Probabilistic Model (DDPM) [21] to generate content consistent with specific control signals though various conditioning manners. Most existing models encode various control information into global conditional vectors, which can be text embedding [45] or semantic embedding [43, 44] . To achieve spatialaware and precise control of the generation, some approaches (e.g., DiffusionRig [12] and SR3 [48] ) concatenate various spatial conditions and denoised images together as the input of the U-Net noise predictor in each denoising step. However, this form of conditioning requires the U-Net to have a unique input layer, resulting in the model having limited generalization and making it hard to reuse existing well-trained diffusion models. Besides, some other methods specially construct spatial conditioning branches to extract spatial-aligned conditional features and insert them into the U-Net [43, 61, 65] . ControlNet [65] has been widely employed to add various spatial visual guidance (e.g., edge maps, pose maps, depth maps, etc.) to text-to-image generation models such as Stable Diffusion [45] . Whereas, it may not suitable for deterministic reconstruction or editing tasks since there still exists uncertainty and randomness in the generation controlled by visual guidance and text prompts. In contrast, our Exp-FaceNet is specially designed to be compatible with a semantic reconstruction DDIM, Diff-AE [44] , aiming to better address facial image editing based on explicit control information.Learning Specific Facial Priors. Effectively extracting the facial appearance priors of the specific person and injecting this global prior into the generation process is crucial for preserving facial semantics such as identity, accessories, hairstyle, and background information in facial image editing. Most existing generative methods address this issue by fine-tuning the network in various settings [15, 24, 36, 47] , or designing special optimization strategies, such as identity penalty, face recognition loss, and latent representation editing [1, 31, 60, 64] . In contrast to previous work like [12, 36] which collect personal albums to learn personalized facial priors, in this work, we focus on the subject-agnostic editing scenario, which is more challenging but practical. On the basis of the inherent and maintained facial semantics capturing capability of Diff-AE, only a fast and yet simple fine-tuning using an out-of-domain new face image is needed to faithfully restore the semantic appearance details of the target image during editing. 
preliminaries 3.1 3d morphable face models: 
 FLAME [29] , a popular 3DMM model, can be expressed as 𝑀 (𝜷, 𝜽, 𝝍) :R |𝜷 | × |𝜽 | × |𝝍 | → R 3𝑁, which takes shape 𝜷, pose 𝜽 , and expression 𝝍 as inputs and outputs a face mesh with 𝑁 vertices. On this basis, some off-the-shelf 3D face estimators, such as DECA [13] and EMOCA [7] , achieve 3D face reconstruction by regressing individual-specific FLAME parameters from in-the-wild images. We utilize EMOCA to obtain a face mesh corresponding to the input image with enhanced expression consistency, and render it to pixel-aligned explicit conditions. Specifically, given a single image 𝐼 , the coarse branch of EMOCA estimate its corresponding 𝜷, 𝜽 , 𝝍, albedo 𝜶 , spherical harmonic (SH) illumination coefficient 𝒍, and camera 𝑐, which can be expressed as 𝐸 𝑐 (𝐼 ) → (𝜷, 𝜽, 𝝍, 𝜶 , 𝒍, 𝒄). The detailed branch further outputs a detail vector 𝛿 and computes the displacement map 𝐷 in UV space, which is specifically expressed as 𝐸 𝑑 (𝐼 ) → 𝜹 and 𝐹 𝑑 (𝜹, 𝝍, 𝜽 𝑗𝑎𝑤 ) → 𝐷.Diff-AE [44] reformulates the traditional diffusion generation model into an autoencoder and captures high-level image semantics for DDIM conditioning, therefore supporting near-exact reconstruction and attribute manipulation of the input image. Specifically, Diff-AE uses a semantic encoder to generate a 512-dimentional latent code 𝑧 with global semantics of the input image 𝑥 0 . Then, 𝑧 can be introduced to the reverse deterministic generative process of DDIM to obtain a noisy map 𝑥 𝑇 which captures the stochastic variations of 𝑥 0 . Last, a conditional DDIM model decodes (𝑧, 𝑥 𝑇 ) to achieve a deterministic reconstruction of 𝑥 0 . By linearly modifying the semantic latent code 𝑧, Diff-AE can manipulate diverse global semantic attributes, e.g., age, gender, and hairstyle. We introduce Diff-AE to our DisControlFace as the reconstruction backbone and freeze its pre-trained weights to provide semantically deterministic DDIM conditioning during the training of explicit face control. 
method: 
 In the pursuit of a robust one-shot explicit facial image editing, we propose a generative framework, namely DisControlFace, providing DDIM conditioning on disentangled face control between highlevel semantics and explicit 3DMM parameters (shown in Figure 2 ). Specifically, we adopt a weight-frozen Diff-AE as a semantic reconstruction backbone and construct an Exp-FaceNet to provide explicit parametric face control (Sec. 4.1). Furthermore, we design a training strategy to effectively enable the training of Exp-FaceNet (Sec. 4.2). Finally, we adopt an one-shot fine-tuning to improve the semantic consistency and faithfulness of the generated facial image under the subject-agnostic editing scenario (Sec. 4.3).On the basis of the adopted Diff-AE reconstruction backbone, an intuitive idea for learning explicit face editing capability is to further build additional DDIM conditioning on 3DMM parameters. Compared to directly adopting the 3DMM parameters as non-spatial control conditions, generating pixel-aligned conditional maps based on those parameters is more conducive and compatible to convolutionbased visual representation, which also helps enable fine-grained spatial control for the denoising diffusion process. Given this, we specially construct Exp-FaceNet, an explicit control network compatible with the adopted Diff-AE reconstruction backbone to perform a disentangled spatial conditioning for DDIM-based generation. Here we firs estimate 3DMM parameters from facial images and transfer them to the corresponding visual guidance map. Specifically, we use EMOCA [7] to predict FLAME parameters (including shape 𝛽, pose 𝜃 , and expression 𝜓 ), SH illumination parameter 𝑙, and camera parameter 𝑐 from an input portrait. Different from those previous work [12, 17, 43] , here we also adopt the person-specific detail vector 𝛿 estimated by EMOCA, which can be combined with 𝜃 and 𝜓 to generate the expression-dependent displacement map for refining the face geometry with animatable wrinkle details. To avoid undesired disturbance to facial appearance priors caused by inaccurate and unrealistic appearance estimation, we set the albedo map 𝛼 to a constant gray value, thereby focusing on controlling the edit of shape, pose, expression, and lighting. We render these explicit parameters into a surface normal snapshot 𝑛:EQUATIONwhere the FLAME model M is used to calculate the 3D face mesh, G and R indicate normal calculation function and the Lambertian reflectance renderer, respectively. By means of this, 𝑛 can reflect the fine-grained geometry of the input face and is compatible with pose parameter 𝜃 and expression parameter 𝜓 . Furthermore, we also generate a shading shape snapshot 𝑠 to illustrate the lighting conditions associated with the SH illumination parameter 𝑙:EQUATIONConsidering U-Net [46] excels at extracting spatial features from images for various vision tasks, we construct Exp-FaceNet in a similar U-shape structure, which takes channel-concatenated snapshots 𝑛 and 𝑠 as the input visual guidance map and generates multi-scale deep features for spatial-aware conditioning. Then we feed back the spatial-condition features outputted by each stage of the U-Net's decoder back to Diff-AE to provide multi-scale conditional control.Please see the detailed architecture in the supplement.DisControlFace can be regarded as a DDIM model that is simultaneously conditioned by a global semantic code and multi-scale explicit-control feature maps (see Figure 2 ). As mentioned before, we freeze the pre-trained weights of Diff-AE to preserve global semantics and enable Exp-FaceNet to learn explicit face control in a disentangled way. However, since Diff-AE backbone already allows a deterministic image reconstruction under this setting, only limited gradients can be generated during error back-propagation, which are far from sufficient to effectively train Exp-FaceNet. Consequently, it is infeasible to train Exp-FaceNet in a traditional conditional DDIM generation form. To address this issue, we design a random semantic masking (RSM) strategy, not for the purpose of representation learning like Masked Autoencoders (MAE) [20] , but rather to achieve the training of Exp-FaceNet effectively. Concretely, we divide the input image 𝑥 0 into regular non-overlapping patches and randomly mask different portions of patches to obtain a masked image 𝑥 𝑚 0 at different timesteps. By means of this, the semantic latent code 𝑧 𝑚 encoded by the semantic encoder E 𝜂 of Diff-AE only contains fragmented and incomplete content and spatial information of 𝑥 0 . Meanwhile, the spatial-condition features 𝑓 generated by Exp-FaceNet F 𝜙 comprise the fine-grained face shape as well as the camera parameter and lighting condition of 𝑥 0 , which can help to restore the masked face regions in 𝑥 𝑚 0 in each random denoising timestep. The overall training objective can thereby be Figure 3 : Qualitative comparison against baselines in one-shot editing. For each selected image, we use EMOCA [7] to estimate the corresponding explicit parameters, then synthesize the edited images using different methods based on the modified parameters of pose, expression, and lighting. We additionally provide the rendered shading shapes in the second row as the references of explicit control conditions. As can be seen, our DisControlFace can edit images that match well with the target control conditions while faithfully synthesizing facial appearances and editing-irrelevant details.parameterized as:EQUATIONwhere 𝜖 𝜃 is the U-Net of Diff-AE which predicts the noise 𝜖 ∼ N (0, I) added in noisy image 𝑥 𝑡 . Throughout the generalized training, we only train F 𝜙 and freeze 𝜖 𝜃 and E 𝜂 with the pre-trained weights.At this point, the well-trained Exp-FaceNet is able to explicitly change pose, expression, and lighting of a facial image. However, there still exists identity shift or background changes during the face editing, which is especially evident when the target face for editing lies outside the domain of the pre-trained Diff-AE. Given this, it is meaningful to fully exploit the semantic priors of the to-beedit image and inject them into the editing process. Concretely, in this stage, we freeze Exp-FaceNet and only fine-tune Diff-AE with the input portrait image using the aforementioned RSM training strategy. As a result, this one-shot fine-tuning enables the model to faithfully restore the personalized appearance details as well as editing-irrelevant factors such as background and accessories when performing explicit face editing.In practice, we first use EMOCA to predict all explicit parameters (mentioned in Sec. 4.1) of the input portrait 𝑥 0 , then we modify the pose parameters 𝜃 , expression parameter 𝜓 , and SH light parameter 𝑙 by manually setting target values or directly transferring these parameters from a driving portrait. After this, we calculate the rendered shading shape snapshot 𝑠 and surface normal snapshot 𝑛 based on the modified parameters, and further generate explicit control conditions using Exp-FaceNet. On the other hand, since we should keep a consistent generative mechanisms in training and inference, here we also utilized masked input images to provide high-level semantic conditioning for DDIM decoder and accordingly design an intuitive patch masking strategy for inference editing.ID ↑ Shape↓ Pose ↓ Exp ↓ Light ↓ GIF [17] 0.22 3.0 5.6 5.0 0.40 DiffusionRig [12] 1 : Quantitative comparisons against compared baselines using identity consistency (ID) and DECA re-inference errors on shape, pose, expression, and lighting.Concretely, for each timestep 𝑡, we generate the masked image 𝑥 𝑚 𝑡 by randomly masking the patches of the input image 𝑥 0 with a linear ratio 𝜌 𝑡 = 0.75 -0.5(𝑇 -𝑡)/𝑇 , where the number of the inference denoising steps 𝑇 is set to 20 in this work. Under this setting, we can generate 𝑥 𝑚 𝑡 with high masking ratios 𝜌 𝑡 to emphasize the facial control of the intermediate denoising result 𝑧 𝑡 -1 in the early stage of the inference, and then gradually decrease 𝜌 𝑡 to recover semantic information. 
experiments 5.1 implementation details: 
 We train the proposed Exp-FaceNet on the FFHQ dataset [26] , which consists of 70k in-the-wild facial images. For evaluations, we select the images of the CelebA-HQ dataset [25] to perform one-shot fine-tuning and face editing. To balance generation quality and computational cost, we resize the images to a resolution of 256×256 for both training and inference. Accordingly, we utilize Diff-AE 1 pre-trained on FFHQ-256 for all experiments. We train Exp-FaceNet for 437,500 iterations, with a learning rate of 1𝑒 -4 and a batch size of 32, while during the one-shot fine-tuning stage, we only finetune the pre-trained Diff-AE for 1,500 iterations, with a learning rate of 1𝑒 -5 and a batch size of 4. In all training stages, we use AdamW [32] as the optimizer and set the denoising timesteps to 1,000.Baselines. We compare our methods against three generative methods for parametric face image synthesis and editing: Head-NeRF [23] , GIF [17] , and DiffusionRig [12] . Among these, HeadNeRF is a NeRF-based head model, while GIF and DiffusionRig are both generative face models built upon GAN and diffusion model, respectively. For a fair comparison, we utilize the models pre-trained on FFHQ at a resolution of 256 2 for each method. Qualitative comparison. We evaluate our DisControlFace against baselines using images from CelebA-HQ and we perform one-shot fine-tuning for all methods. The qualitative comparison results are provided in Figure 3 . For all methods, we visualize the editing results of pose, expression, and lighting on three identities. In order to intuitively measure the editing performance, we further give the rendered snapshot 𝑠 of the shading shape corresponding to the modified explicit parameters as the references of explicit control conditions. (second row of Figure 3 ). We can find that compared to the other methods, our method synthesizes images with overall best identity consistency and parametric editing accuracy. Specifically, HeadNeRF cannot generate realistic facial appearance as well as the background of the original image. GIF has a good control ability of parametric face editing, however, is completely unable to preserve the face identity. DiffusionRig, another diffusion-based method achieves better editing results than HeadNeRF and GIF, especially in identity preservation, which can also be attributed to using one-shot fine-tuning to boost the diffusion model with more personal facial priors. Nevertheless, since DiffusionRig trains the whole model together from scratch which prevents the model from learning robust disentangled control, the edited results still have visible identity shift (e.g., eyes, skin color, and hair) especially in lighting editing.Quantitative comparisons. Table 1 (top part) provides the quantitative editing results of all methods on 1000 in-domain images of FFHQ. Following previous work [9, 12, 17] , we apply DECA reinference on edited images and calculate the Root Mean Square Error (RMSE) between the input and re-inferred face vertices as well as spherical harmonics to evaluate the editing accuracy on shape, pose, expression, and lighting. We additionally measure the identity consistency (ID) score by computing the cosine similarity between the deep features generated by ArcFace [8] of the original and edited images. The results indicate that our method achieve the overall best performance on all metrics. Note that DisControlFace outperforms other methods by a large margin in ID score, which demonstrates the superiority of our method in identity preservation during one-shot editing.One-shot fine-tuning. Based on the deterministic semantic reconstruction capability of Diff-AE [44] , DisControlFace can extract global semantics of the input face image. However, since the target portrait image tends to have different face priors with pre-trained Diff-AE, there still exists semantic information shift under the zeroshot editing scenario, as shown in Figure 4 . Given this, we adopt a simple one-shot fine-tuning on the pre-trained Diff-AE to fully exploit the semantic priors of the to-be-edit image and inject them into the editing process. We can observe that by means of this, the face identity and other high-level semantic information (e.g., background, accessories, and hair) can be well preserved.Effectiveness of RSM training. In this paper, we achieve a disentangled training of the proposed Exp-FaceNet by freezing the pre-trained weights of Diff-AE and according designing a RSM training strategy. To demonstrate the necessity of the proposed RSM training, we separately train Exp-FaceNet with and without random patch masking for the input image of semantic encoder of the Diff-AE backbone. Figure 5 shows that both training strategies enable the model to reconstruct the input image. However, only the model trained with PSM strategy can generate images with novel poses. This result is consistent with our claim that since the pre-trained Diff-AE backbone can already allow deterministic image reconstruction, limited gradients can be generated during error To further demonstrate the advantages of our method over traditional pipeline in terms of disentangled control, we train the whole model (Diff-AE+Exp-FaceNet) from scratch without using RSM training. The result in Fig. 6 (a) shows that our disentangled pipeline can significantly reduce identity shift, both visually and in terms of quantitative metrics.Exp-FaceNet structure. Our Exp-FaceNet is inspired from Con-trolNet [65] , a popular deep network which has been widely employed to add various spatial visual guidance (e.g., edge maps, pose maps, depth maps, etc.) to Stable Diffusion (SD) [45] for text-toimage generation. However, there exists many differences between two models. First, ControlNet is specially designed for SD which can generate specific content based on the input prompts, while there still exists uncertainty and randomness in the generation controlled by visual guidance and text prompts. In contrast to this, this paper focuses on semantics preservation and explicit editing of the input portrait image. Given this, our Exp-FaceNet is designed to be compatible with a Diff-AE backbone, aiming to provide semantically deterministic DDIM conditioning. Second, we construct Exp-FaceNet as a U-Net instead of adopting zero convolutions introduced in ControlNet. With this setting, we can endow Exp-FaceNet with strong capability of U-shape models in extracting spatial deep features from pixel-aligned visual guidance map (rendered explicit snapshots in this work). Here we compare the generation performance between using Exp-FaceNet and ControlNet to learn explicit face control. The results in Fig. 6 (b) show that compared to Con-trolNet, our Exp-FaceNet can help to edit the face image with less DECA re-inference error and visualized better identity consistency. To further analyze the disentangled conditioning of our DisControl-Face, we show the synthesis results of mixed conditioning generation in Figure 7 . Specifically, we replace the encoded global semantic code of facial image 𝐴 with global code extracted from another image 𝐵, while fixing all the estimated all explicit face parameters of 𝐴. We can observe that the synthesis face has the same face shape, pose, expression, and lighting condition with image 𝐴. Meanwhile, all facial appearance priors like skin color, eyes, and lips color as well as non-facial high-level semantics such as background and hair of image 𝐵 have been successfully transferred to the synthesized image. Note that we can achieve robust facial semantics transfer without requiring a few-shot fine-tuning on the personalized images of an specific individual as adopted by DiffusionRig [12] . This result can intuitively show the effectiveness of the disentangled face control capability of DisControlFace. Also benefiting from this, the proposed DisControlFace can preserve original high-level semantics while performing fine-grained explicit face editing.Semantic manipulation. Since we adopt Diff-AE [44] as the reconstruction backbone and freeze the pre-trained weights without tuning, our DisControlFace inherits the encoding capability of global facial semantics. Therefore, compared to previous diffusionbased model such as DiffusionRig [12] , our model also supports manipulating face attributes (e.g., hairstyle, age, and accessories) of the input portrait by linearly editing global semantic codes. The visualized results of both semantic manipulation and explicit editing are shown in Figure 8 , which once again demonstrate the effectiveness and flexibility of the proposed disentangled conditional generation mechanism.Image inpainting. Benefiting from the proposed RSM training strategy, our model inherently supports image inpainting. Figure 9 shows the results of zero-shot inpainting on center-masked facial images as well as the subsequent editing. We can observed that the restored face in the inpainted image is smooth and natural, also has good similarity with the original face. Besides, the explicitly edited image still shows a consistent identity with the restored facial image. Meanwhile, since it is inappropriate to use masked images to perform one-shot fine-tuning on Diff-AE, some editingirrelevant high-level semantics such as hairstyle and background might not be well preserved in the generated images. 
limitations and conclusion: 
 Limitations and future work. In this work, both adopted Diff-AE and constructed Exp-FaceNet were trained using the FFHQ dataset, which consisting 70,000 in-the-wild images. However, this data size is still not sufficient to train the model to learn a more generalized face priors. It can be expected that collecting much more face data with abundant face conditions (e.g., pose and expression) for training can substantially improve the editing performance, especially for some challenging tasks like zero-shot explicit editing. We adopt EMOCA to estimate explicit face parameters of the input image and generate spatial-aware conditions representing control information based on them. However, EMOCA struggles to model the detailed geometry of eyeballs as well as some extreme expressions, which could hinder the model from restoring the corresponding facial details when generating edited images. Furthermore, our editing framework is constructed as a denoising diffusion pipeline, which therefore is still unable to compete with GANs in terms of generation speed. In the future, with the development of fast sampling algorithms, we expect the generation time of our model to further decrease. 
 
 Appendix: We have presented DisControlFace, a novel diffusion framework for one-shot facial image editing. Through exploiting disentangled conditioning on high-level semantics and explicit 3DMM parameters in the generation process, our model excels in explicit and fine-grained face control while preserving semantic information and facial priors in face editing. This may boost a series of related applications including various semantic and explicit facial image editing, zero-shot image inpainting, and cross-identity face driving.  


Decision: This paper was reviewed by four experts in the field. The paper received mixed reviews BR, BA, WA, A. Reviewers like the generation of realistic facial images and the effective disentangling control achieved by the proposed framework. The only negative reviewer raises cocnerns on missing details and typos, which the AC believes did not challenge the contribution of this paper. Based on the scores, we recommend the acceptance of this paper.

Reviews: review: paper_summary: paper_summary: In this research, the authors propose a novel diffusion-based editing framework called DisControlFace, which aims to achieve fine-grained control of generative facial image editing. They address the challenge of disentangled conditional control between high-level semantics and explicit parameters by leveraging a Diffusion Autoencoder (Diff-AE) and an Exp-FaceNet. The model is trained using 2D portrait images and demonstrates superior editing accuracy and identity preservation compared to existing methods.
strength_weakness: strength_weakness: 1. The framework generates facial images that are visually realistic and faithful to the original input. The Diffusion Autoencoder (Diff-AE) and Exp-FaceNet components ensure that the generated images maintain the desired facial appearances and accurately represent the input semantics.
2. DisControlFace ensures consistent semantic details in the generated images. By leveraging the Diff-AE and Exp-FaceNet, the framework maintains the semantic conditioning capability and reduces semantic information shift during the editing process. This helps in preserving the identity and overall coherence of the generated images.
3. Comprehensive experiments demonstrate that DisControlFace outperforms state-of-the-art methods in terms of editing accuracy and identity preservation. The framework generates realistic facial images with a high level of accuracy, making it a reliable tool for generative facial image editing tasks. By the way, the demo video is awe-inspiring for me.
limitations: limitations: I am very curious about how this framework works in the case of a profile face. The authors are encouraged to discuss this case.
paper_summary: paper_summary: The paper introduces a diffusion-based generative framework for explicit fine-grained control of generative facial image editing. This framework utilizes a pre-trained Diff-AE model with an Exp-FaceNet to condition the diffusion process based on 3DMM parameters. The authors also introduce a Random Semantic Masking (RSM) training strategy to enable a disentangled explicit face control of the Exp-FaceNet. The proposed method achieves explicit facial image editing and also supports various one-shot face editing tasks. The comparison experiments show the effectiveness of the proposed method over existing methods.
strength_weakness: strength_weakness: 1. The random semantic masking training strategy effectively enables disentangled editing based on 3DMM parameters and preserves the identity and details of the original image.
2. The proposed method leverages the strengths of the Diff-AE backbone and ControlNet to synthesize photo-realistic facial images corresponding to the desired pose, expression, and lighting while only trained with 2D in-the-wild images.
limitations: limitations: 1. The proposed RSM training strategy has already been proposed in Masked Autoencoders. Despite the authors stating that the purpose is not for representation learning, it cannot be considered a novel contribution. And the Exp-FaceNet is essentially ControlNet using 3DMM parameters as the input conditioning. The contribution of the proposed framework is incremental.
2. The Exp-FaceNet needs to be pre-trained, so the proposed method is not one-shot.
3. The face reconstruction quality in the one-shot explicit editing shown in the supplementary video is unconvincing. The reconstruction results of DiffusionRig seem to preserve identity better and also have more details. For example, in the lighting editing result at 03:23, the result of DiffusionRig is more faithful.
paper_summary: paper_summary: This paper focuses on fine-grained control in facial image editing while maintaining realistic facial appearances and consistent semantic details. The key contributions include a diffusion-based editing framework, DisControlFace, leveraging a Diffusion Autoencoder (Diff-AE) as the backbone for semantic reconstruction and an Exp-FaceNet for spatial-wise explicit control conditions based on 3DMM parameters. Experimental results show that the proposed method outperforms existing methods in controllable facial image editing.
strength_weakness: strength_weakness: 1. **Nice figures**: The pipeline overview and experimental results are very polished.
2. **Rich paper review**: The related work review is rich and diverse.
limitations: limitations: 1. **Missing details and mistakes**: 

   - > To address this issue, we design a random semantic masking (RSM) strategy, not for the purpose of representation learning like Masked Autoencoders (MAE) [20], but rather to achieve the training of Exp-FaceNet effectively. 

     Sections 4.2 and 4.3 are unclear and lack sufficient detail. What are the main differences between RSM and MAE?

   - [Figure 3] "DECA"->"EMOCA"
   - [Line 677] Which is the "top part"? Any other bottom part?

2. **Typos**: There are several grammar mistakes and typos.

   - [Line 121, 137,140] "generations"->"generation"
   - [Line 168] "conditoning"->"conditioning"
   - [Line 173, 202] "parameteric"->"parametric"
   - [Line 182] "reconstrcution"->"reconstruction"
   - [Line 243] "though"->"through"
   - [Line 309] "dimentional"->"dimensional"
   - [Line 347] "firs"->"first"
paper_summary: paper_summary: This paper proposes DisControlFace which supports fine-grained control of generative facial image editing under a one-shot scenario.
DisControlFace is built on the top of a Diff-AE. Based on it, this paper introduces Exp-FaceNet with a random semantic masking training strategy to learn to disentangle face control.
strength_weakness: strength_weakness: - This paper is well-written and is easy to follow. 
- This paper suggests freezing Diff-AE and implementing a random semantic masking training strategy to enable training different DDIM conditioning together. The effectiveness of this approach is verified through ablation studies, highlighting its potential to inspire future research.
- DisControlFace achieves sota qualitative and quantitative results for one-shot explicit facial image editing.
limitations: limitations: - The improvement in lighting change in Fig. 3 is remarkable when compared with DiffusionRig. Some instructive discussion of this is missing, e.g., does it benefit from the RSM training strategy and how? On the contrary, the improvement is not obvious in pose and expression. It is recommended to highlight the improvment and add more comparisons to support the claim. 

- I appreciate the visualization of disentangled face control in Fig.7. However, It seems that the ID is not controlled by global semantics and the ID information will be restored by the one-shot fine-tuning. An ablation study of disentangled face control w.o one-shot fine-tuning is recommended for a better understanding of the ID control in DisControlFace.

Typos:
Line 750. PSM -> RSM;
Line 835. all the estimated all explicit … -> all the estimated explicit