InstructPix2NeRF: Instructed 3D Portrait Editing from a Single Image
Authors: Jianhui Li
Shilong Liu
Zidong Liu
Yikai Wang
Kaiwen Zheng
Jinghui Xu
Jianmin Li
Jun Zhu
Venue: ICLR 2024 poster (2024)
Keywords: 3D-aware editing
Human instruction
Conditional latent 3D diffusion
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We propose an end-to-end diffusion-based pipeline termed InstructPix2NeRF, which enables instructed 3D-aware face editing from a single real image with user instructions.

Abstract: With the success of Neural Radiance Field (NeRF) in 3D-aware portrait editing, a variety of works have achieved promising results regarding both quality and 3D consistency. However, these methods heavily rely on per-prompt optimization when handling natural language as editing instructions. Due to the lack of labeled human face 3D datasets and effective architectures, the area of human-instructed 3D-aware editing for open-world portraits in an end-to-end manner remains under-explored. To solve this problem, we propose an end-to-end diffusion-based framework termed $\textbf{InstructPix2NeRF}$, which enables instructed 3D-aware portrait editing from a single open-world image with human instructions.  At its core lies a conditional latent 3D diffusion process that lifts 2D editing to 3D space by learning the correlation between the paired images' difference and the instructions via triplet data. With the help of our proposed token position randomization strategy, we could even achieve multi-semantic editing through one single pass with the portrait identity well-preserved. Besides, we further propose an identity consistency module that directly modulates the extracted identity signals into our diffusion process, which increases the multi-view 3D identity consistency. Extensive experiments verify the effectiveness of our method and show its superiority against strong baselines quantitatively and qualitatively. Source code and pretrained models can be found on our project page: https://mybabyyh.github.io/InstructPix2NeRF.

Published as a conference paper at ICLR 2024
INSTRUCTPIX2NERR: INSTRUCTED 3D PORTRAIT EDITING FROM A SINGLE IMAGE
Jianhui Li', Shilong Liu', Zidong Liu', Yikai Wang', Kaiwen Jinghui Xu² Jianmin Li"; ; Jun !h
'Dept. of Comp. Sci. & Tech., Institute for AI, BNRist Center, THBI Lab, Tsinghua-Boo Joint ML Center, Tsinghua University, Beijing, 100084 China
?Shengshu Technology, Beijing
(lijianhu214 liusl2lll 120, 1iu-zdd00 ( @mmals tsinghua eeighuaaedd...o edu cn lokkweeottlok... com, z zkwthu& & com, jinghui. xuêshengshu. ai {lijianming ( d edu.
ABSTRACT
With the success of Neural Radiance Field (NeRF) in 3D-aware portrait edit- ing, a variety of works have achieved promising results regarding both quality and 3D consistency. However, these methods heavily rely on per-prompt op- timization when handling natural language editing instructions. Due the lack of labeled human face 3D datasets and effective architectures, the area of -ustnninstructed 3D-aware editing for open-world portraits an end-to-end manner remains under-explored. To solve this problem, we propose end-to-end diffusion-based framework termed InstructPix2NeRF, which enables instructed 3D-aware portrait editing from single open-world image with human instruc- tions. At its core lies a conditional latent 3D diffusion process that lifts 2D edit- ing 3D space by learning the correlation between the paired images' differ- ence and the instructions via triplet data. With the help of our proposed token position randomization strategy, could even achieve multi-semantic editing through one single pass with the portrait identity well-preserved. Besides, we further propose an identity consistency module that directly modulates the ex- tracted identity signals into our diffusion process, which increases the multi-view 3D identity consistency. Extensive experiments verify the effectiveness of our method and show its superiority against strong baselines quantitatively and qual- itatively. Source code and pretrained models can be found on our project page: https: htttp::/mmybayy/ //tpbbbyybbb...mm/m/ git.ub. thub. io/ io/Inne Inst x22NRF
INTRODUCTION
While existing 3D portrait editing methods (Cai et al. 2022 Lin et al. 2022 Sun et al. 2022b Li
al. 2023 Xie al. 2023 Lan et al. 2023) explored the latent space manipulation of 3D GAN models and made significntt progress, they only support preset attribute editing and cannot handle natural language. The explosion language models has made possible to enjoy the freedom and friendliness of the natural language interface. Recently, many excellent text-supported image editing methods have emerged, such as Falk-To-Edis (Jiang et al. 2021). StyleCLIP (Patashnik et al.) 2021), TediGAN (Xia et al. 22221), AnyFace (Sun et al.) 2022a) and InstructPix2Pix (Brooks al, 2023 y. However, these methods are typically limited to the 2D domain and cannot directly produce 3D results. While it is possible to connect 2D text-supported editing model
3D inversion model for text-supported 3D-aware editing, this will result in extra loss of identity information as the original face is invisible in the second stage. Thus, an end-to-end model is more desirable for better efficiency and performance.
Rodin (Wang et al., 2023) and ClipFace (Aneja et al. 2023 explored end-to-end text-guided 3D- aware face editing. Rodin trains a conditional diffusion model in the roll-out tri-plane feature space
"Corresooniigg authors.

---

Published as a conference paper at ICLR 2024
beard
iair
Make her look like a cartoon
Put eyeglasses on bet and turn the portrait into a bronue statue
Figure I: Our instructed 3D-aware portrait editing model allows users to perform interactive global and local editing with human instructions. This can be a single attribute editing or style editing instruction, multiple attribute editing instruction together, or even attribute and style instructions together.
with 100K 3D avatars generated by synthetic engine. Rodin achieves text-guided 3D-aware ma- nipulation by conditioning the diffusion process with the manipulated embedding, which consists of the CLIP (Radford et al. 2021) image embedding and a direction in the CLIP text embedding. However, since Rodin can only generate avatar faces, cannot handle real-world face editing. Clip- Face learns texture mapper and expression mapper with CL.IP loss and uses the generated UV map and deformed 3D mesh to achieve text-guided manipulation the synthetic domain. However, ClipFace cannot handle real-world face editing and is consumin and inconvenient because separate encoder needs to be trained for each text prompt.
better for expressing editing intent than descriptive prompts. However, focusing on noun phrases, existing text-guided 3D-aware face editing methods are challenging to understand verbs in instructions. For example, descriptive-pei cessriptive-promp--rivi prompt-driven editing methods usually treat. "remove the eyeglasses" as putting on the eyeglasses, as shown in Appendix A.2. Therefore, end-to-end 3D-aware portrait editing from an input image with human instructions fascinating task, which aims to achieve conssisstt editing with user- beer-frienlly interface. To our knowledge, we are the first to explore this area. We analyze that is critical to design an efficient framework incorporating human instructions with 3D-aware portrait editing. In addition, due to the lack of multi-view supervision, it challenging to maintain the consistency of identity and editing editing,
Human instructions
critical and
effects when performing 3D-an aware editing from single image, especially
In this work, we address these issues by proposing a novel framework termed InstructPix2NeRE, which enables precise 3D>aaare portrait editing from single image guided by human instructions. We prepare a triplet dataset each sample consisting of a 2D original face, a 2D edited face, and human instruction semantically representing single change from the original face the edited face. Training on large number of images and single human instructions, our model enables instructed 3D-aware editing with many single instructions even multiple instructions, rather than training the model for each human instruction. Although trained on a instruititttttiionn dataset, In- struetPix2NeRF enables inltiiiinstruction editing during inference. InstructPix2NeRF can perform instructed 3D-aware face editing thanks to the three key ingredients below.
Firstly, we design a novel end-to-end framework combining diffusion and NeRF-hased generators. The state-of saate-of-th--ale the-art architecture for text- gxided image editing is typically based on diffusion models (Rombach et al. 2022; Zhang Agrawala 2023 Brooks et al. 2023). At the same time, 3D portrait modeling commonly relies on NeRF-based GANs (Chan al. 2022 Sun et al., 2022b Niemeyer & Geiger 2021: Chan et al. 2021). We efficiently combine these two aspects to design an effective cross modal editing model. Specifically, we the inversion encoder PREIM3D (Li et al., 2023) to obtain the w++ latent codes of the 2D original and edited faces, then train a transformer- based diffusion model in the 3D latent space. The CLIP's text embedding of instructions is injected into
2

---

Published as a conference paper at ICLR 2024
the diffusion process via cross- cross-attention block after the self-attention block. Latent codes with biconditional sampling of text and images are decoded to geometry and multiple views with a single pass through the WeRF-based generator.
Secondly, we propose a Token Position Randomization (TPR) training strategy to handle multiple editing requirements via single pass. By TPR, instruction tokens are put in random position of the token sequence, enabling the model to fulfill multiple editing requirements simultaneously while preserving the facial identity large extent.
Thirdly, we propose an identity consistency module that consists of an identity modulation part and identity regularization part. We replace layer-normalization layers in the transformer block with adaptive layer norm (adaLN), which modulates the identity signal. The identity regularization loss calculated between the original face image and typical face image generated by a one-step prediction of latent code when the diffusion timestep is less than the threshold.
Combining the three key components together, InstructPix2NRRF enables instructed and 3D con- sistent portrait editing from single image. With one 15-step DDIM(Song et al.) 2022] Lu et al.] 2022) sampling, our model can output a portrait and geometry with attributes or style guided by instructions in a few seconds. Figure [ shows the editing results produced by our method. We recommend watching our video containing a live interactive instruction editing demonstration. To facilitate progress in the field, we will be completely open-soureing the model, training code, and the data we have curated. We expect our method to become strong baseline for future works towards instructed 3D-aware face editing.
RELATED WORK
NeRF- 3D generation and manipulation. Neural Radiance Field (NeRF) (Mildenhall al. 202T) has significantly impacted 3D modeling. Early NeRF models struggle to generate diverse scenes since the models are trained on a lot of pose images for every scene. Recent NeRF methods, such as GIRAFFE(Niemeyy
& Geiger 202I). EG3D(Chan et al. al., 22022). and IDE-3D(Sun ct al. 2022b), integrate NeRF into the GAN framework to generate class-specific diverse scenes. These works have paved the way for 3D-aware objects editing methods like IDE-3D (Sun al. 20226) PREIM3D (Li et al. 2023), HFGI3D (Xie et al.) 2023). and E3DGE (Lan et al. 2023) that perform semantic manipulation in the latent space of NeRe-based GANs in specific domain, such as human faces, cars, and cats. Despite the promising results of these methods, they cannot handle natural language.
Diffusion models. The diffusion probabilistic models (DPMs) (Sohl-Dickstein et al. 2015 Song
& Ermon 2019 Ho et al., 2020) can generate high-quality data from Gaussian noise through a for- ward noise addition process and a learnable reverse denoising process. By training noise predictor
using UNet (Ronneberger et al. 2015; Dhariwal & Nichol 2021 Song & Ermon 2019 Bao et al.)
2022b) or transformers (Vaswani et al. 2017 Bao et al. 2022a Peebles & Xie 2022 backbone,
the diffusion model can stably learn the probability distribution on ultra-large datasets and hold ex-
tensive applications such image generation (Rombach al. 2022), multi-model data generation
[aoo et al. (2023), image editing (Zhang & Agrawala| 2023) Brooks et al. (003), and likelihood es- timation(Zheee al. 2023). Initially, the diffusion model had difficulty generating high-resolution images in pixel space. Recently, the Latent Diffusion (Rombach et al. 2022), which combines the
Table I: An overview of 3D-aware portrait editing methods.
Per-image Optimization-free
Real-world Images
Per-text Optimization
Method
Text-supported
Instructed
IDE-3D
E3DGE
X
X AAXAA
PREIM3D
V
X
ClipFace
X
X
IDE3D-NADA
X
X
Rodin
Ours
V
3

---

Published as a conference paper at ICLR 2024
VAE and the diffusion model, significantly improves generation efficiency and image resolution. In- spired by latent diffusion (Rombach et al., 2022), ). we combine the diffusion model with NeRF-base GAN to efficiently implement latent 3D diffusion.
Text-guided editing. The previous approach for text-guided editing (Jiag et al, 2021 usually involves training a text encoder map text input human instructions to a linear non-linear editing space, which may not handle out-of-domain text and instructions well. Recently, The pre- trained CLIP (Radford et al., 2021) model contains vast vision and language prior knowledge, sig- nificantly accelerating the development of vision-language tasks. StyleCLIP (Patashnik et al., 202T) calculates the normalized difference between CLIP text embeddings of the target attribute and the neutral class as the target direction At, which is then applied to fit a style space manipulation direc- tion As. InstructPix2Pix combines the abilities of GPT (Brown et al.) 2020) and Stable Diffusion (Rombach et al. 2022) to generate multi-modal dataset and fine-tunes Stable Diffusion to achieve instructed diverse editing. However, these methods are focused on editing 2D images and do not enable 3D-aware editing. Recently, Rodin (Wang et al. 2023), ClipFace (Aneja et al. 2023), and IDE3D-NADA (Sun et al.] 22022b Gal et al. (022) explored text-guided 3D-aware editing. How- ever, Rodin and ClipFace only be applied to the synthesis face, and ClipFace and IDE3D-NADA require optimization for each text prompt. Meanwhile, Instruct-NeRF2NG NeRF2NeRF (Haque et al. 2023),
AvatarStudio (Mendiratta al., 2023), and HeadSculpt (Han et al.) 2023) have achieved success in optimizing text-driven single-3D-scen than single image, and they take tens of minutes for each scene.
editing. However, these methods require a 3D scene rather
As shown in Table | IDE-3D, E3DGE, and PREIM3D cannot handle natural language, ClipFace and IDE3D-NADA rely on per-prompt optimization, Rodin and ClipFace cannot be applied real- world faces.
3 DATA PREPARATION
Inspired by InstructPix2Pix (Brooks et al., 2023). we prepared a multimodal instruction instruction-followin following triplet dataset, where triplet data consists of original face, edited face, and human instruction representing a single change from the original to the edited. Specifically, given a face, we use pre- trained 2D editing models produce 2D edited face and use the large language model ChatGPT Brown et al. 2020) to generate the corresponding editing instructions.
Paired image generation. We leverage two off-the-shelf 2D image editing methods, e4e Tov et al. 2021) and InstructPix2Pix (Brooks et al. 2023), to generate paired images on FFHQ (Karras al. 2.19p. We use e4e, the widely used of-te-of-the-ar face attribute editing method, to generate 22K pairs of images with 44 attributes editing. As for InstructPix2Pix, we have selected 150 portrait- related instructions from its instruction set using keywords. The faces edited with these instructions were filtered by a face identification model (Deng et al. 2021) to obtain 18K paired images.
Instruction generation. In-context learning in large language models (LMs) demonstrates the power that rivals some supervised learning. For the paired images generated by e4e, we provide ChatGPT with some example data consisting of the paired attribute labels and corresponding hand- written transfer instructions. We tell it that the instructions represent single change between at- tributes. After that, we asked ChatGPT to generate 20-30 transfer instructions with the provided attributes of the pairs. For the paired image generated by InstructPix2Pix. since there are already human instructions, we asked ChatGPT to generate 20-30 instructions representing the same seman- tic meaning on this basis.
Assigning human instructions to 40K paired images, we obtained a final triplet dataset containing 640K examples, each consisting of single semantic instruction and paired image. See Appendix for more detailed data preparation.
4 METHOD
Given input image and human instructions, our goal is to generate multi-view images and geom- etry, which behave intended by human instructions and can keep other attributes and identities unchanged, With the NeRF- generator and inversion encoder, our goal can be translated into generating latent code that represents the edited face image. The latent code will be given the
4

---

Published as a conference paper at ICLR 2024
Input:
Add noise
W (512
Have painting by Edward Hopper
Inpot: 2D edited face
We (512144)
noise
Cross-atteation
Wot (5124444)
Concarenate
Lyp
Lest
Input: original face
Wo(5125e44)
adal
Diffusion Trausformer
(13125444)
Output: 3D edited face
Figure 2: An overview of our conditional latent 3D diffusion model. We use NeRF-based gener- ator inversion encoder E to obtain the latent code of the image. Then, we train a diffusion model conditioned on human instructions and the original face. Text instruction conditioning is introduced using the cross-attention mechanism with the CLIP text embedding, and the original image condi- tioning realized via concatenating and adaptive layer norm. F is face identification model. G is NeRF-based generator. The Diffusion Transformer is trainable, the other models are fixed.
NeRF-based generator to produce multi-view images conditioned camera pose. Figure illus- trates the whole architecture of our method. We will introduce the key components of our pipeline in the following subsections.
4.1 CONDITIONAL. LATENT 3D DIFFUSTON
Our method is based on the NeRF-basee generator G and inversion encoder E. The NeRF-based generator, such as EG3D (Chan et al. al.| 2022), can generate multi-viww images from Gaussian noise E 2 C R252 conditioned on camera parameters The noise is mapped to the intermediate latent code w f(z) E W C k= which used produce tri-plane features through the convolu- tional network. A small MLP decoder is used to interpret the features retrieved from 3D positions as color and density, rendered into multi-view images conditioned on camera pose c, described as X G(w.e).
The NeRe-based inversion encoder E (Li et al. 2023 Lan al. 2023) learns the features of large number of images with different poses to reconstruct 3D representation from single image. The encoder maps input image X the latent code w, which can be used produce novel view X of the input image:
E(X),
(I)
x' = G(E(X).c)
where c is the camera pose.
Performing the diffusion process in the latent space combines the strength other generators and accelerates training, especially in the 3D generation spaces, such as 2, W. and W+. The W Wsacee consists of l: repetitive 512-dimensional latent vectors fed into convolutional layers different resolutions, but the W+ space consists of k distinct latent vectors. It's demonstrated that k distinct latent vectors can increase the representation capacity of the generator than k identical latent vectors (Shen al. 2020 Li al. 2023). Although EG3D is trained on the real-world face dataset FFHQ, our simple experiments in Appendix demonstrtte that it is possible to generate some out-of-domain 3D faces, such as bronze statues, and cartoons, using fixed EG3D fed W+ latent code. Moreover, W/+ space is considered more disentangled than 2 and W spaces (Tov al.] 202T Patashnik et al. 2021). Therefore, we choose W4 space to perform the diffusion process
Here, there several candidate latent
work.
obtain the latent code w guided by human instructions, we employ diffusion model to learn the correlation between the paired images and the instructions. Notably, transformers (Vaswani et al., (017) show a promising ability to capture complex interac- tions and dependencies between various modalities (Bao al. 2023 In this paper, diffusion backbone is Diffusion Transformer (DiT) (Peebles & Xie 2022). We have made the following modifications DiT: (i) add input header that enables paired latent codes diffusion, (ii) add
Diffusion model architecture.
5

---

Published as a conference paper at ICLR 2024
cross-attention block after the self- block and introduce CLIP text embedding here, and (iii) add an identity embedding module that is plugged into the norm layer of transformer block using adaptive layer norm described in section 4.3
Given an input image X, and a human instruction T, our goal can be formulated as learning the conditional distribution p(w] X, T). The inversion encoder E is applied to obtain the latent code wo = =(x)) E IR 14+512 we Exx,) E IR 14+512 for the original image X, and corresponding 2D edited image X. respectively. We add noise the latent code we with fixed schedule, producing a noisy version Wes at step 1,7 E T. Our model is trained as a noise predictor eg to predict the added noise conditioned on image and text. The image conditioning сд consists of two parts: concatenation of wi and We, and identity modulation. The text conditioning or is realized by adding a multi-head cross-attention block with the CLIP text embedding following the multi-head self-attention block. To adapt to the model, the latent code would be reshaped. Take 512*512 resolution for example, after padding zero vectors for w E R144512 we reshape the latent code to the shape 512 4 4. The conditional latent diffusion objective is:
x
Laif
- 4, cr, (T) (())
where oj is the image conditioning, er is the text conditioning.
4.2 TOKEN POSITION RANDOMIZATION
In our preliminary experiments, we have observed that when editing with multiple instructions, the more forward prraard-ositioned instructions are easier to show up in the edited image. We analyze this issue and attribute it to the training data being single instruction edited. In natural language processing, text conditioning requires that text be tokenized into a sequence of tokens, which is 77 in length in this paper. Only the first few sequence positions usually non-zero when train with single instruction data. It is intuitive to think that the cross-attention mechanism might pay more attention to the head of multiple instructions.
To achieve better editing results for the multiple instructions, we propose token position randomiza- tion, randomly setting the starting position of the text instruction tokens. This strategy makes the model more balanced in its attention to the components of multiple instructions. As a result, multi- semantic editing can be performed a way that fulfills all the editing requirements while preserving identity well. In our paper, we randomly set the starting position in (0, with the last non-zero token position being less than 77. Ablation studies Figure 5 and Table show the effectiveness of the token position randomization strategy.
4.3 IDENTITY CONSISTENCY MODULE
For precise face editing, preserving the input subject's identity is challenging, especially in 3D space. Latent code can be considered compression of an image that loses some information, including identity. To tackle this, We impose an identity compensation module that directly modulates the ex- tracted identity information into the diffusion process. A two-layer MIP network maps the identity features extracted from the original face into the same dimension as the diffusion timestep embed- ding. We regress dimensionwise scale and shift parameters 7 and from the sum of the embeddings of diffusion timestep f, and the embeddings of identity feature extracted from the portrait using a face identification model (Deng et al. 2021).
To improve 3D identity consistency further, we explicitly encourage face identity consistency at different poses by adding identity regularization loss between the 2D edited image and the rendered image with yow = 0, and pitch- = 0.
LID=1 = (F(X.), F(G(W.o, co))),
where F' is the face identification model which extracts the feature of the face, Xe is the 2D edited face, Weo is the one-step prediction of latent code, co is the camera pose with yaw=0, pitch=0.
When the diffusion timestep is large, it will lead poor w for one-step prediction. Thus, this loss is calculated only for samples with timestep less than the timestep threshold tzh. The total loss is:
L = Laifs + XiaLID,
where Xea is the weight of LTD- Aia is set 0.1 in our experiments.

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative evaluation for editing with single instruction on multi-view faces. ID score denotes the identity consistency between before and after editing. CLIP score measures how well the editing result matches the editing instructions. Attribute altering (AA) measures the change of the desired attribute. Attribute dependency (AD) measures the change in other attributes.
Attribute
Instruction example
Method
IDM
CLIPt
AAt
AD
Talk- Talk-To-Edit
0.41
0.07
0.97
0.56
InstructPix2Pix
0.40
0.07
0.80
0.64
Bangs
Let's add some bangs
img2img
0.40
0.10
0.99
0.61
Ours
0.56
0.13
1.05
0.53
To-k-To-Edit**
0.46
0.04
0.69
0.67
Make the person wearing glasses
InstructPix2Pix
0.51
0.17
3.27
0.65
Eyeglasses
img2img
0.42
0.17
3.33
0.79
Ours
0.59
0.20
3.37
0.64
Talk-To To-Eit*=d
0.43
0.10
0.54
0.62
The person should smile more happily
InstruetPix2Pix
0.48
0.16
1.46
0.66
Smile
img2img
0.46
0.15
1.47
0.76
Ours
0.60
0.18
1.50
0.61
*; Talk-To- Ed-Edit don't recognize some instructions. Only the results of recognized instructions are considered.
4.4 IMAGE AND TEXT CONDITIONING
Classifier-free guidance trains an unconditional denoising diffusion model together with the condi- tional model instead of training separate classifier (Ho & Salimans 202I). Liu al. (2022) show that composable conditional diffusion models can generate images containing all concepts condi-
tioned on set of concepts by composing score estimates. Following (Ho Salimans) 2021 Liu
et al. 2022 Brooks et al., 2023), we train single network to parameterize the image-text condi- tional, the only-image conditional, and the unconditional model. We train the unconditional model simply by setting P2 for the only-image-conndtionall model Wa cz, 0). In our paper, we set P1 0.05, P2 = 0.05 hyperparameters. In the inference phase, add a little bit of noise to the latent of the input image (usually 15 steps) to obtain Wot and then use our model to perform conditional denoising. The model predicts three score estimates, the image-text conditional eø (Cat; Cr, Cr,CT), the only-image conditional eo (wa,, Cr, 0), and the unconditional ep(was. 0, 0). eT = indicates that the text takes an empty character. с₁ 0 means that the concatenation w, takes zero and identity modulation takes zero. Image and text conditioning sampling can be performed as follows:
0, cT 0 with probability Pl. similarly, only setting er 9 with probability
Eg(Wot: CT) =c (wo, 0,0)
$1 (co (Wot CJ. 0) - cg (Wot: 0, 0))
(5)
sr C1, cr) co (Wat, C1, 0))
where 8y and 8T are the guidance scales for alignment with the image and the text, respectively.
EXPERIMENTS
Given that the field of instructed 3D-aware face editing is under-explored, we have designed and developed series of models that serve our baselines. We compare our method with three base- lines: Talk-To-BE Edit (Jiang et al.) (02T) combined with PREIM3D (Li et al. (023), InstructPix2Pix (Brooks et al. (223) combined with PREIM3D, and method similar to the img2img mode of Stable
Diffusion Rombach et al. 2022). More implementation details are provided in the Appendix A.
We use the official pre-trained models and code for Talk-To-Edit InstructPix2Pixx and PREIM3D in the comparison experiments. The metrics are calculated on the first 300 images from CelebA-HO. To evaluate the 3D capability, we uniformly rendered 4 views from yaw angles between [-30", 30 and pitch angles between 1-20%, ,20) for an input image. We evaluate our method using ID, CLIP, AA, AD, Ma and Sa, defined in the Appendix A.1.3

---

Published as a conference paper at ICLR 2024
Have her smiling. make the person wearing eyeglasses
make her
Add bangs to the hairstyle
smillng mouth, put eyeglusses face, add some bongs.
Input
Talk-To-Edit
InstructPixelPPxell
img2img
Figure 3:
Qualitative comparison. Our method achieves the requirements of the text instruction while preserving identity consistency, especially multiple instruction editing.
Table 3: Quantitative evaluation for editing with multiple instructions multi-view faces. Ma and Sa measure 3D consistency
Method
IDt
CLIP
AAavg
AD
AAmin t
Ma +
S +
Talk-To- *o-Edit*
0.44
0.05
0.24
0.58
-0.12
0.125
0.042
InstruetPix2Pix
0.46
0.19
1.48
0.71
0.31
0.109
0.039
img2img
0.37
0.17
1.39
0.88
0.12
0.119
0,039
Ours(w/o TPR)
0.50
0.18
1.50
0.73
0.08
0.114
0.038
Ours
0.55
0.20
1.53
0.69
0.52
0.105
0.038
Same as Table 2
Qualitative evaluation. We present examples of the instructed editing results in Figure Due to using a word bank, Talk-To-Edit does not recognize part of the instructions and cannot handle multiple instructions. Img2img does disentangle the attributes and struggles in aligning text editing requirements with images, leading to changes in some other attributes. For example, the fourth in Figure the additional semantics of becoming younger. InstructPix2Pix usually leads to variations in hue and identity consistency, such as the skin tones in the third row of Figure Our method achieves better text instruction correspondence, disentanglement, and 3D consistency than baselines. More editing results are provided in Appendix A.3
Quantitative evaluation. We chose three typical facial attributes, bangs, eyeglasses, and smile, to evaluate our method and baseline quantitatively. To be fair, our instruction test set consists of selected instructions from Talk-To-Edii, InstructPix2Pix, and InstructPix2NeRRF where each model contributes 15 instructions, such as 'Make her happy instead." and 'The eyeglasses could be more obvious.' The metrics on the multiple instructions are measured with six combinations of the above three attributes in different orders. As shown in Table 2 3 method performs better than the baselines. Since img2img doesn't 't disentangle the editing requirements and is prone to cause changes other attributes, it has low ID score and high AA score. Talk-To-Fdii sometimes responds
8

---

Published as a conference paper at ICLR 2024
only marginally editing instructions, its CLIP and AA scores significantly than other methods. InstructPix2Pix scores well, but still below our method. See Appendix [A.2.3] for more evaluation of attribute editing. We conducted user study and as shown in Table 6 method outperforms the baselines.
Table 4: Effects of identity modulation and reg- ularization loss. w/o Lin and w/o ID cond rep- resent the model without regularization loss and identity modulation, respectively.
ausiECUP TPRO]
tursjaa tee]]
22
2.0
Config
Daong
Doyotoosees
Dmstte
Dastti
Ciu w/o ID cond
0.47
0.55
0.44
0.54
0.57
Potto
1.6
Ours
0,56
0.59
0.60
0.55
12
Ph oyeeloe hirz
count iradructions
Thre hor Plice sysilaso un het inttuction
Figure 5: The improvement rate of AA and CLIP score for our model with different number of editing instructions against the model without to- ken position randomization training strategy.
ID
Figure 4: Visual improvements of identity mod- ulation and regularization loss.
Ablation of token position randomization To verify the effectiveness of token position random- ization training, two models were trained, one using the token position randomization training strat- egy and the other not. Table 3 and Figure X show show that the model with TPR performs better than the other model when editing with multiple instructions.
To further compare the performance of the two models for multiple instructions different lengths, we performed multiple instructions editing of lengths 1-4, involving bangs, eyeglasses, smile, age. The average improvement AA and CLIP scores measures the effect. As shown Figure [5] the two models give comparable AA and CLIP scores when using single instruction editing. Still, the instruction length increases, our model shows a better correspondence for text instruction.
Ablation of identity consistency module Injecting learned identity information into the diffusion process, the identity modulation compensates for losing information in latent space. The identity regularization loss explicitly guides the model to preserve the identity of the input subject. As shown Table and Figure 5 the model with the identity consistency module significantly improves the identity consistency scores and visuals.
6 CONCLUSIONS
In this paper, to solve the instructed 3D-aware face editing, we propose novel conditional latent 3D diffusion model, enabling instructed 3D-aware precise portrait editing interactively. To support the editing of multiple instructions editing that was not available in previous methods, we propose the token position randomization training strategy. Besides, we propose an identity consistency module consisting of identity modulation and identity loss to improve 3D identity consistency. We expect method to become a strong baseline for future works towards instructed 3D-award face editing. Our method can used for interactive 3D applications such as virtual reality and metaverse.
Limitations. One limitation of our work is that some semantically identical instructions may pro- duce slight errors. For example, although "turn the hair color to pink" and "change her hair into pink" both want to change the hair color to pink, they will obtain different pink. Moreover, the same instruction will also have color differences for different people. We have some losses for details such eye shape and eyelashes. These issues that need to be addressed in this area in the future.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
This work was supported by NSFC Projects (Nos. 62350080, 62076147, U19A2081), BNRist (BNR2022RC01066) Tsinghua Institute for Guo Qiang, and the High Performance Computing Center, Tsinghua University. J.Z. was also supported by the New Comeertonne Science Foundation through the XPLORER PRIZE.
REFERENCES
Rameen Abdal, Hsin-Ying Ying Lee, Peihao Zhu, Menglei Chai, Aliaksandr Siarohin, Peter Wonka, and Sergey Tulyakov. 3davatargan: Bridging domains for personalized editable avatars. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR, pp. 4552-4562. 2023.
Shivangi Aneja, Justus Thies, Angela Dai, and Matthias Nießner. Clipface: Text-guided editing of textured 3d morphable models. In ACM SIGGRAPH 2023 Conference Proceedings, SIGGRAPH,, pp. 70:1-70:11 2023.
Fan Bao, Chongxuan Li, Yue Cao, and Jun Zhu. All are worth words: vit backbone for score-based diffusion models. CoRR, abs/2209. 12152, 2022a.
Fan Bao, Chongxuan Li, Jun Zhu, and Bo Zhang. Analytic-dpm: an analytic estimate of the optimal reverse variance in diffusion probabilistic models. In The Tenth International Conference on Learning Representations, ICLR, 2022b.
Fan Bao, Shen Nie, Kaiwen Xue, Chongxuan Li, Shi Pu, Yaole Wang, Gang Yue, Yue Cao, Hang Su, and Jun Zhu. One transformer fits all distributions in multi-modal diffusion scale. In International Conference on Machine Learning, ICML, volume 202, pp. 1692-1717, 2023.
Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18392- 18402, 2023.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901,200
Shengqu Cai, Anton Obukhov. Dengxin Dai, and Lue Van Gool. Pix2nerf: Unsupervised condi- tional pi-gan for single image to neural radiance fields translation. IEEF/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp. 3971-3980, 2022.
Eric R. Chan, Marco Monteiro, Petr Kellnhofer, Jiajun Wu, and Gordon Wetzstein. Pi-gan: Periodic implicit generative adversarial networks for 3d-aware image synthesis. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 5799-5809, 2021.
Eric R. Chan, Connor Z. Lin, Matthew A. Chan, Koki Nagano, Boxiao Pan, Shalini De Mello, Orazio Gallo, Leonidas Guibas, Jonathan Tremblay, Sameh Khamis, Tero Karras, and Gordon Wetzstein. Efficient geometry-aware 3d generative adversarial networks. In IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, CVPR, pp. 16102-16112, 2022.
Jiankang Deng, Jia Guo, Jing Yang, Niannan Xue, Irene Cotsia, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. IEEE Transactions Pattern Analysis and Machine Intelligence, 2021.
Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in neural information processing systems, 34:8780-8777 2021.
Rinon Gal, Or Patashnik, Haggai Maron, Amit H. Bermano, Gal Chechik, and Daniel Cohen-Or. Stylegan- nada: Clip-guided domain adaptation image generators. ACM Trans. Graph., 41(4): 141:1-144:13, 2022.
Xiao Han, Yukang Cao, Kai Han, Xiatian Zhu, Jiankang Deng, Yi-Zhe Song, Tao Xiang, and Kwan- Yee K. Wong. Headsculpt: Crafting 3d head avatars with text. arXiv preprint AXi:2230.00303. 2023.
10

---

Published as a conference paper at ICLR 2024
Ayaan Haque, Matthew Tancik, Alexei Efros, Aleksander Holynski, and Angjoo Kanazawa. Instruct-nerf2neff. Editing 3d scenes with instructions. In Proceedings of the IEEE/CVF In- ternational Conference on Computer Vision, 2023.
Jonathan Ho and Tim Salimans. Classifier-free diffusion guidance. In NewrlPS 2021 Workshop on Deep Generative Models and Downstream Applications, 2021.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Xun Huang and Serge J. Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In IEEE International Conference on Computer Vision, ICCV, pp. 1510-1519, 2017.
Yuming Jiang, Ziqi Huang, Xingang Pan, Chen Change Loy, and Ziwei Liu. Talk-to-edit: Fine- grained facial editing via dialog. In IEEF/CVF International Conference on Computer Vision, ICCV, pp. 13779-13788, 2021.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. In International Conference on Learning Representations, ICLR, 2018.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings the IEEE/CVF conference on computer vision and pattern recognition, pp. 4401-4410, 2019.
Yushi Lan, Xuyi Meng, Shuai Yang. Chen Change Loy, and Bo Dai. Self-supervised geometry- aware encoder for style-based 3d gan inversion. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 20940-20949, 2023.
Jianhui Li, Jianmin Li, Haoji Zhang, Shilong Liu, Zhengyi Wang, Zihao Xiao, Kaiwen Zheng, and Jun Zhu. Preim3d: 3d consistent precise image attribute editing from a single image. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8549 8558, 2023.
Connor Z Lin, David B Lindell, Eric R Chan, and Gordon Wetzstein. 3d gan inversion for control- lable portrait image animation. arXiv preprint arXiv:2203.1 13441, 2022.
Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B. Tenenbaum. Compositional visual generation with composable diffusion models. In Computer Vision ECCV 2022 17th European Conference, volume 13677, pp. 423-439, 2022.
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver: fast ODE solver for diffusion probabilistic model sampling in around 10 steps. In NeurIPS, 2022.
Mohit Mendiratta, Xingang Pan, Mohamed Elgharib, Kartik Teotia, Mallikarjun R, Ayush Tewari, Vladislav Golyanik, Adam Kortylewski, and Christian Theobalt. Avatarstudio: Text-driven edit- ing of 3d dynamic human head avatars, 2023.
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021.
Michael Niemeyer and Andreas Geiger. GIRAFFE: representing scenes as compositional generative neural feature fields. In IEEE Conference Computer Vision and Pattern Recognition, CVPR, pp. 1443-111664, 2021.
Or Patashnik, Zongze Wu, Eli Shechtman, Daniel Cohen-Or, and Dani Lischinski. Styleclip: Text- driven manipulation of stylegan imagery. In JEEE/CVF International Conference on Computer Vision, ICCV, pp. 2065-2074, 2021.
William Peebles and Saining Xie. Scalable diffusion models with transformers. arXiv preprint 1229992 2022.
11

---

Published as a conference paper at ICLR 2024
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In Proceedings of the 38th International Conference on Machine Learning, ICMI,, volume 139, pp. 8748-8763, 2021.
Daniel Roich, Ron Mokady, Amit H Bermano, and Daniel Cohen-Or. Pivotal tuning for latent-based editing of real images. ACM Transactions on Graphics (TOG), 42(1):1-13, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In IEEE/CVF Conference on Computer Vision and Puttern Recognition, CVPR, pp. 10674-10685, 2022.
Olaf Ronneherger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomed- ical image segmentation. In Medical Image Computing and Computer-Assisted Intervention MICCAI, volume 9351, pp. 234-241, 2015.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.
Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamies. In Proceedings of the 32nd International Conference on Machine Learning, ICML, volume 37, pp. 2256-2265 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. In 9th International Conference on Learning Representations. ICLR, 2021.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribu- tion. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems, pp. 11895-19907 2019.
Jianxin Sun, Qiyao Deng, Qi Li, Muyi Sun, Min Ren, and Zhenan Sun. Anyface: Free-style text- to-face synthesis and manipulation. In /EEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR, pp.
2022a.
Jingxiang Sun, Xuan Wang, Yichun Shi, Lizhen Wang. Jue Wang, and Yebin Liu. IDE-3D: interac- tive disentangled editing for high-resolution 3d-aware portrait synthesis. ACM Trans. Graph., 41 10:270:1-270:11 2022h.
Omer Tov, Yuval Alaluf, Yotam Nitzan, Or Patashník, and Daniel Cohen-Or. Designing an encoder for stylegan image manipulation. ACM Trans. Graph., 40(4): 133:1-133:14, 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. In Advances Neural Infor- mation Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pp. 5998-6008, 2017.
Tengfei Wang, Bo Zhang, Ting Zhang, Shuyang Gu, Jianmin Bao, Tadas Baltrusaitis, Jingjing Shen, Dong Chen, Fang Wen, Qifeng Chen, et al. Rodin: A generative model for sculpting 3d digital avatars using diffusion. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
2023.
Zongze Wu, Dani Lischinski, and Eli Shechtman. Stylespace analysis: Disentangled controls for stylegan image generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12863 12872, 2021.
Weihao Xia, Yujiu Yang. Jing- Hao Xue. and Baoyuan Wu. Tedigan: Text-guided diverse face image generation and manipulation. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, pp. 2256-2266222 2265, 2021.
Jiaxin Xie, Hao Ouyang, Jingtan Piao, Chenyang Lei, and Qifeng Chen. High-fidelity 3d gan in- multi-view view optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 321-331, 2023.
version by pseudo-mon cocuudo-muti-ve
12

---

Published as a conference paper at ICLR 2024
Lvmin Zhang and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. arXiv preprint 0rXiv:2302.05544 2023.
Kaiwen Zheng, Cheng Lu, Jianfei Chen, and Jun Zhu. Improved techniques for maximum likelihood estimation for diffusion odes. In International Conference Machine Learning, ICML, volume 202, pp. 42363-42389, 2023.
A APPENDIX
In the Appendix, first provide implementation details, including the model parameters and train- ing dataset. We follow with additional experiments and visual results. We highly recommend watch- ing video, which contains a live demonstration of interactive instructed editing.
A.1 IMPLEMENTATION DETAILS
A.1.1 EXPERIMENT SETTING
We train our conditional diffusion model the dataset prepared from FFHQ (Karras al. 2019) and use CelebA-HQ Karras et al. 2018) for evaluation. In our experiments, we use pretrained EG3D model (Chan et al. 2022), pretrained PREIM3D model (Li et al. 2023), and pretrained 'VIT- H-14' CLIP model (Radford et al. 2021). The Diffusion Transformer is modified from the backbone of DDIT B/1', adding input header, text condition module with the cross-aftention mechanism, and an identity modulation module. The number of parameters in the model 1.15 Billion. We set les 600, And 0.1 and trained the model 4-card NVIDIA GeForce RTX 3090 for 6 days with a batch size of 20 on a single card.
A.1.2 TRAINING DATASET
The triplet data encourages the model to learn the correlation between the change from pairs of im- ages and the instruction. Since the perceptual changes between the paired images well-defined, we can easily generate corresponding instructions. Figure 6 shows the data preparation. For the paired images generated by e4e, provide ChatGPT with example data consisting of the paired attribute labels and a few corresponding handwritten transfer instructions. We guide Chat- GPT by following these steps.
a. You are now an excellent data generation assistant. The rule for generating data is that I give you an imput label and an output label, and you help me generate instruction that represents the change from the input label to the output label. These are some examples.
example I
input label: eyeglasses
output label: without eyeglasses
instruction: remove the eyeglasses. example 2
input label: no beard
output label: beard man instruction: give him some beard example 3
input label: an old lady
output label: a young girl
instruction: make her look more youthfud.
example 4
input label: brown hair output label: blond hair
instruction: turn the hair to blond I will give you an input label and an output label to generate 10 institutions based on the rules
and examples above.
input label: a small nose output label: big nose
13

---

Published as a conference paper at ICLR 2024
For the paired image generated by InstructPix2Pix, guide ChatGPT like this, "You are cellent instruction generation assistant. give you a face editing instruction, please generate 30 instructions that are semantically identical this one.".
Our dataset has about 40K paired images and the corresponding 3K instructions, with e4e generating 22K paired images and InstructPix2Pix generating 18K paired images. For the images generated by e4e, 500 images share about 30 instructions. For the images generated by InstructPix2Pix, 120 images share about 10 instructions. The resolution of the image is 512 All instructions single editing requirements. We train our model with the dataset cropped as EG3D (Chan et al. 2022 and PREIM3D (Li et al. 2023). The dataset examples are provided in Figure
Paired images generation
e4e
InstructPixelPPixel
Instruction generation
Keyword: eyeglasses
Instruction: Turn the portrait into bronze statuc.
Instruction: Add eyeglasses to his face.
ChatGPT
ChatGPT
Instruction: Give him a pair of glasses.
Instruction: Transform the painting into a bronze statue.
Make him wear glasses.
Sculpt the painting in bronzc.
Triplet data examples
Add bungs to her huirstyle.
Make him wear glasses.
Have it be puinting by Edward Hopper.
Trrmsform the puinting into a bronze statue.
Figure 6: We use e4e and InstrucPixe/2Pixel to generate paired images and use ChatGPT to produce the instructions.
A.1.3 EVALUATION
Test data. The image test dataset the first 300 images from CelebA-HQ Karras et al. 2018) The instruction test set used in the comparison experiments consists of selected instructions from Talk-To-Edit InstructPix2Pix, and InstructPix2NeRR. where each model contributes 15 instructions, such as 'Make her huppy instead.' and 'The eyeglasses could be more obvious.'. The multiple instruction comprises single instructions that are concated together. We show the test instructions in Table 7 g and
Baselines. Talk-To-Edi and InstructPix2Pix are the most popular methods enabling instructed 2D editing. We perform 2D portrait editing guided by the instructions and then get the 3D-aware edited images using the of-te-of-the-ar 3D inversion encoder PREIM3D. For another baseline img2img only trained text-to-3D diffusion model using the instructions and edited images in the prepared dataset. Similar to the ing2img mode Stable Diffusion, we apply denoising sampling to the latent code of the input image with small amount of noise added, conditional on the instruction.
14

---

Published as a conference paper at ICLR 2024
Trim his hair have bangs
Add bungs her hairstyle.
Add some bangs
make her have blonde bair
change her hair to blond
turn the bair blonde
add a pair of eyeglasses
Make her wear eyeglasses.
Add glasses to her face.
give him mustache
have him mustache
Add mustache him
Make him appear to be grinning.
make her have smiling mouth
Have her wear smile
Figure 7: The dataset examples.
Metries. The subject's multi-view identity consistency (ID) is measured by the average ArcFace feature similarity score (Deng al.) (022) between the sampled images and the input image. We evaluate the precision of instructed editing with the directional CLIP similarity (Gal et al.) 2022). which calculated by the cosine similarity of the CLIP-space direction between the input image and multi-view edited images and the CLIP-space direction between the input prompt and edited prompt. Here, our input prompt is composed of attribute labels with a probability greater than 0.9 in trr-the-shlff multi-labe classifier based on ResNet50 (Huang & Belongie 2017), and the edited prompt is appended by the input prompt with the attribute label you want to edit. Following eLi al.) 2023
Wu et al. 2021) We attribute altering (AA) measure the change of the desired attribute
15

---

Published as a conference paper at ICLR 2024
Style her hair with pink wig, put eyeglasses on her.
Put eyeglasses on her, make her happy instead, the hair color blue.
Put eyeglasses on her, crea- smiling expression, and turn the painting into bro-
statue.
instruction
input
w/o TPR
TPR
Figure 8: Editing results of multiple instructions. w/o TPR denotes the model without token position randomization training scheme.
and use attribute dependency (AD) to measure the change in other attributes when modifying specific attribute. Attribute altering (AA) is the attribute logit change Ai; normalized by the standard deviation (l) when detecting attribute by the classifier, and attribute dependency (AD) is the other computed the mean differences (Ma) and standard deviation differences (S₂) metrics between the depth maps of the editing results and the depth map of the randomly sampled images in EG3D measure 3D consistency.
attribute logit change. Following Abdal et al. (2023),
A.2 MORE EXPERIMENTS
A.2.1 PROMPT-DRIVEN EDITING
Text-image pairing contributes significantly to text-to-image and image captioning. Focusing on ad- jectives and nouns in the text, prompt-driven editing methods perform better on descriptive prompts than instructed text. For example, StyleClip (Patashnik et al.] [202]). the state-of- soate-of-the-and th-tth--rt text-guided face editing method, usually misinterprets some verbs, as shown in Figure 9
input
remove the eyeglasses
take off the eyeglasses
input
shave the beard
remove the heard
Figure 9: The instructed editing results of StyleClip.
A.2.2 EXPLORING THE W+ SPACE OF EG3D
EG3D, the state-of-the-art NeRF-based generator in the face domain, is trained on real-world face dataset. The original EG3D can generate high-resolution multi-view-consistentt images and high- quality 3D geometry from the Gaussian noise. Although the original EG3D can only generate faces with a similar distribution to the FFHQ data, get wider range of reasonable face images by navigating in the W+ space. The intermediate latent space of EG3D is manifold of identical 5 312-dimensional latent vectors. Using k distinct latent vectors rather than identical ones, greatly extend the generation capability of the fixed EG3D. Figure 17 illustrates our exploration of W4 space in EG3D.
16

---

Published as a conference paper at ICLR 2024
A.2.3 MORE ATTRIBUTE EDITING
We evaluated the instructed editing performance of more attributes, as shown in Table Our method outperforms InstructPix2Pix and img2img in AA, AD, and ID metries.
Table 5: Quantitative evaluation for more instructed attribute editing.
Instruction Method
five o' c'oock shadow
arched eyebrows
attractive
bags under eyes
bald
IP2P
121
IP2N
IP2P
121
IP2N
IP2P
121
IP2N
IP2P
121
IP2N
IP2P
121
IP2N
ID1
0.45
0.52
0.52
0.64
0.59
01.48
0.55
0.63
0.42
0.47
0.56
AAt
0.29
0.94
0.95
-0.30
0.09
0.13
-0.29
-0.01
0.40
0.35
1.09
1.06
0.93
0.93
1.04
ADD
0.76
0.77
0.74
0.50
0.67
0.48
0.55
0.62
0.53
0.65
0.61
0.60
0.76
0.74
0.71
Instruction
big lips
hig nose
black hair
hlond hair
brown hair
ID1
0.53
0.50
0.59
0.42
0.49
0.59
0.48
0.54
0.63
0.54
0.47
0.60
0.51
0.52
0.62
AAT
0.37
0.37
0.44
0.42
0.65
0.97
1.05
1.12
1.09
0.96
0.87
0.98
0.36
0.14
0.28
ADI
0.53
0,52
0.49
0.75
0.66
0.65
0.59
0.60
0.52
0.58
0.64
0.55
0.58
0.61
0,56
Instruction
bushy eyelrows
chubby
double chin
young
goolee
ID1
0.56
0.53
0.64
0.48
0.50
0.64
0.48
0.51
0.64
0.43
0.47
0.50
0.57
0.47
0.61
AAT
0.89
1.02
1.00
0.98
1.05
1.05
0.94
0.96
1.00
0.21
0.01
0.77
1.25
1.30
AD
0:75
0.62
0.51
0.59
0.65
0.54
0.76
0.76
0.63
0.68
0.65
0.63
0.80
0.85
0.78
Instruction
gray
heavy makeup
lugh cheekbones
male
mouth open
ID1
0.52
0.54
0.64
0.54
0.51
0.63
0,55
0.51
0.64
0.38
0.48
0.62
0.45
0.50
0.65
AAT
1.11
1.06
0.99
0.11
0.03
0.53
0.29
0.96
1.07
1.03
1.16
1.03
0.84
1.12
1.21
AD
0.59
0.63
0.57
0.52
0.55
0.50
0.57
0.62
0.54
0.80
0.78
0.68
0.71
0.57
0.50
Instuction
mustache
Barrow eye
no beard
pale skin
pointy nose
ID1
0.51
0.46
0.50
0.65
0.45
0.50
0.62
0.50
0.66
0.52
0.48
AAT
0.55
0.50
0.47
0.15
1.23
1.46
-0.39
-0.09
0.27
1.02
1.03
1.11
-0.33
-0.16
0.57
ADI
0.64
0.72
0.68
0.55
0.56
0.50
0.72
0.70
0.49
0.51
0.53
0.47
0.56
0.60
Instruction
rosy checks
sideburss
lipstick
straight hair
wavy hair
IDa
0.51
0.52
0.60
0.34
0.52
0.62
0.55
0.51
0.62
0.44
0.50
0.62
0.33
0.47
0.59
AAT
-0.33
0.48
0.52
1,02
1.25
1.29
0.15
0.07
0.41
0.57
0.27
0.45
0.72
0.89
0.99
ADI
0.49
0.55
0.51
0.89
0.82
0.75
0.51
0.53
0.50
0.57
0.62
0.53
0.84
0,54
0.49
note:
IP2P, 121, and IP2N denote InstructPix2Pix, img2ing, and InstructPix2NeR1 respectively.
A.2.4 USER STUDY
To perceptually evaluate the instructed 3D-aware editing performance, we conduct a user study in T We collected 1,440 votes from 30 volunteers, who evaluated the text instruction correspon- dence and multi-view identity consistency of editing results. Each volunteer is given source image, our editing result, and baseline editing, and asked to choose the better one, as shown in Figure [8 The user study shows our method outperforms the baselines.
Table 6: The result of our user study. The value represents the rate of Ours > others. Multiple instructions indicate editing with the combinations of the above three attributes.
Method
bang
eyeglasses
smile
multiple instructions
Talk-to-Eche Edit
0.742
0.958
0.817
0.875
InstructPix2Pix
0.833
0.667
0.683 0.725
img2img
0.733
0.758
0.750
0.783
A.2.5 W+ OPTIMIZATION ABLATION
In 2D image editing, although optimization-basee inversion may be more time-consuming than incdder-based inversion, it produces better identity consistency. We replace the inversion encoder with latent optimization in our pipeline to see if it improves identity consistency. Since each opti- mization takes several minutes, we cannot perform optimizations during training, but only at infer- ence time.
17

---

Published as a conference paper at ICLR 2024
In our experiments, considered two configurations: Direct W+ optimization and PTI Roich al. (202)) optimization. Direct W+ optimization involves optimizing the W+ vector while keeping the generator fixed. PTI (Pivotal Tuning Inversion) technique fine-tunes the generator based on the initial value provided by direct optimization. We conducted 500 steps of optimization on the W++ vector, and PTI added 100 steps of fine-tuning the generator.
The results of these experiments are presented in Figure [0] where we compare the outcomes of di- rect W++ optimization, PTI, and the encoder-based method. The results show that directly replacing the encoder with an optimization method during inference will lead to a severe decrease in both edit- ing effect and identity consistency. We attribute this issue to the deviation between the model and data distribution. The model learns a conditional distribution within the encoder's inversion space during training. When the encoder replaced by an optimization method during inference, the data distribution used for inference mismatches the learned model distribution. This mismatch results in greater identity drift and undesirable editing outcomes.
While conducting W+ optimization during training (much larger compute) could potentially ad- dress the distribution deviation problem, it may introduce artifacts novel views, as pointed out by PREIM3D. This is due to optimization being performed on single image during training. In sum- mary, while direct optimization of the W++ vector is an interesting concept, our experiments suggest that it may not necessarily lead to improved identity preservation and editing results compared to the encoder-based approach.
A.2.6 EFFECTS OF THE BACKGROUND
To verify the effect of background the editing results, we edited images of the same person in different scenes. We show the results in Figure where the first two rows are the same subject, the middle rows are the same subject and the last two that the background has no obvious impact on the editing results. However, note that when editing colors, particularly when the color being edited close to the background color, there can be some blending between the foreground and background elements.
the same subject. The results show
A.2.7 THE TRIPLET DATA MECHANISM A ABLATION
The triplet data mechanism plays crucial role in achieving accurate and disentangled image edit- ing. To provide a more thorough understanding of its importance, we conducted an ablation study. Our comparison involves the img2img model, which can be considered as using a text-image pair- ing data mechanism, in contrast to our method which utilizes the triplet data mechanism. Unlike InstrucdPix2NeRR, img2img has no paired images, but the rest of the network structure is the same.
The results of the ablation study, as shown in 3 and Table 3 and [ demonstrate that the triplet data mechanism significantly contributes to the quality of editing terms of identity preservation (ID) and attribute dependency (AD) when attribute altering (AA) is close to equal.
Our method consistently outperforms img2ing preserving identity across various attributes, as indicated by the higher ID scores. Moreover, the triplet data mechanism helps reduce attribute de- pendency, ensuring that changes to one attribute do not excessively affect others. These results highlight that the triplet data mechanism encourages the model learn the correlations between changes in pairs of images and the corresponding instructions, leading to more precise and disentan- gled editing. In conclusion, the triplet data mechanism is essential for achieving high-quality image editing results.
A.3
MORE VISUAL RESULTS
We provide large number of instructed editing results produced by InstructPix2NeRF in Figure 12 13
14
15]
and 16
Diversity and generalization. We realize the importance of diversity in the evaluation of image editing methods and strive to provide a comprehensive evaluation. As described in Appendix A. Experimental Settings, our model is trained on the FFHQ dataset, which consists of 70,000 high- quality faces featuring vast diversity in terms of age, ethnicity, and background, while also exhibit- ing comprehensive representation of accessrries such glasses, sunglasses, and hats. This diverse
18

---

Published as a conference paper at ICLR 2024
Make the
Mike her lave smillng
Tum her hair color to pink
Make her look like cantoon
bronre statue
Input
W+ optimization
PTI
Encuder
Figure 10: Replacing the inversion encoder with the optimization method.
19

---

Published as a conference paper at ICLR 2024
Make smile
Make him smile
Give him bushy eyebrows
Give him bushy eyebrows
O
Change the hair color to blonde
Change the hair color to blonde
Make her cartoon character
Make her cartoon character
Turn his hair color to pink
Tur his hair color to pink
Change the hair color to blonde
Change the hair color blonde
Figure 11: The effects of background.
20

---

Published as a conference paper at ICLR 2024
Table 7: The test human instructions on bangs.
Source
Instruction
Add a fringe to the wig.
Make the hairstyles have fringe.
InstructPix2Pix
Make the bangs fringes.
Add a fringe.
Flapper hairstyles with a fringe.
Let's try long bangs.
What about adding longer bangs?
Talk-To-Edh
Emm, feel the bangs can be longer.
How about trying adding longer bangs? The bangs should be much longer.
Style his hair with bangs.
Move the bangs to the front.
InstructPix2NeRF
Let's add some bangs.
Give the girl bangs.
Add bangs to the hairstyle.
Table 8: The test human instructions eyeglasses.
Source
Instruction
with glasses
Make the person wearing glasses
InstructPix2Pix
Make him wear glasses.
Have the faces be wearing glasses.
The women are wearing eyeglasses
Make the eyeglasses more obvious. How about trying thick frame eyeglasses. It should be cycglasses with thin frame. The eyeglasses could be more obvious. Try eyeglasses.
Falk-ToEEdit
Make her wear glasses. Give her pair of eyeglasses.
InstructPix2NcRF
Apply glasses to her face.
Place eyeglasses on her nose. Add a pair of eyeglasses
training data ensures that model generalize various races and attributes. As shown in Figures 14 and 15, our models are capable of handling different races, hairstyles, ages, and other at- tributes. In Figure 16, we demonstrate this capability using the results of editing scenarios featuring characters from this year's 's movie "Mission: Impossible Dead Reckoning Part One."
21

---

Published as a conference paper at ICLR 2024
Turn her into vampire
Turn her into vampire
Give it zombic makeover
Give it zombic makeover
GNI
M
DS
Make look like sketch by Edward
Make it look like sketch by Edward Hopper
Hopper
O
Make a cartoon character
Make her cartoon character
painting into a bronze statue
Turn painting into bronze statue
Figure 12: More Visual Results.
22

---

Published as a conference paper at ICLR 2024
Turn the hair color to pink
Turn the hair color to pink
Turn the hair color to pink, put eyeglasses her
Turn the hair color to pink, eyeglasses her
Give the portrait comic book look
Give the portrait comic book look
ER
RDS
TISH
A
Angeles
Put eyeglasses on face, give him
Put eyeglasses on face, give him
gonlee
goalee
Remove the beard
Remove the beard
Figure 13: More Visual Results.
23

---

Published as a conference paper at ICLR 2024
Make eyes appear narrower
Make her eyes appear narrower
Style her hair with pink wig
Style her with pink wig
Shave the beard off
Shave the beard off
Have his smile
Add smile her face
OTE
ATERN
Tum him into vampire
Give it a zombie makeover
Figure 14: More Visual Results.
24

---

Published as a conference paper at ICLR 2024
Put eyeglasses on her
eyeglasses on her
Add some beard
Add some beard
Give her thicker, bushier eyebrows
Add bangs to the hairstyle
Turn the portrait into bronze statue
Turn the portrait into bronze statue
Make her look like cartoon
Give the portrait comic book look
Figure 15: More Visual Results.
25

---

Published as a conference paper at ICLR 2024
Give her a happy expression
Turn her hair to pink
Turn the portrait into bronze statue
Make her cartoon character
Figure 16: Characters in Mission: Impossible Dead Reckoning Part One.
Table 9: The test human instructions on smile.
Source
Instruction
Have her be smiling
The faces should be smiling
InstructPix2Pix
Have open-mouthed shark smiling Add "smiling" emoji
Make her happy instead
The person should smile more happily.
The person smile to some degree more happily.
Talk-To-Ddi
I kind of want the face to be smiling with the mouth wide open. I would like the face be smiling with teeth visible.
would like change the pokerface face smiling face.
Alter her expression to appear cheerful and merry.
Give her happy expression with smile and bright eyes.
InstruetPix2NRRF Adjust her features convey sense of happiness and positivity.
Make her look cheerful and full of good cheer.
Make the person have smiling face
26

---

Published as a conference paper at ICLR 2024
Figure 17: Our exploration of W+ space.
27

---

Published as a conference paper at ICLR 2024
imuge plaase
image bekon beteer
+1. The tao image the insut image, plaase choose the mage belin with better odiking roulla (slatity conastercy hew wel matches ledt intructin)) end
resulss ldbetity consitercy
the
edtting
Sekentgg
lidariity and
Eilting
matches intruction).
bargs person should
Editing insruction Let's and some hangs.
inamuction Make
happly gloses.
a
Figure 18: Examples from user study, the first choice (A: Ours, B: InstructPix2Pix), the second choice (A: img2img. B: Ours), the third choice (A: ours, B: Talk-To-Edin the fourth choice (A: InstructPix2Pix. B: Ours)
28


Reviews:
[{"review_id": "ehuGGxG3iO", "reviewer": "Reviewer_TVmA", "summary": "The proposed approach, InstructPix2NeRF, is an end-to-end model designed for 3D-aware human head editing using a single image and an instructive prompt as inputs. To achieve this results, firstly the authors construct a multimodal 2D human head dataset by leveraging pretrained diffusion models such as e4e and InstructPix2Pix. Secondly, they propose a token position randomization strategy to enhance the model's ability to edit multiple attributes simultaneously. Last, an identity consistency module is incorporated to extract facial identity signals from the input image and guide the editing process. Experimental results demonstrate the effectiveness and superiority of the method.", "questions": "Besides the weaknesses above, I may have some questions that hope the authors can answer:\n1. There lack the reason for generating and using 20-30 instruction prompts for one single paired image. Will the number of instructions affect the training?\n2. How will the model perform when it deals with novel characters as in the movie, long hair examples, black men/women, and human of different ages?\n3. Will the background affect the edited results? It would be interesting to see the outcomes obtained when editing the same subject against various backgrounds.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "The strengths of the proposed paper can be summarized as:\n1. The authors propose a token randomization strategy that can increase the model's capability for editing multiple attributes simultaneously.\n2. An identity-preserving module is proposed to guide the editing process and present the original identity in the final outcomes.\n3. The proposed method is reported to be time-friendly, producing the results in few seconds.", "weaknesses": "The weaknesses of the proposed method can be summarized as:\n1. Through the visualization in Figure 1, I find that the original identity and RGB image attributes are not well preserved. Large differences can still be observed in the areas that are not supposed to be edited.\n2. Qualitative comparisons. (1) The proposed method seems to struggle with expression editing, e.g., it fails to make the head smiling; The instruct-pix2pix model doesn't encounter this problem; (2) Regarding the \"bangs\" example, I would prefer the instruct-pix2pix as it contains real bangs; (3) There is no comparisons with Instruct-NeRF2NeRF, AvatarStudio, and HeadSculpt, considering they are more similar works than the compared Talk-to-Edit and img2img; (4) More examples and more scenarios will largely improve the validation. Currently, there are only three types presented.\n3. Quantitative comparisons. (1) The evaluations are not comprehensive. Still, only three examples are presented; (2) More quantitative evaluations, e.g., user studies, would be beneficial.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Y3I41VMxtI", "reviewer": "Reviewer_e7JS", "summary": "This paper proposes a method that enables the creation of 3D portraits that have been edited based on text prompts. Leveraging the latent space of EG3D to impose 3D consistency, the proposed method finds a latent vector in the W+ space that matches the edits specified by the prompt and the identity in the input image. A diffusion model conditioned on the input 2D image and the editing prompt is used to predict this latent vector. Additionally, the paper proposes the following 1) Token position randomization to improve the quality of multi-instruction editing 2) An identity consistency module to improve identity preservation during edits.", "questions": "1) Instead of an Encoder, if direct optimization of the W+ vector was used (assuming much larger compute), would it preserve the identity better? What if this is only done during inference and not training?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1) While each individual component of the method isn’t novel, the whole method itself is\n\n2) Qualitative results in both the paper and appendix demonstrate plausible editing, though some identity loss remains\n\n3) Quantitative results demonstrate that the method better preserves the identity across edits. The user study additionally bolsters the main contribution of the paper.", "weaknesses": "1) The methods section could be written better, with a clear exposition of losses during training and the forward pass during inference. To that end, Fig 2 should be expanded to include both training and inference settings. \n\n2) While the identity consistency is better preserved that prior work, the still remains and identity drift during editing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qFi7C7D3YM", "reviewer": "Reviewer_1rfe", "summary": "This paper proposes a method that generates a 3D-edited NeRF from a single portrait image. The style is defined by an instructing prompt. Two streams of inputs of a real face and a 2D-edited face are passed through an encoder to generate identity conditions. The identity condition, together with text condition, is sent to a diffusion model to generate tri-plane features for NeRF rendering. Experimental results show that the proposed method outperforms compared baseline approaches under the authors' settings.", "questions": "I would like to see the authors address my concerns mentioned in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- It is the first (to my knowledge) paper that allows \"instructed\" 3D portrait editing from single images.\n\n- The experiments show that the proposed method outperforms compared baselines under the authors' settings.", "weaknesses": "- The results shown in the paper lack race diversity. There are almost no Asian or black people. I'm worried whether the proposed method does not perform well on those cases.\n\n- The identity may change after applying the proposed method. For example, in Fig. 1 first example, the eye shape changed after the beard was removed. In Fig. 3 middle example, the girl seems to look more Asian and the nose shape changed after editing. These are not analyzed in the limitation section.\n\n- The proposed method adopts two streams of inputs (real and edited images). However, the ablation study does not show the necessity of  \n them. Will only one stream work?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)