Enjoy Your Editing: Controllable GANs for Image Editing via Latent Space Navigation
Authors: Peiye Zhuang
Oluwasanmi O Koyejo
Alex Schwing
Venue: ICLR 2021 Poster (2021)
Keywords: Image manipulation
GANs
latent space of GANs

Abstract: Controllable semantic image editing enables a user to change entire image attributes with a few clicks, e.g., gradually making a summer scene look like it was taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global image identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute regression into the training of transformation functions, and apply a content loss and an adversarial loss that encourages the maintenance of image identity and photo-realism. We propose quantitative evaluation strategies for measuring controllable editing performance, unlike prior work, which primarily focuses on qualitative evaluation. Our model permits better control for both single- and multiple-attribute editing while preserving image identity and realism during transformation. We provide empirical results for both natural and synthetic images, highlighting that our model achieves state-of-the-art performance for targeted image manipulation. 

Published as a conference paper at ICLR 2021
ENJOY YOUR EDITING: CONTROLLABLE GANs FOR IMAGE EDITING VIA LATENT SPACE NAVIGATION
Peiye Zhuang, Oluwasanmi Koyejo, Alexander G. Schwing University of Illinois at Urbana-Champaign (peiye, sanmi, aschwing) linois......... edu
ABSTRACT
Controllable semantic image editing enables a user to change entire image at- tributes with few clicks, c.g., gradually making summer scene look like it taken in winter. Classic approaches for this task use a Generative Adversarial Net (GAN) to learn a latent space and suitable latent-space transformations. However, current approaches often suffer from attribute edits that are entangled, global im- age identity changes, and diminished photo-realism. To address these concerns, we learn multiple attribute transformations simultaneously, integrate attribute re- gression into the training of transformation functions, and apply content loss and an adversarial loss that encourages the maintenance of image identity and photo-realism. We propose quantitative evaluation strategies for measuring con- trollable editing performance, unlike prior work, which primarily focuses on qual- itative evaluation. Our model permits better control for both single- and multiple- attribute editing while preserving image identity and realism during transforma- tion. We provide empirical results for both natural and synthetic images, high- lighting that our model achieves totte-of-the-ar art performance for targeted image manipulation.
INTRODUCTION
Semantic image editing is the task of transforming a source image to a target image while modifying desired semantic attributes, e.g., make an image taken during summer look like was captured in winter. The ability to semantically edit images is useful for various real-world tasks, including artistic visualization, design, photo enhancement, and targeted data augmentation. To this end, semantic image editing has two primary goals: (i) providing continuous manipulation of multiple attributes simultancously and (ii) maintaining the original image's s identity as much as possible while ensuring photo-realism.
Existing GAN-based approaches for semantic image editing can be categorized roughly into two groups: (i) amage-space editing methods directly transform one image to another across do- mains (Choi et al.] 2018 2020) Isola et al.] [017 Lee et al., 2020: Wu et al.] 2019 Zhu et al.]
2017a b). usually using variants of generative adversarial nets (GANs) (Goodfellow et al.) 2014). These approaches often have high computational cost, and they primarily focus on binary attribute (on/off) changes, rather than providing continuous attribute editing abilities. (ii) latent-space edit- ing methods focus on discovering latent variable manipulations that permit continuous semantic image edits. The chosen latent space is most often the latent space of GANs. Both unsupervised and (self- jsupervised latent space editing methods have been proposed. Unsupervised latent-space editing methods (Härkönen et al., 2020 Voynov Babenko 2020) are often less effective at pro- viding semantically meaningful directions and all too often change image identity during an edit. Current (self-) suppevieed methods (Jahanian et al., 2019 Plumerault et al., 2020) are limited to ge- ometric edits such as rotation and scale. To our knowledge, only one supervised approach has been proposed (Shen et al.) 2019) developed to discover semantic latent-space directions for binary attributes. As we show, this method suffers from entangled attributes and often does not preserve image identity during manipulation.
Contributions. We propose latent-space editing framework for semantic image manipulation that fulfills the aforementioned primary goals. Specifically, we use a GAN and employ a joint sampling strategy trained to edit multiple attributes simultaneously. To disentangle attribute transformations

---

Published as a conference paper at ICLR 2021
Ours
Original
Inversion
+ Night
Winter
+ Clonds
+ Sunny
C
Original
Inversion
Smile
Hlond
Smile Blond
Figure 1: Real image manipulation on scene (top two rows, photo from Flickr) and face (bottom two rows, unseen image from CelebA-HQ) using pretrained StyleGAN2 (Karras al. 2019b): We reconstruct the real images (col.1) 1) by finding a latent vector with the best inversion result (col.2) on StyleGAN2 (Abdal al. 2019). After that, we transform the latent vectors for single- and multiple- attribute manipulations (col.3-6). Note that unlike ours, the baseline method (Shen et al. 2019) either changes image identity or confounds semantic properties, or both.
the latent space of GANs, we integrate a regressor to predict the attributes that an image exhibits. The regressor also permits precise control of the manipulation degree and is easily extended to multiple attributes simultancously. In addition, we incorporate a perceptual loss (Li al. 2019 and
adversarial loss that helps preserve image identity and photo-realism during manipulation.
We compare our method to several popular frameworks, from existing image-to-image translation methods (Choi al. 22200] Wu et al.) 2019 Zhu et al.) 201 to latent space transformation-aet based approaches (Shen et al. 2019 Voynov & Babenko 2020). We mention that prior work primarily uses qualitative evaluation like the in Fig. [ In contrast, propose quantitative evaluation to measure controllabiliry. Both qualitative and quantitative results provide evidence that our approach outperforms prior work in terms of quality the semantic image manipulation while maintaining image identity.
RELATED WORK
Generative Adversarial Networks (GANs) Goodffeloo et al. 2014) have significantly improved realistic image generation in recent years Brook al. 2018
2019 Karras
et al. 2017] 2019a by Park et al., 2019 Zhang et al. 2018). For this, a GAN formulates a 2-player deep nets: generator that produces an image given random noise vector in the latent space, sampled from a known prior distribution, usually a normal or uniform distribution; (ii) a discriminator whose input is both synthetic and real data, which is be differentiated.
non-cooperative game between
Semantic image editing seeks to automate image manipulation semantics. Encouraged by the success of deep nets, recent works have applied deep learning methods for semantic image editing
tasks such as style transfer (Li et al. 2017 [uan et al. (2017), 77.. image-to-image translation (Choi et al.
2018 2020 Isola et al. 2017 Lee et al. 2020 Wang et al. 2018 Wu et al. 2019 Zhu et al. 17
and discovering semantically meaningful directions
GAN latent space (Härkönen et al. 2020
Jahanian et al. 2019; Plumerault al. 2020 Shen al. 2019 Voynov & Babenko 2020 Note
that our task is an extended version of semantic image editing that requires more comprehensive
control to satisfy user-desired operations. Therefore, most aforementioned approaches do not meet
2

---

Published as a conference paper at ICLR 2021
Regresoor
Trainable module
G
R
Goontent
Area
G
R
Separate binary
Ono-ss-Ohie
sampling
sampling
transformation
transformation
Outdor
(Choaa
Blckk
Freale
Sangled lploon
Male (tef)) Female (right)
(Botiom
(Toprrght)
(b)
Figure 2: Our overall framework (top row) and 3 training strategies different from prior work (bottom row). In (a), G takes 2 and an edited latent vector separately to synthesize images. T (di,..., ...,dy) are the trainable latent-space directions and & represents transformation extent. Original and edited image attributes, a and å, are predicted by the pre- pre-trained regressor R. The discriminator loss Ldise (the discriminator not shown due limited space). the regression loss Lreg. and the perceptual loss Ceontet are used to update T. We explain (b-d) in Sec. S.c. 3.2.
the requirements. Nonetheless, we categorize the approaches most relevant to our task into two groups: (i) image editing via manipulation in image space, and (ii) image editing via latent space navigation.
Image-space editing using GANs directly manipulates an image for targeted editing. Early work (Isola et al. 2017) used GANs implement semantic translations between two image do- mains with paired data, c.g., day to night. Follow-up work focused multi-modi (Bhattarai & Kim 2020 Choi et al. 2018 2020 Lee et al. 2020 Wang et al. 2018 Wu et al.) 2019) and un- paired image domain translation (Zuu al. 70) In this case, they primarily consider binary (on/off) attribute changes, regardless of whether the process is dynamic, for example, day to night.
Latent-space editing via GANs has received increasing amount of recent interest. Most prior work focused on identifying semantically meaningful directions in the latent space of GANs, so that shifting latent vectors towards these directions achieves the desired image manipulation. Re- cent papers found semantics in the latent space of GANs, such as color transformations and camera movements (Jahanian et (2019), or face attribute changes (Shen et al. a 2019), such as smile. Other work considered unsupervised methods (Härkönen et al. 2220) Voynov & Babenko) (2020) to dis- cover interpretable latent space directions. However, additional challenges inherent unsupervised manipulation of the latent space arise, and have not been addressed in prior work, e.g., direction quality with regard to degree control and image identity preservation.
3 METHOD
3.1 PROBLEM STATEMENT
We consider controllable semantic image editing via latent space navigation in GANs. We begin with fixed GAN model that consists of generator G and a discriminator D. The input of G latent vector z E Imm from a latent space Z. Here, m is the dimensionality aim to discover semantically meaningful latent-space GAN directions, T {di,..., dx) to manipulate the attributes of synthetic images G(z) with an assigned step size {E,,.... EN)), where d, Rm for all (1,.... N). In the end, each attribute of an edited image can be changed with the corresponding degree from a = {ai..... ony. ). the original attributes of G(z) predicted by a regressor R.
the latent
space. Given N attributes,
3

---

Published as a conference paper at ICLR 2021
3.2 PROPOSED APPROACH
We provide an overview of our method in Fig. (a), and further illustrate how our approach differs from prior work in Fig. (b-d).
Overview. As shown in Fig. 2 (a), we employ a GAN model consisting of a generator G and a discriminator D (not shown due to limited space), as well as regressor R. all of which are pre- trained. Our goal is to find latent directions T that provide attribute specific image edits. At each training step, G takes and an edited latent vector referred as z'. We follow prior work which suggests that direction vectors in latent space of GANs permit image editing (Jahanian et al. 2019). Formally, given a transformation degree E, we obtain the latent-space edit as z 2 Ts
+ Exs E,d,. G provides the recovered and edited images G(z) and G(z'), which are separately processed by the regressor R to predict attribute values. The original attributes of ((z) are R()( The (pseudo~) ground truth and predicted attribute of G(z') are a' and &', respectively, where a' a + e and á' R(G((z)).
Intuitively, the goal of T is to transform by adding semantically meaningful information such that the corresponding output image G(z') exhibits attribute changes e from a. In practice, normalize the range of attribute values to [0, 1]. i.e, both a and a' E [0, 1]. During training, we maintain the unit range by controlling the given e. Formally, E is drawn from a distribution D: uniform in [-1, 1 while considering the constraint that 0 a a++ a 1.
Objective function. To find T we minimize the weighted objective:
min T
L A >i + A2hadisc + Ag
Note, the objective is only used for optimizing T, while the other modules remain fixed. Hyperpa- rameters λ1, '2, and A adjust the loss terms.
The regression loss Crgg assesses whether T performs the transformations indicated by E. We express the regression loss via binary cross entropy:
Cree E==z,e=D,I-a' log a' - (1 - â') (1) log (1 - a')].
Note that a' is from the distribution generated by z and e. Please refer to the appendix C. for more details on this distribution.
The second loss term Laise is computed using the discriminator D and measures the quality of the generated images, i.e.,
Laise Egroz [log(1 - D(G(=))))
Here we refer to the domain of z' via Z', which is conditioned on z. We write this using Z'|z.
Lastly, we use content loss Lconentt often also referred to as perceptual loss. It is designed to estimate the distance between two images and it is employed to maintain the image identity during the transformation. Specifically, we use the content loss term
Larntent BznZ,x/2''
Σ [[[(G(= -F(G(=))
where (.()) denotes feature function which extracts intermediate features from images. Decnttent indicates the layers of a pre-trained model which are used as features. We approximate the afore- mentioned expectations by empirical sampling. We defer algorithm details to Appendix A.
distt-iistribution sampling and training. The regressor operates in multi-label setting, i.e., each data possesses multiple attributes. To ensure the disentanglement of attribute edits we sample synthetic images from the entire data distribution and find all transformations at once (illustrated in Fig. (b,c) right). In contrast, Shen et al. (2019) prepare training samples on the two opposing data subsets with regards to attribute, e.g.. no clouds many clouds, and find the directions with N one-vs-one classifiers (sketched in Fig. 2 (b,c) left).
Transformation module T. We study two types of transformations T: (i) global and (ii) local. A global transformation T refers to a semantic latent-space transformation identical for all x during inference. This is illustrated via parallel red dashed arrow in Fig. 2 (d) left. These global directions
4

---

Published as a conference paper at ICLR 2021
Clonds
Night
Snow
Original
(6)
[e] Ours
Figure 3: Comparison of image-space editing approaches with respect to "removing clouds" (top), "enhancing night" (middle) and "adding snow" (bottom). The original images (col.1) are followed by given editing task. From (a-e): (a) CycleGAN (Zhu al. 2017a) (b) StarGAN v2 (Choi et al. 2020), (c) RelGAN (Wu et al. 2019), (d) DRIT++ (Lee et al. 2020). and (e) Ours.
+ Night
Snow
Original
(a)
(h)
(c) Ours
(9)
(b)
(c) Ours
Smile
Age
Original
(c)
(c)
Figure 4: Comparison of latent-space editing approaches StyleGAN2. (a) Shen et al. (211 (b) Voynov & Babenko (2020); (c) Ours. Original synthetic image (col. I); edited results of a seman- (olee-
tic manipulation noted top (col.2-4; col.5-7). [Shen & Zhou (202T) mention that "age," "gender,"
and "glasses" directions are hard to disentangle potentially due to data bias, while our results suggest better direction disentanglement.
are commonly used (Härkönen et al. 2020 Jahanian et al. 2019 Shen et al., 2019 Viazovetskyi
et al. 2020). However, globally identical direction might not serve all data. In contrast, local transformation is function of which provides various directions for different & (in Fig. 2 (d)
right, shown via non-parallel red arrows). Formally, d; fz), where f is implemented via deep net. Sec. we show that the local transformation succeeds on failure cases global transformation.
4 EXPERIMENTS
We introduce datasets, implementation details, evaluation metrics, and show results next.
Datasets. We evaluate the proposed approach
types of datasets: (i) face datasets FFHQ (Karras et al.] (019a)), CclebA (Liu et al., 2018) and CelebA- HQ (Karras et al. (017). com- monly used in prior work (Härkönen al. 2020 Karras al. 2017 2019 9 Shen et al. 2019
5

---

Published as a conference paper at ICLR 2021
Viazovetskyi et al. 2020). (ii) natural scene datasets Transient Attribute Database (Lffont al. 2014) and MIT Places2 data (Zhou et al.) 2017). which contain attributes suitable for continuous semantic image editing. We briefly introduce the scene datasets:
Transient Attribute Database (Laffont et al., 2014): It contains 8,571 scene images with 40 at- tributes in 5 categories including lighting (c.g., "bright"), weather (e.g.. "cloudy"), seasons (e.g., "winter"), subjective impressions (e.g., "beautiful"), and additional attributes (e.g., "dirty"). Each attribute is annotated with real-valued score between and 1, where 0 indicates absence of the attribute.
MIT Places2 data (Zhou et al. 2017): Using the provided category annotations (i.e., in- door/outdoor and natural/attifcciaa) we select the natural outdoor scenes, obtaining a total of 144,543 images.
Implementation details. We choose \ = 10 = 0.05, As = 0.05 in Eq. I and compute the perceptual loss using the conv1.2, conv2.2, conv3_2, conv4_2 activations in VGG- 19 network (Si- monyan & Zisserman 2014) pre-trained on the ImageNet dataset (Russakovsky et al. 2013). We train T for 50k iterations with a batch size of An Adam optimizer is used with a learning rate of le-4. For the latent space 2 in the proposed method we use the W space of StyleGAN2 and
the 2 space of PGGAN, For the attribute regressor Rt we adopt a ResNet-50 (He et al. (016) for attribute prediction on the Transient Attribute (Laffont et al. 2014) and the CelebA (Liu et al. 2018) data. The last fully connected layer in the ResNet-50 is replaced by a linear layer with an output dimension of 40. We train the regressors for 500 epochs and use the weights with the best validation mean squared error (MSE) on CelebA and the best test MSE on the Transient Scene Database. The GAN follows the StyleGAN2 architecture and is pretrained with 200k iterations two natural scene datasets using a resolution of 256 x 256. The training batch size is 32 and an Adam optimizer is used with a learning rate of 2c-3. For the face datasets, we use FFHQ pre-trained weights of StyleGAN2, and CelebA-HQ weights of PGGAN
union of the
Baselines. We compare our approach to several popular image-to-image translation ap- Wu et al. 2019 Zhu et al. 2017a) and latent space direction discovery methods (Shen et 2019 Voynov & Babenko 2020). Since et al. (2020): SShn et al. (2019);
proaches (Choi et al.
2020
Zhu et al. (2017a) cannot deal with continuous attributes. we split the data into domains using threshold value of 0.5 for each of the 40 attributes. Afterward, the models are trained on contrast domains, e.g., with or without certain attribute. We use the official code of all baseline methods and rigorously follow their training steps. Note that work by Voynov & Babenko (2020) is unsuper- vised, i.e., the latent-space directions are human interpreted. To avoid bias during the selection of directions, we use the attribute regressor to automatically identify the most significant directions that can edit predetermined attributes. The details of preparing the baselines are given in Appendix C.2.
Evaluation metrics. There is no good numerical metric to evaluate image editing (Shen et al. 2019 Voynov & Babenko 2020). In an attempt address this concern, we automate quantitative evaluation based on property that editing of attributes should maintain image identity. To achieve this, we employ a popular image identity recognition mode
( pre-trained on the VGGface2 data (Cao et al. 2018). Cosine similarity is used to represent the similarity between paired original and edited images (Cao al. 2018). In addition, we evaluate changes of the other independent attributes using the pre-trained regressor.
4.1 RESULTS ON NATURAL SCENE DATASETS
translation: As shown in Fig. 3 (a-d), we observe image-to-image translation to perform poorly when editing image details (e.g., removing clouds). Moreover, some- times artifacts (orange spots in (a) and (b)) are introduced. In contrast, our model shows overall compelling transfer performance on all these attributes.
Comparison to image-to-imast
Comparison to latent-space translation:
Inspecting Fig. I we observe our approach to improve upon work by (Shen et al. (2099) in attribute edit quality and number of artifacts. Similarly, in
hhttps//pyytorch.or//uub http:://giuuh.co/oo.vggggge face2

---

Published as a conference paper at ICLR 2021
0.8
0.6
0.4
Questionl: Compared to the picture the left, how cloudy is the picture on the right?
0.2
Question2: Ignoring the attribute clouds, determine if the two pictures record the same scene.
clouds
sunrise&sunset
night
Mean resporse for
llgnoring attributue m, are i and different
Figure 5: User study for image editing and image identity preservation. Blue bars show the predicted presence of a given attribute for images (I+, higher is better). The orange bars measure the difference between paired images (lower better).
Bandiine
Our
Original
Inversion
Blonnd Hair Smile
+ Black Hair Smile
+ Blond Hair Smile
+ Black Hair
+ Blond Hair Smile
+ Smile
+ Age
Figure 6: Real image synthesis with multiple attributes. Shen et al. (2019) (top row); ours (bottom row): original real image (col.1); inverted result (col.2) from Abdal al. (2019); edited
results (col.3-7) with target attribute changes shown at the bottom.
Fig. notice that our method performs well in attribute editing, and has a better ability to preserve image identity when adding "night" and "snow." We show additional analysis and results Appendix and E.I.
We follow a user study proposed by Kowalski et al. (2020) to evaluate controllability of our scene edits. Concretely, given an image -= G(z), we generate paired images (-+ that have opposing values for attribute n. 40 users took the test to answer: (a) the presence and strength the attribute /+, and (b) /+ and I- are identical when ignoring the attribute. Both types of questions have options on 5-level scale from ("not at all"") to ("totally"). Ideally, the response to question (a) should be while we expect replies for (b) to be O. We test performance of image editing with "clouds", "sunrise&sunset"" "night" and "snow" attributes. A question ample and statistical results over 50 image pairs are shown in Fig.5l Statistical evidence supports claim that the proposed approach performs well with regards to image edits while maintaining image identity.
Quantitative evaluation:
4.2 RESULTS ON FACE DATASETS
Manipulation results on StyleGAN2: The comparison on real images shown in Fig. T suggests that method works well for attribute edits. Further, the synthetic image edit results in Fig. Cindicate that our edits are disentangled, while the baselines unexpectedly add "glasses" when aging the face. To increase the task difficulty, edit real images with multiple attribute changes simultaneously. Results are summarized in Fig. which highlights the controllability of our edits.
We measure the changing degrees of the other independent attributes and the image identity when editing attributes with various degrees é. Here é is a predicted changing degree target attribute by R. For thorough comparison, evaluate the performance seg-
Quantitative evaluation:

---

Published as a conference paper at ICLR 2021
Smile
Hair color
Smile
Hair color
(0, [3]
(3, .6]
(.,, .9]
(0, 33
(3, 6]
(.6,
.9]
(0, .3]
(3,, .6]
(.6, .9]
.202
.204
.224
.256
.272
.277
.299
.318
.329
Shen et al.
5e-2
6e-2
4e-2
= le-2
2e-2
2e-2
5e-
+ le-2
3e-2
.115
.211
.277
.162
.166
.177
.155
.220
.284
Voynov et al.
6c-3
4e-2
= 7e-3
+ le-2
3e-2
te-3
i 4e-
+ 4e-
2e-2
.085
.084
.098
.075
.083
.084
.088
.111
.134
Ours
4e-2
3e-3
4e-3
+ 7e-4
Se-3
5e-3
+ 7e-3
+ 4e-
+ 4e-3
Table I: Quantitative evaluation of numerical changes the other semantically independent attributes (lower is better). (first row) edited attributes; (second row) the absolute changing range of the edited attributes; (bottom three rows) averages (up) and standard deviations (down) in each row.
Smile
Hair color
Smile + Hair color
(0, 3]
(.3, .6] (.6, .9]
(0, .3
(3,,
(.6,
(0, .3]
(.3, .6]
(.6, .9]
.918
.916
.907
.887
.877
.874
.877
.819
.801
Shen et al.
1. 4e-3
+ le-3
5e-3
+ 3e-2
4e-2
+. 4c-2
1 6e-3
+ 3e-2
+ 4e-2
.979
.896
.869
.955
.909
.904
.940
.829
.811
Voynov al.
2e-3
+ 4e-3
= 9e-3
+ 6e-2
+ 4e-2
+ 3e-2
+ 9e:3
+ 4e-2
+ 3e-2
.995
.994
.992
.993
.993
.993
.992
.986
.984
Ours
7e
- le-3
= 9e-4
+ Se.4
t le-4
- 2e-4
+ 9
+ 2e-3
t 4e-4
Table 2: Quantitative evaluation on image identity preservation (higher is better). Notation is identical Tab.
ments according to the absolute value of é, i.e., [é] in the range of (0, 0.3), (0.3, 0.6] and (0.6, 0.9]. Evaluation "smile," "hair color," and "smile+hair color" attributes are shown in Tab. [and Tab. The "hair color" attribute includes "blond" and "black" colors where we average the results on both cases. For multiple attribute editing, "smile hair color," evaluate the case that both the two tar- geted attributes change within the | range. We use around 1k original images, generate 10k edited images with regard to each target attribute, and repeat the experiment times. The averaged results and the standard deviations presented in Tab. T and Tab. 2 suggest that our model outperforms the baselines with regard to disentanglement and image identity preservation.
Manipulation results on PGGAN: For PGGAN, we use local transformations parameterized by three-layer perceptrons with leaky ReLU activations. We show the detailed MLP architecture in Appendix B.1. Editing results in Fig. suggest that the proposed approach is able to continuously edit faces with various semantic attributes. We also study the differences between local and global directions in Fig. IT Particularly, global T (in Fig. II (a)) is matrix where each column is given by d, For a local (in Fig. II (b)) d: is given by a fully-conneetee
layer with identical input and output dimensions. Fig. 11 (c) and (d) the MLP components with the aforementioned structure.
We use Grey to train in (c), while the total loss used for (d). The results in Fig. [T indicate that local T, with either linear MLP structure works well for face attribute edits. Moreover, our loss L stabilizes the training process of T.
(a) Black hair
(b) + Smile
Figure? 7: Continuous editing results on PGGAN KKrras et al. 2017) with the "black hair" (left) and the "smile" (right) attribute.
8

---

Published as a conference paper at ICLR 2021
Smile:
Original
(A) Global
(b) Local
(G) Local T MLP Lreg
(d) Local T MIP+
+
Lincar
Figure 8: Comparison of global and local transformations on PGGAN. Original synthetic images (col.1); the edited results on enhancing "smile" (col.2-5): (a) global T, (b) local T, with linear- layer direction functions, (c) local T with MLP direction functions trained via Lreg loss only, and (d) local with ML. direction functions training via the entire loss
CONCLUSION
We propose effective approach to semantically edit images by transferring latent vectors towards meaningful latent space directions. The proposed method enables continuous image manipulations with respect to various attributes. Extensive experiments highlight that the model achieves state-of- the-art performance for targeted image manipulation.
Acknowledgements: This work is supported in part by NSF under Grant No. 1718221, 2008387, 1934986 and MRI #1725729, NIFA award 2020-67021-32799 UIUC, Samsung, Amazon, 3M, and Cisco Systems Inc. (Gift Award CG 1377144). We thank Cisco for access the Arcetri cluster. We thank Amazon for EC2 credits.
REFERENCES
Rameen Abdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In ICCV, 2019.
Binod Bhattarai and Tae-Kyun Kim. Inducing optimal attribute representations for conditional gans. ECCV, 2020.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv, 2018
Q. Cao, L. Shen, W. Xie, al. Vggface2: dataset for recognising faces across pose and age. In IEEE FG, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, al. Stargan: Unified generative adversarial networks for multi-domain image-to-image translation. In CVPR, 2018.
Yunjey Choi, Youngjung Uh, Jaejun Yoo, et Stargan v2: Diverse image synthesis for multiple domains. CVPR, 2020.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, et al. Generative adversarial nets. In NewrIPS, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, et al. Ganspace: Discovering interpretable gan controls. arXiv, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In CVPR, 2016.
Phillip Isola, Jun- Yan Zhu, Tinghui Zhou, et al. teaggeto--image translation with conditional adver- sarial networks. In CVPR, 2017.

---

Published as a conference paper at ICLR 2021
Ali Jahanian, Lucy Chai, and Phillip Isola. On the" steerbtrrabllity" of generative adversarial networks. ICLR, 2019.
Alexia Marttnnnaa The relativistic discriminator: key element missing from standard gan. ICLR, 2019.
Tero Karras, Timo Aila, Samuli Laine, et al. Progressive growing of gans for improved quality, stability, and variation. arXiv, 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In CVPR, 2019a.
Tero Karras, Samuli Laine, Miika Aittala, et Analyzing and improving the image quality of stylegan. arXiv, 2019b.
Marek Kowalski, Stephan J Garbin, Virginia Estellers, Tadas Baltrušaitis, Matthew Johnson, and Jamie Shotton. Config: Controllable neural face image generation. /CML, 2020.
Pierre- Yves Laffont, Zhile Ren, Xiaofeng Tao, al. Transient attributes for high-level understanding and editing of outdoor scenes. TOG, 2014.
H.Y. Lee, H.Y. Tseng, Q. Mao, et al. Drit++: Diverse image-to-image translation via disentangled representations. IJCV, 2020.
Bowen Li, Xiaojuan Qi, Thomas Lukasiewicz, and Others. Controllable text-to-imante generation. In NeurlPS, 2019.
Y. Li, C. Fang, J. Yang, et al. Universal style transfer via feature transforms. In NeurlPS, 2017.
Ziwei Liu, Ping Luo, Xiaogang Wang, et al. Large-scale celebfaces attributes (celeba) dataset. 2018.
Luan, S. Paris, and E. andothers Shechtman. Deep photo style transfer. In PCVPR, 2017.
Taesung Park, Ming-Ye Yu Liu, Ting-Chun Wang, and Jun- Yan Zhu. Semantic image synthesis with spatially-adaptive normalization. In CVPR, 2019.
Antoine Plumerault, Hervé Le Borgne, and Céline Hudelot. Controlling generative models with continuous factors of variations. ICLR, 2020.
Olga Russakovsky, Jia Deng, Hao Su, et Imagenet large scale visual recognition challenge. ICCV, 2015.
Yujun Shen and Bolei Zhou. Closed-form factorization of latent semantics in gans. CVPR, 2021.
Yujun Shen, Jinjin Gu, Xiaoou Tang, et al. Interpreting the latent space of gans for semantic face editing. 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv, 2014.
Yuri Viazovetskyi, Vladimir Ivashkin, and Evgeny Kashin. Stylegan2 distillation for feed-forward image manipulation. arXiv, 2020.
Andrey Voynov and Artem Babenko. Unsupervised discovery of interpretable directions in the gan latent space. ICML, 2020.
Sheng-Yu Yu Wang, Oliver Wang, Richard Zhang, et al. Con-generated images are surprisingly easy to spot... for now. CVPR, 2019a.
Sheng- Yu Wang. Oliver Wang, et al. Detecting photoshopped faces by scripting photoshop. In ICCV, 2019b.
Ting Chun Wang, Ming- Yu Liu, Jun- Yan Zhu, et al. High-resolution image synthesis and semantic manipulation with conditional gans. In CVPR, 2018.
10

---

Published as a conference paper at ICLR 2021
Po- Wei Wu, Yu-Jing Lin, Che-Han Chang, al. Relgan: Multi-domain image-to-image translation via relative attributes. In ICCV, 2019.
Han Zhang, lan Goodfellow, Dimitris Metaxas, et al. Self-attention generative adversarial networks. PMLR, 2018.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, al. Places: 10 million image database for scene recognition. TPAMI, 2017.
Jun- Yan Zhu, Taesung Park, Phillip Isola, et al. Unpaired image-to-ine imaae translation using cycle- consistent adversarial networks. In ICCV, 2017a.
Jun- Jun-Yan Zhu, Richard Zhang, Deepak Pathak, et al. Toward multimodal image-to-image translation. In NeurPPS, 7b.

---

Published as a conference paper at ICLR 2021
APPENDIX
A.
ALGORITHM
The overall procedure of the proposed method in Algorithm T example.
taking local transformation T as an
Algorithm Training Procedure
Input: A pre-trained GAN with G. D, and input noise distribution 2; pre-trained regressor R for predictions on N attributes; an initialized T;; max iteration number M.
for iteration m L,...,.u. .M do
2: Sample random noise ~ Z, and e
3: Compute internal attributes for a synthetic image, i.e., a R(G(z))
4:
Compute the actual shift value 8 = CLIP(a + E, (0. 1)) a
5:
Compute the transformed latent vector = = + Compute attribute predictions 6' R(1')
6:
Compute I' = G(z'),
7:
8:
Compute the loss C
9: Update T
10: end for
Return: T
B.
METHOD DETAILS
B. METHOD ON PGGAN
We apply a multi-layer perceptron (MLP) to parameterize latent space path on PGGAN, Fig. shows the MLP architecture that we use.
2
linear, 512
leaky ReLU (0.2)
linear, 1024
leaky ReLU (0.2)
linear, 512
d;
Figure 9: Example MLP architecture PGGAN.
In practice, we find that normalizing the direction vector d, helps preserve image identity during manipulation on PGGAN. Formally, the transformed latent vector can be written
d,
z + X5
(5)
| where À is weight adjusting the direction scale. We set A 3.
12

---

Published as a conference paper at ICLR 2021
C. EXPERIMENTAL DETAILS
C.I THE EXPECTATION IN LuFG
We define the regression loss Lreg as
Lrgg Ez~z,ew,,1 1-0pl-0'lopp 1....e.,,llalloee log (1-à') (1-a))
(6)
where a' is from the distribution generated by 2 and e. We now discuss how we derive the distri- bution of a'. To begin, sampled from E and is sampled from D.. The other variables generated by and e:
z' + Te
a' R(G(z)) +e
á' R(G(z')).
As a result, the distribution of a' is dependent on 2 and Dz.
C.2 UNSUPERVISED LATENT-SPACE EDITS OF GANS
Voynov & Babenko 2020) learn matrix where each column is direction. Yet. Voynov & Babenko (2020) require a human to interpret the learnt directions. To avoid bias during the selection of directions in our comparison,
the pre-trained attribute regressor automatically identify the most significant directions. Concretely, we first generate 100 images and edit them with target attributes of various degrees. Next, we use the regressor I the edited images to predict attributes. We choose for an attribute edit the direction on which all the edited images have the overall highest changing response on the target attribute.
D. ADDITIONAL ANALYSIS
D.I GAN INVERSION PERFORMANCE
We use an optimization-bant based GAN inversion method (Abdal et al.] 2019) find optimal latent vectors that can best reconstruct the real images via the generator. To examine the effect of GAN inversion performance on our approach, we terminated the GAN inversion approach at different training steps, i..., 500 and 4,000 iterations. We show averaged reconstruction MSE loss in Tab, 3 In this case, we reconstructed 20 real face images and edited their "Smile" and the "Blond hair attribute with 10 different degrees e, i.e., 200 images in total for each attribute editing. Quantitative evaluation on image identity preservation and numerical changes on the other semantically indepen- dent attributes for the reconstructed face images that the performance of the GAN inversion method affects our method to some degree. Visualized inversion and editing results are shown in Fig. 10 The qualitative results suggest that our method still works remarkably well on the worse inversion image.
given in Tab. The results in Tab. suggest
Table 3: Averaged MSE loss of reconstructing 20 real face images. The GAN inversion method (Abdal et al., 2019) was trained and terminated at 4k and 500 iterations with averaged MSE loss in the right column.
Training iterations
MSE
4k iters
1657.40
500 iters
3096.75
D.2 ABLATION STUDY
We conduct an ablation study with single- and joint-distribution training strategies in our method. Specifically, single-distribution sampling refers to learn one attribute direction at a time. In contrast, joint-distribution training means to train multiple attribute directions simultaneously. Fig. II shows the visualized results with the two training strategies on the face and the scene dataset. We observe that the model with the single-distribution training strategy generates more unexpected changes as the manipulation degree is getting large, e.g., darker scene colors by the model trained for a single attribute, shown in Fig. 11 (b).
13

---

Published as a conference paper at ICLR 2021
Table Quantitative evaluation of identity preservation (ID) and attribute changes (Attr). We edited two attributes for the real face images, i.c., "Smile" (col.2-7) and "Blond hair" (col.8-13).
Smile
Blond hair
ID
Attr (4)
ID (t)
Attr (L)
E
(0,
(.3,
.6]
(.6, .9]
(0, .3]
(.3, .6]
(.6, .]
(0, .3
(33, .6]
(.6, .]
(0, 33]
(.3,.6]
(.6, .9]
4k iters
.997
.994
.96
.077
.108
.147
.88
.979
.94
.1
.112
.16
500 iters
.996
.99
.922
.087
.138
.165
.978
.936
.914
.122
.158
.185
Smile
+ Smile
500 iters
4k iters
Original
Inversion
Editing results
Figure 10: Visual comparisons of "Smile" editing with different reconstructed images: (col.1) Original image; (col.2) reconstructed images with 500 (top) and 4k (bottom) inversion optimization steps; (col.3-6) results of editing the "Smile" attribute.
E. E. SCENE IMAGE EDITS ON STYLEGAN2
ADDITIONAL RESULTS
We show more results of our method on continuous scene image edits in Fig. 12 tically edited attributes include "clouds" (Fig. 12p. "brightness" (Fig. 13 "summer" (Fig. 15
15
The seman-
"snow" (Fig. 14), and
F. BROADER IMPACT
From an application perspective, our method effective and efficient with regard to image manip- ulation and photo-realism, which hope will contribute to 2D and 3D controllable image editing tasks. Moreover, usage of deep nets to learn mappings between spaces is still not well understood, e.g., mappings between low-dimensional space and image space classification and generation tasks. We hope our method provides inspiration for representation learning and a first step for new view with regard to deep net interpretability.
Obviously, we are aware of the dangers of automated image manipulation. Similar to deepfake tasks whose aim is to produce fabricated images and videos that appear to real, improper use image manipulation approaches might raise negative issues with regard to information security, property, etc. Beyond that, edited image detection techniques Wang et al. (2019ajb) are proposed recently to avoid the aforementioned issues, which promotes growth in both domains.
14

---

Published as a conference paper at ICLR 2021
Young
Sugle
Single
Josa
Joiis
Ediiig (a) Visual comparisons single- and joint-distribution sampling on the face dataset
Original
Original
Editing
(b) Visual comparisons of single- and joint-distribttion sampling the scene dataset
Figure 1I: Visual comparisons of single- and joint-distribution training. Single-distribution training refers to train one attribute direction at time, while cint-distribution training means to train multiple attribute directions simultancously.
Figure 12: Additional results. Continuous image edits using StyleGAN2 KKrras et al. the "clouds" attribute.
20199
Figure 13: Additional results. Continuous image edits using StyleGAN2 Karras et al. 2019b on the "brightness" attribute.
Figure 14: Additional results. Continuous image edits using StyleGAN2 Karras et al. 2019b on
the "snow" attribute.
15

---

Published as a conference paper at ICLR 2021
Figure 15: Additional results. Continuous image edits using StyleGAN2 KKrras et al. 201 (01b) on
the "summer" attribute.
16


Reviews:
[{"review_id": "vBJy1XGzom", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presented a latent-space editing framework for semantic image manipulation. The idea is interesting and plausible, and experiments also show its effectiveness. However, I still have some concerns:\n\n1. The pre-training of regressor R should be given in more details. \n2. The results of the proposed method heavily depends on the GAN inversion. However, it can be seen from Fig. 6 that some details are still lost by GAN inversion.\n3. In term of simultaneous multiple attribute transformations, it is interesting to refer to the following reference to guarantee the latent consistency in the proposed framework:\n[r1] Inducing Optimal Attribute Representations for Conditional GANs, ECCV 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "iVgoiFh7UTf", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "#### Summary:\nThe paper proposes a new simple, yet powerful and alternative method of editing the semantic attributes of images generated using pre-trained GAN models as well as a pre-trained regressors. The approach allows for the manipulation of single or multiple various image attributes, while preserving the identity of the original image in contrast to the baseline method of Shen et. al 2019. The method focuses on the manipulation of the latent space, in contrast to the popular image space editing methods. \n\nThe paper is easy to read and understand. The authors have presented their method and analysis which are clear and understandable, supported quite well with the provided examples and figures.\n\n#### Strengths:\nExperiments are conducted on various datasets and compared to various methods - supervised and unsupervised\nTheir method is able to consistently preserve the image content (in case of scenes) and identity (in case of faces) \nShowing inversion results along the transformation shows the robustness and tractable nature of their method.  \n\n#### Weakness:\nIt is not entirely clear how the local transformation are discovered for different z values independently, or rather what the difference is in discovering the global transformation vs discovering the local transformation\n\n\n#### Questions to the Authors:\n1. In comparison with the Voynov & Babenko model, how many of the directions that their model was able to find did you look at?\n2. What is the accuracy / results of the performance metric of the different used pre-trained regressors? Given that the perceptual losses were used while training, the regressors’ performance values could shed some light in this direction.  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "ho7cKZcVYQ1", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a new approach for the semantic image editing task by allowing the controllable transformation on the latent space. Authors proposed to integrate an attribute regression network for training the transformation functions. The local transformation T is learned from a simple MLP conditioned on the latent vector z. Two outputs of the regression module for the original latent vector z and the transformed one z+T*epsilon are used to minimize the cross-entropy loss. Experiments validate the effectiveness of the proposed method in terms of manipulation quality.\n\n* Pros\n1) Local transformation function T would be a more appropriate choice for transforming the latent vector.\n\n* Cons\n1) The manuscript contains lots of vague parts.\n- Generator G, discriminator D, and regressor R are all pre-trained. How did you obtain the pre-trained regressor R?\n- ‘Joint-distribution sampling and training’ part in Section 3.2 is hard to understand. More details would be needed, including its practical implementation and pros over the separate binary sampling.\n- This method requires using inversion results. Does the accuracy of the inversion results affect the performance of the proposed method?\n\n2) An ablation study on the joint-distribution sampling would be needed to validate their effectiveness.\n\n3) It seems that editing control (e.g., +Night or +Snow) is related to how to sample the epsilon vector as shown in Figure 2. Did you use the sampling strategy that is adaptive with respect to the given editing control? For instance, when editing images into a night scene, how do the method sample the epsilon vector? It seems that there is no such functionality in Algorithm 1. Please revise the manuscript by specifying this part.\n\n4) Table 2 seems to measure the cosine similarity, while Table 1 evaluates the change of the attributes. How did you calculate the change of the attributes?\n\n5) The results in Figure 8 (a) are based on the global transformation T with the proposed losses. Did you obtain these results using MLP direction functions or simple linear-layer direction functions?", "extra_scores": {}, "extra_text": {}}, {"review_id": "0kg4ffQ3mr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "In this paper, the authors propose an image attribute editing method by manipulating the GAN latent vector. Specifically, this paper uses a pre-trained GAN to synthesize images, a pre-trained regressor to get the image attributes, and trains a network T to find meaningful latent-space directions. It then edits image attributes by modifying the input latent vector, described as z' = z + T(z)ε. The experimental results show that the proposed method performs better than other selected methods to some degree.\n\nStrengths: \n1) The idea of controllable editing is intuitive and interesting.\n2) Quantitative results on face datasets (Tab. 1 and Tab. 2) show its superiority over other selected methods.\n3) Visualization results on both the natural scene and face datasets show its effectiveness on the parts to be edited (e.g. clouds for “remove clouds” and mouth for “add smile”).\n\nWeaknesses: \n1) Visualization results on natural scene datasets show that its ability to maintain image identity needs to be improved. For example, the image identities in Fig. 3 are changed during the editing, like the mountains' shapes in row 1 and trees' shapes in row 3.\n2) The authors claim that their edits are disentangled, but the visualization results on face datasets don’t support this point very well. For example, the baselines unexpectedly add \"glasses\" when aging the face in Fig. 4. However, the proposed method also adds glasses on the man's right eye (figure on the bottom right corner). \n3) Lack of comparisons of continuous image editing with other methods.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: All the reviewers rate the paper above the bar. They like the experiment results and think the proposed latent space editing approach makes intuitive sense. While several weakness points were raised, including a lack of continuous editing comparison and sometimes vague descriptions, they were not considered major to reject the paper. After consolidating the reviews and rebuttal, the AC agrees with the reviewer assessment and recommends accepting the paper.
Decision: Accept (Poster)