Look Ma, No GANs! Image Transformation with ModifAE

1 INTRODUCTION . Image modification is a difficult task that requires changing some aspects of an image while keeping others static . These changes can include modifying the class of an image , often called image to image translation Isola et al . ( 2016 ) , or they can refer to modifying a continuous trait in an image , which we will refer to as continuous image modification . These image modification tasks are often performed on datasets like CelebA because human faces have many easily recognizable traits Liu et al . ( 2015 ) . Initially , much of this work required paired examples or was done with hand-tailored methods , Kemelmacher-Shlizerman ( 2016 ) ; Laffont et al . ( 2014 ) ; Hertzmann et al . ( 2001 ) ; Khosla et al . ( 2013 ) ; Duong et al . ( 2016 ) . This can be time consuming and can struggle with generalization . More general approaches have been introduced , but only for image to image translation . Some of these newer methods can modify continuous traits , such as the apparent age of a face , but they can only achieve this by breaking images into binary categories such as “ young ” and “ old ” or multiple age groups , such as 0-18 , 19-29 , and etc Upchurch et al . ( 2017 ) ; Gardner et al . ( 2015 ) ; Liu et al . ( 2017a ) ; Antipov et al . ( 2017 ) . Another level of complexity is introduced when dealing with unpaired data Zhu et al . ( 2017 ) . That is , given unrelated images from different domains , translate an image from one class to another . Without pairs of images in each domain , the problem can not be solved as a simply mapping an input image to its paired output image . The unpaired image to image translation task has been been tackled as a latent space traversal problem . For example , Upchurch et al . ( 2017 ) use deep features from a pretrained classifier to form a latent space in which different classes of images are loosely clustered . Then , by projecting an image into the latent space , interpolating the embedding towards the mean of a different class , and projecting back into the image space , class modifications can be achieved Upchurch et al . ( 2017 ) ; Gardner et al . ( 2015 ) . However , these methods require multiple steps and use hand-tailored linear interpolations through a latent space . These methods take more time to design and do not generalize well to new domains . A more desirable method of modifying an image would do so based purely on learned weights , using a single forward pass of a neural network . Over the past four years , generative adversarial networks Goodfellow et al . ( 2014 ) have become the most popular tool for the image to image translation task , allowing for the image modification to be fully controlled by the generator network in one forward pass . They use adversarial training to ensure results are believable and often use cycle training to ensure that identifying traits from the original image are preserved Zhu et al . ( 2017 ) ; Isola et al . ( 2016 ) ; Liu et al . ( 2017b ) ; Choi et al . ( 2017 ) ; Creswell et al . ( 2017 ) ; Liu et al . ( 2017a ) ; Larsen et al . ( 2016 ) ; Perarnau et al . ( 2016 ) . However , by definition , GANs require at least two networks to be trained . These networks also need to remain in equilibrium throughout the training process to prevent modal collapse Berthelot et al . ( 2017 ) . A single-network image modification method would be better since it would have fewer parameters and train in a faster and simpler manner . Additionally , none of these methods are built to work on continuous trait modification . They currently only use binary classes ( old vs. young , or beard vs. clean shave ) . While it is possible to linearly interpolate between these binary classes , traits may not scale linearly between these classifications , e.g . aging between 10 to 15 years old would require a different change than aging from 30 to 35 years old . As mentioned above , existing techniques either use hand-tailored methods , rely on multiple steps like projection and linear interpolation , or require two networks for training . To fill this gap , we introduce ModifAE : a general image modification method which works without GANs or hand-tailored features . It also works for continuous traits . To our knowledge , ModifAE is the first standalone neural network which can perform general image modification in a single forward pass . 2 RELATED WORK . 2.1 AUTOENCODERS . One method of generating efficient encodings of images is the autoencoderHinton and Salakhutdinov ( 2006 ) ; Cottrell et al . ( 1987 ) . These networks take in a high dimensional input , reduce to a lower dimensional representation , and then decode back to the original input . A bottleneck occurs in the middle of the autoencoder network , creating a latent space with high level features about the input . When the network is linear and has one hidden layer , they implement a version of PCA P. Baldi ( 1988 ) . Additional hidden layers result in nonlinear encodings Hinton and Salakhutdinov ( 2006 ) ; DeMers and Cottrell ( 1993 ) . More recently , variational autoencoders Kingma and Welling ( 2013 ) have been introduced . These networks create more continuous latent spaces where linear interpolations can be performed smoothly . 2.2 DEEP FEATURE INTERPOLATION . Some general methods of image to image translation are based on stepped linear traversals in a learned latent space . For example , Deep Feature Interpolation ( DFI ) Upchurch et al . ( 2017 ) relies on linear interpolation of the latent representation of an input image . DFI involves training a deep convolutional neural network Simonyan and Zisserman ( 2014 ) ; Krizhevsky et al . ( 2012 ) to classify images , then the network performs an image traversal based on a statically defined procedure , which is not learned by the network . Since DFI is designed to interpolate between means of class clusters , it can not be used for continuous trait modification . Also , all modifications require a forward pass of the network , an interpolation in the latent space , then deconvolutions to arrive back at an image . 2.3 CONDITIONAL GENERATIVE ADVERSARIAL NETWORKS . Conditional and Controllable GANs Mirza and Osindero ( 2014 ) ; Lee and Seok ( 2017 ) ; Yin et al . ( 2017 ) made it possible to generate images conditioned on certain traits . The generator is trained with an image and trait input , then the discriminator gives feedback based on whether the image appears real or fake and whether it appears to belong to the target class Lee and Seok ( 2017 ) . However , such networks can still struggle with preserving identifying traits of the original image . Zhu et al . ( 2017 ) proposed CycleGAN , a model for image to image translation which addresses the challenge of maintaining identifying traits in the modified image . It enforces cycle consistency by modifying an image with a different target class , then modifying it back to its original classification . After the modification cycle , pixel-wise autoencoding loss is calculated for the original and reconstructed images . However , CycleGAN is incapable of modifying multiple traits with a single model . Recently , StarGAN Choi et al . ( 2017 ) was introduced and beat other methods on image to image translation . It uses cycle training to preserve identity traits and can also modify multiple classes in the same forward pass . Choi et al . ( 2017 ) found that their model produced superior quality modifications when simultaneously supervising multiple traits , achieving state-of-the-art performance . These networks can perform image to image translations on multiple traits in a single forward pass of the generator , while maintaining identifying traits of the original image . However , as mentioned in the introduction , they are currently all built to modify binary class traits . They also all require training multiple networks in equilibrium , which takes more parameters and more time . ModifAE offers a solution to both of these issues . 3 METHODS . ModifAE is a single network , trained exclusively on an autoencoding task , that implicitly learns to modify perceived traits in images ( illustrated in Figure 2 ) . In this section , we discuss the dataset , ModifAE architecture , training procedure , and how ModifAE works . 3.1 CONSTRUCTING A CONTINUOUS TRAIT DATASET . To evaluate ModifAE on continuous continuous face traits , we need a large dataset . We use images from the CelebA datasetLiu et al . ( 2015 ) , which consists of over 200,000 images of celebrities ( including , ironically , B-list celebrities ! ) . The images in CelebA are annotated with 40 facial traits , such as “ mustache ” , “ eyeglasses ” , “ hat ” , etc . Given that these are binary traits , rather than continuous ones , these labels are not appropriate for continuous trat testing . To generate continuous traits of these faces , we use a system that was trained to “ see people like people ” Song et al . ( 2017 ) to rate faces from the CelebA dataset on continuous traits . Bainbridge et al . ( 2013 ) had workers judge 40 social traits of 2,222 faces from the the MIT 10k US Adult Faces Database . These traits include trustworthiness , attractiveness , responsibility , aggressiveness , etc . The 2,222 labeled faces from the MIT Faces Database are not enough to train a robust generative model . Hence , we used a trained regression network to judge the social traits of 190,000 faces from the CelebA dataset Liu et al . ( 2015 ) ; Song et al . ( 2017 ) and use the predicted trait ratings to train ModifAE . Example faces and their predicted ratings are shown in Figure 1 . To verify the effectiveness of this artificial dataset , we ran an AMT experiment checking how the predicted values align with human judgments in four attributes : aggressive , responsible , trustworthy and emotional . Humans have high agreement in these traits . For each trait , we picked 40 images rated highest by the prediction network Song et al . ( 2017 ) , and 40 images rated lowest . Next , we form 40 pairs by picking one image from the high rating group , and one image from the low group . We then ask AMT workers which face better exemplifies the corresponding trait in the each pair , ( e.g . which one looks more trustworthy in this pair ) . Each trait ’ s 40 pairs are evaluated by 30 workers . Then , we calculated the overall likelihood ( among all the workers and among all the pairs ) that the face of higher predicted score is chosen by human subjects for each attribute . As can be seen in Table 1 , all the attributes predicted by the regression network Song et al . ( 2017 ) align well with human judgments . Therefore , we consider the predicted scores as being roughly equivalent to human judgments of those faces .

Summary: Overview and contributions: The authors propose the ModifAE model that is based on an autoencoder neural network for continuous trait image modifications. ModifAE requires fewer parameters and less time to train than existing generative models. The authors also present experiments to show that ModifAE produces more convincing and more consistent continuous face trait modifications than the current baselines.