CONTROLLING THE MEMORABILITY OF REAL AND UNREAL FACE IMAGES

1 INTRODUCTION . In our everyday life , we are exposed to many pictures of scenes , objects and faces . Research has shown that all images do not have the same likelihood to be recalled later ( Isola et al. , 2011 ; Bainbridge et al. , 2013 ; Isola et al. , 2014 ) . Although different people have different abilities in memorizing visual contents ( image or video ) , it has been shown that memorability is an intrinsic feature of an image and it is consistent across different observers ( Isola et al. , 2011 ; Bainbridge et al. , 2013 ; Isola et al. , 2014 ) . In other words , memorability of an image is an attribute of that image which can be measured , predicted or manipulated ( Isola et al. , 2011 ) . So far there have been several studies that have attempted to understand , predict , and even modify image memorability . ( Khosla et al. , 2013a ; 2015 ; Goetschalckx & Wagemans , 2019 ; Fajtl et al . ; Needell & Bainbridge , 2021 ; Squalli-Houssaini et al. , 2018 ; Almog et al . ) , and a few work attempted to modify image memorability ( Khosla et al. , 2013b ; Siarohin et al. , 2017 ; Sidorov , 2019 ; Goetschalckx et al. , 2019 ) . However , for practical applications ( e.g. , education and advertisement ) , it is most important to have methods for memorability modifications . Moreover , such approaches will help understanding what constitute as image memorability , i.e . ” What makes an image memorable ? ” . This work proposes a new framework based on Generative Adversarial Networks ( GANs ) ( Goodfellow et al. , 2014 ) for modifying face memorability as a facial attribute . Older approaches on modifying face memorability manually used face features ( Khosla et al. , 2013b ) , such as SIFT ( Lowe , 2004 ) , HOG2x2 ( Dalal & Triggs , 2005 ) , and Local Binary Pattern ( LBP ) ( Ojala et al. , 2002 ) . Recently , GANs have been used to modify the memorability of images . ( Goetschalckx et al. , 2019 ; Sidorov , 2019 ) . Goetschalckx et al . ( 2019 ) leverage latent vector modification to change the memorability of the fake food , scenes , and animal images generated by BigGAN ( Brock et al. , 2018 ) . This memorability modification affects several attributes of the image , such as size , color , and shape . In our work , we aim to modify memorability of face images of real people while keeping their identity , consequently , their method can not be used here . The largest dataset of the human faces with their memorability annotations is US 10k Face database ( Bainbridge et al. , 2013 ) , which includes 2222 face images with their memorability scores acquired from human observers in an experiment . StyleGANs are the state-of-the-art models for generating real-looking faces . Our utilization of StyleGANs is required to create a dataset of realistic-looking faces . Not only that , for modifying the memorability of real faces , we need StyleGANs to reconstruct real faces with high accuracy . To date , StyleGANs are the state-of-the-art models in reconstructing real-face images . StyleGANs provide an extended latent space which we leverage to derive a more accurate memorability hyperplane . Also , the face attributes of StyleGANs are especially disentangled in comparison to other GANs , which is required to accurately modify faces for memorability and study the attributes contributing to this . For this , we employed pre-trained StyleGAN1 ( Karras et al. , 2019 ) and StyleGAN2 ( Karras et al. , 2020 ) on the FFHQ dataset ( Karras et al. , 2019 ) to generate 100k fake faces . Next , we adopted computational memorability models which we trained on the US 10k Face Database , to predict the memorability of the generated face images and organized them into faces with high or low memorability . Inspired by Shen et al . ( 2020 ) , we found a hyperplane in latent subspace to separate the highly-memorable and low-memorable faces . We showed that both latent space and extended latent space can be used for finding the separating hyperplane . After finding the hyperplane , we moved the latent vector of each image , in the positive or negative direction of the normal vector of that hyperplane and changed the distance of the latent vector from the separating hyperplane to manipulate the memorability of that image . We name the normal vector of this hyperplane , memorability modification vector . With this proposed approach , we could control the amount of change in memorability by using different weights for the memorability modification vector . In contrast to the method proposed by Sidorov ( 2019 ) , our method does not require training another auxiliary network for modifying face memorability and the amount of change in memorability is controlled by a hyperparameter . Since different hyperplanes for different facial attributes in StyleGAN latent space ( Shen et al. , 2020 ; Härkönen et al. , 2020 ) can be found , our method can be used to modify the memorability of the images conditionally . For example , we are able to change the memorability of the face while maintaining the length of the hair and the existence of eyeglasses . For this , we first find the corresponding hyperplanes for these attributes , and then leverage projection to those subspaces to have the desired attributes fixed while changing memorability . StyleGAN produces high quality real-looking images which are near impossible to differentiate from real images . To make sure our memorabilitymodified faces still look real , we considered the Frechet Inception Distance ( FID ) ( Heusel et al. , 2017 ) and Kernel Inception Distance ( KID ) ( Bińkowski et al. , 2018 ) scores of the generated faces from the StyleGAN as the baseline . Then by calculating the FID score and KID score of our memorability modified images , we showed that our modified faces still looked real . For real faces , we first embedded the face images into GAN latent space using Image2Style ( Abdal et al. , 2019 ) and the method provided by Karras et al . ( 2020 ) . After finding the image latent vector , we modified the real face memorability in the same way previously explained for synthetic faces . Figure 1 illustrates the general idea of our approach . Finally , we examined how different layers of extended latent space of StyleGAN affect the image of each face and its memorability . 1.1 RELATED WORKS . Image Memorability . People have different capabilities in memorizing different visual events ( Hunt et al. , 1981 ) . In spite of these differences , through a series of experiments , Isola et al . ( 2011 ) showed that people consistently remember some images and forget others . They designed an online memory game experiment and recruited a large number of participants through Amazon Mechanical Turk . In this experiment , participants observed a series of images presented in a sequence and were tasked to detect repetitive images in the sequence . Then Isola et al . ( 2011 ) measured a memorability score for each image , which corresponded to the rate of people remembering an image after single exposure to that image in the sequence . Khosla et al . ( 2015 ) created the largest annotated image memorability dataset ( LaMem ) , which consists of 60,000 images , mostly objects , scenes , and animals . Using this dataset , they introduced the first deep model for predicting image memorability . This model uses AlexNet ( Krizhevsky et al. , 2012 ) as its backbone architecture . Moreover , Needell & Bainbridge ( 2021 ) introduced new architectures based on residual networks ( He et al. , 2016 ) to improve the performance of memorability prediction . In addition to these models , Fajtl et al . leveraged Attention Maps to introduce AMNet for predicting memorability . Generative Adversarial Networks ( GANs ) . With the development of Generative Adversarial Networks ( Goodfellow et al. , 2014 ) , we are now capable of generating real-looking synthetic images that are indistinguishable from real images . Generally , these networks are composed of two parts ; a generative and a discriminative network . The goal of the generative network is to generate reallooking images to fool the discriminator and the goal of the discriminator is to learn to distinguish generated images from real images . These two networks are optimized through a minmax game where both sides compete to reach their specified goals . In recent years , there have been huge improvements in this area and many different GANs have been introduced to produce natural-looking images such as Progressive GAN ( Karras et al. , 2017 ) , DCGAN ( Radford et al. , 2015 ) , and CycleGAN ( Zhu et al. , 2017 ) . In this work , we used pre-trained StyleGAN1 ( Karras et al. , 2019 ) and StyleGAN2 ( Karras et al. , 2020 ) on the FFHQ dataset ( Karras et al. , 2019 ) to generate real-looking faces . Modifying Image Memorability . While image memorability modification has many potential applications ( e.g . in education or advertisement ) , it has not been adequately investigated . Khosla et al . ( 2013b ) proposed a pioneering method for changing face memorability . It leveraged Active Appearance Models ( AAMs ( Cootes et al. , 2001 ) ) to represent faces by their shape and appearance . Then the loss function was defined based on the cost of modifying identity , modifying facial attributes , and memorability . As a result , in their method , the identity was fixed . Another approach ( Sidorov , 2019 ) used VAE/GAN ( Larsen et al. , 2016 ) , StarGAN ( Choi et al. , 2018 ) , and AttGAN ( He et al. , 2019 ) and trained them with three memorability levels ( poorly memorable , moderately memorable , and highly memorable ) of faces and modified the memorability of different faces to these three levels only . Additionally , Siarohin et al . ( 2017 ) utilized style transfer to increase the memorability of an input image . However , the added style adversely affected the realness of the modified images , such that it barely could be used in real-world applications . Most recently , Goetschalckx et al . ( 2019 ) trained a transformer network to change the memorability of each generated image by BigGAN ( Brock et al. , 2018 ) through modifying their latent vectors . Their method works on generated images of objects and scenes . 2 METHOD . The overview of our proposed method is depicted schematically in Figure 1 . Below we explain the approach step by step . 2.1 CREATING THE DATASET . For the purpose of analyzing the latent vectors of the GANs and their relation to memorability , we need a large dataset of face images with their memorability scores . The largest dataset available for face images is the 10k US Adult Faces Database ( Bainbridge et al. , 2013 ) . This database contains 10,168 natural human face images and 2,222 of these images are annotated with memorability scores . To create a larger dataset for face images with their corresponding memorability scores , we leveraged StyleGAN1 and StyleGAN2 , which are the state-of-the-art models for creating realisticlooking face images . These models were pre-trained on the FFHQ dataset ( Karras et al. , 2019 ) which consists of 70,000 high-quality face images with 1024 × 1024 resolution with variations in age , gender , and glasses . We created two different datasets with StyleGAN1 and StyleGAN2 . We randomly sampled 100k z ∈ R1×512 from a standard normal distribution with truncation to produce high-quality synthetic face images and saved their mappings in the extended latent space ( R18×512 ) of both GANs .

Summary: This paper proposes to find the hyperplane to separate images with different memorability in the latent-vector spaces of StyleGAN1 and StyleGAN2, then they can move the latent-vectors along the normal vector of the hyperplane to control the memorability of the generated images. A SENet50 trained on the US 10k Face Database with memorability labels is used to find the hyperplane. The experimental results show that the proposed method can change the memorability of images measured by the scores of the SENet50, and this method can be extended to modify real facial images.