CONTROLLING THE MEMORABILITY OF REAL AND UNREAL FACE IMAGES
Authors: Mohammad Younesi
Yalda Mohsenzadeh
Venue: ICLR 2022 Submitted (2022)
Keywords: Memorability
Face Memorability
Face Memorability Modification
StyleGAN
Latent Vector
Image2Style

Abstract: Every day, we are bombarded with many face photographs, whether on social media, television, or smartphones.  From an evolutionary perspective, faces are intended to be remembered, mainly due to survival and personal relevance.  However, all these faces do not have the equal opportunity to stick in our minds. It has been shown that memorability is an intrinsic feature of an image but yet, it’s largely unknown what attributes make the images more memorable.  In this work, we aim to address this question by proposing a fast approach to modify and control the memorability of face images.  In our proposed method, we first find a hyperplane in the latent space of StyleGAN to separate high and low memorable images. We then modify the image memorability (while keeping the identity and other facial features such as age,  emotion,  etc.)  by moving in the positive or negative direction of this hyperplane normal vector.  We further analyzed how different layers of the styleGAN augmented latent space contribute to face memorability. These analyses showed how each individual face attribute makes images more or less memorable.  Most importantly, we evaluated our proposed method for both real and unreal (generated) face images.  The proposed method successfully modifies and controls the memorability of real human faces as well as unreal(generated) faces.  Our proposed method can be employed in photograph editing applications for social media, learning aids, or advertisement purposes.

Under review as a conference paper at ICLR 2022
CONTROLLING THE MEMORABILITY OF REAL AND UN- REAL FACE IMAGES
Anonymous authors Paper under double-blind review
ABSTRACT
Everyday, we are bombarded with many photographs of faces, whether on social media, television, or smartphones. From an evolutionary perspective, faces are intended to be remembered, mainly due to survival and personal relevance. How- ever, all these faces do not have the equal opportunity to stick in our minds. It has been shown that memorability is an intrinsic feature of an image but yet, it largely unknown what attributes make an image more memorable. In this work, we aimed to address this question by proposing a fast approach to modify and control the memorability of face images. In our proposed method, we first found a hyperplane in the latent space of StyleGAN to separate high and low memo- rable images. We then modified the image memorability (while maintaining the identity and other facial features such as age, emotion, etc.) by moving in the pos- itive or negative direction of this hyperplane normal vector. We further analyzed how different layers of the StyleGAN augmented latent space contribute to face memorability. These analyses showed how each individual face attribute makes an image more or less memorable. Most importantly, we evaluated our proposed method for both real and unreal (generated) face images. The proposed method successfully modifies and controls the memorability of real human faces as well as unreal (generated) faces. Our proposed method can be employed in photograph editing applications for social media, learning aids, or advertisement purposes.
INTRODUCTION
In our everyday life, we are exposed to many pictures of scenes, objects and faces. Research has shown that all images do not have the same likelihood to be recalled later (Isola et al. 2011 Bain- bridge et al. 2013: Isola et al. 2014). Although different people have different abilities in memoriz- ing visual contents (image or video), has been shown that memorability an intrinsic feature of an image and it is consistent across different observers (Isola et al.) 2011] Bainbridge et al. 2013 [sola et al., 2014). In other words, memorability of an image is an attribute of that image which can be measured, predicted manipulated (Isola et al., 2011). So far there have been several studies that have attempted to understand, predict, and even modify image memorability. (Khosla et al.) 2013a 2015 Goetschalckx & Wagemans 2019 Fajtl et al.; Needell & Bainbridge 2021 Squuali- et al. Almog et al.), and a few work attempted to modify image memorability (Khosla et al. 2013b] Siarohin et al.J [2017] [idorov 2019; Goetschalckx
al. (019). However, for practical applications (c.g., education and advertisement), it is most important to have methods for memora- bility modifications. Moreover, such approaches will help understanding what constitute as image memorability, i.e. "What makes an image memorable?".
This work proposes a new framework based on Generative Adversarial Networks (GANs) (Good- fellow et al. 2014) for modifying face memorability as facial attribute. Older approaches on mod- ifying face memorability manually used face features (Khosla et al. (013b), such as SIFT (Lowe)
2004 HOG2x2 (Dalal & Triggs 2005), and Local Binary Pattern (LBP) (Ojala et al.] 2002 Re- cently, GANs have been used to modify the memorability
images. (Goetschalckx et al. 2019 Sidorov 2019). Goetschalckx et al. (2019) leverage latent vector modification to change the memo- rability of the fake food, scenes, and animal images generated by BigGAN (Brock et al. 2018). This memorability modification affects several attributes of the image, such size, color, and shape. In our work, we aim to modify memorability of face images of real people while keeping their identity, consequently, their method cannot be used here.

---

Under review as a conference paper at ICLR 2022
The largest dataset of the human faces with their memorability annotations is US 10k Face database (Bainbridge et al. 2013), which includes 2222 face images with their memorability scores acquired from human observers in experiment. StyleGANs are the state-of-the-art models for generating real-looking faces. Our utilization of StyleGANN is required to create a dataset of realistic-looking faces. Not only that, for modifying the memorability of real faces, we need Style- GANs to reconstruct real faces with high accuracy. To date, StyleGANs are the state-of-the-art models in reconstructing real-face images. StyleGANs provide an extended latent space which we leverage to derive a more accurate memorability hyperplane. Also, the face attributes of Style- GANs are especially disentangled in comparison to other GANs, which is required to accurately modify faces for memorability and study the attributes contributing to this. For this, we employed pre-trained StyleGANI (Karras et al.) 2019) and StyleGAN2 (Karras et al. 2020) on the FFHQ dataset (Karras et al. 2019) to generate TOOk fake faces. Next, we adopted computational memo- rability models which we trained on the US 10k Face Database, to predict the memorability of the generated face images and organized them into faces with high or low memorability. Inspired by Shen et al. (2020), we found a hyperplane in latent subspace to separate the memorabe and low-memorable faces. We showed that both latent space and extended latent space can be used for finding the separating hyperplane. After finding the hyperplane, we moved the latent vector of each image, in the positive the distance of the latent vector from the separating hyperplane to manipulate the memorability of that image. We name the normal vector of this hyperplane. memorability modification vector. With this proposed approach, we could control the amount of change in memorability by using different weights for the memorability modification vector. contrast to the method proposed by [Sidorov (219),, our method does not require training another auxiliary network for modifying face memora- bility and the amount of change in memorability is controlled by a hyperparameter.
negative direction of the normal vector of that hyperplane and changed
Since different hyperplanes for different facial attributes in StyleGAN latent space (Shen et al., 2020 [ärkööen et al., 2020) can be found, our method can be used to modify the memorability of the images conditionally. For example, we are able to change the memorability of the face while maintaining the length of the hair and the existence of eyeglasses. For this, we first find the corre- sponding hyperplanes for these attributes, and then leverage projection to those subspaces to have the desired attributes fixed while changing memorability. StyleGAN produces high quality real-looking images which are near impossible differentiate from real images. To make sure our memorability- modified faces still look real, we considered the Frechet Inception Distance (FID) (Heusel et al.) 2017) and Kernel Inception Distance (KID) (Bińkowski et al., 2018) scores of the generated faccs from the StyleGAN as the baseline. Then by calculating the FID score and KID score of our mem- orability modified images, we showed that our modified faces still looked real.
For real faces, we first embedded the face images into GAN latent space using Image2Style (Abdal jt al. 2019) and the method provided by Karras et al. (2020) After finding the image latent vector, we modified the real face memorability in the same way previously explained for synthetic faces. Figure | illustrates the general idea of our approach. Finally, we examined how different layers of extended latent space of StyleGAN affect the image of each face and its memorability.
1.1 RELATED WORKS
Image Memorability. People have different capabilities in memorizing different visual events (Hunt et al. 1981). In spite of these differences, through series of experiments, [soola et al. (2011) showed that people consistently remember some images and forget others. They designed an online memory game experiment and recruited a large number of participants through Amazon Mechanical Turk. In this experiment, participants observed series of images presented in a sequence and were tasked to detect repetitive images in the sequence. Then Isola et al. (20IT) measured memorability score for each image, which corresponded to the rate of people remembering an image after single exposure to that image in the sequence. Khosla et al.) (2015) created the largest annotated image memorability dataset (LaMem), which consists of 60,000 images, mostly objects, scenes, and animals. Using this dataset, they introduced the first deep model for predicting image memorability. This model uses
AlexNet (Krizhevsky et al. 2012) as its backbone architecture. Moreover, Needell & Bainbridge (202)) introduced new architectures based on residual networks (He et al. (016)) to improve the performance of memorability prediction. In addition to these models, et al. leveraged Attention Maps to introduce AMNet for predicting memorability.
2

---

Under review as a conference paper at ICLR 2022
lugh memorable
High cumonide locs
Pace Mansmniilty
menoralle
groupe
Genenbel Face Imsges
Riding the apparting sperilir
Paze Ganaddar
Endasdiag in StyleCion spacn
lyog liteet
or inage
Medine image
Figure 1: The proposed method. We first generate synthesized face images from random latent vectors and predict their memorability scores by face memorability assessor network. Then, di- vide them into high-memorable and low-memorable faces. Using either latent vector extended latent vector subspace we find memorability separating hyperplane in that subspace. The second row shows the proposed framework for modifying the memorability score of face images. Using GAN inversion technique we first map faces to StyleGAN's latent space. Then modify the face memorability by moving the latent vector (or extended latent vector) towards negative or positive direction of memorability discriminating hyperplane obtained in previous step. The modified la- tent vector (or extended latent vector) is fed to the GAN to generate the face image with modified memorability. It is optional to ovalize the face images before feeding the assessor.
Generative Adversarial Networks (GANs). With the development of Generative Adversarial Net- works (Goodfellow et al. 2014), that are indistinguishable from real images. Generally, these networks are composed of two parts; generative and diseriminative network. The goal of the generative network is generate real- looking images to fool the discriminator and the goal of the discriminator is to learn to distinguish generated images from real images. These two networks are optimized through a minmax game where both sides compete to reach their specified goals. In recent years, there have been huge im- provements this area and many different GANs have been introduced to produce natural-looking images such as Progressive GAN (Karras et al. 2017). DCGAN (Radford et al. 2015)), and Cycle- GAN (Zhu et al. 2017). In this work, we used pre-trained StyleGANT KKrras et al. 2019) and
now capable of generating real-looking synthetie images
StyleGAN2 (Karras et al. 2020) on the FFHQ dataset (Karras al. 2019 to generate real-looking faces.
Modifying Image Memorability. While image memorability modification has many potential ap- plications (e.g. education or advertisement), it has not been adequately investigated, Khosla et al. (2013)) proposed pioneering method for changing face memorability. It leveraged Active Appear- ance Models (AAMs (Cootes et al. 2001h) to represent faces by their shape and appearance. Then the loss function defined based the cost of modifying identity, modifying facial attributes, and memorability. As result, in their method, the identity was fixed. Another approach (Sidorov 2019) used VAE/GAN (Larsen al., 2016). StarGAN (Choi al. 2018), and AttGAN (He al. 2019) and trained them with three memorability levels (poorly memorable, moderately memorable, and highly memorable) of faces and modified the memorability of different faces these three levels only. Additionally, Siarohin et al. (017)) utilized style transfer to increase the memorability of an input image. However, the added style adversely affected the realness of the modified im- ages, such that it barely could be used in real-world applications. Most recently, Goetschalck.. et al.) (2019) trained transformer network to change the memorability of each generated image by Big- GAN (Brock et al.) 2018) through modifying their latent vectors. Their method works generated images of objects and scenes.
METHOD
The overview our proposed method is depicted schematically Figure approach step by step.
Below we explain the

---

Under review as a conference paper at ICLR 2022
2.1 CREATING THE DATASET
For the purpose of analyzing the latent vectors of the GANs and their relation to memorability, we need large dataset of face images with their memorability scores. The largest dataset available for face images is the 10k US Adult Faces Database (Bainbridge et al. 2013). This database con- tains 10,168 natural human face images and 2,222 of these images are annotated with memorability scores. To create a larger dataset for face images with their corresponding memorability scores, we leveraged StyleGANI and StyleGAN2, which are the state-of stat-oof-the-at the-art models for creating realistic- looking face images. These models were pre-trained on the FFHQ dataset (Karras et al. 2019) which consists of 70,000 high-quality face images with 1024 1024 resolution with variations in age, gender, and glasses. We created two different datasets with StyleGANI and StyleGAN2. We randomly sampled 100k & E 1x12 from a standard normal distribution with truncation to produce high-quality synthetic face images and saved their mappings in the extended latent space (R (8x5222) of both GANs.
2.2 PREPROCESSING STEP
The generated images from the StyleGAN have 1024 X 1024 resolution in three channels. For the purpose of acquiring the memorability of the generated images, we had preprocess them to predict their memorability with the computational memorability predicting models (assessor). We leveraged VGG16 (Simonyan & Zisserman 2014), ResNet50 (He et al. 2016), and SENe150 (Hu al. 2018) that are pre-trained on VGGFace2 (Cao et al. 2018) and fine-tuner them the US 10k Face Database to correctly estimate face memorability scores. This dataset consists of ohaleshaped human faces with white backgrounds. 10k US Adult Face images have the same height of 256 with different widths. Hence, we had to first ovalize the generated faces, then compute their memorability scores. We used MTCNN (Zhang et al. (016)) for detecting the face in the generated images and masked an oval on it to make all the images similar to the US 10k Face Database (See Figure 2
SIng STYM 34 LENGETT
Suppl poomat ingy
Denmoie diee entiiiig Any
Figure 2: Preprocessing step to make the shape of the synthesized images similar to the dataset that the assessor trained on.
2.3 PREDICTING THE MEMORABILITY SCORES
As explained in previous section, memorability assessors trained 10K US face database which includes ovalized faces. We first, tested whether the ovalization process is necessary to obtain memo- rability scores for generated faces. For this, fine-tuned the pre-trained SENet50 (Hu al. 2018). VGG16 (Simonyan & Zisserman, 2014), and ResNet50 (He et al.) 2016) for predicting face mem- orability scores on 10K US face database. We calculated the memorability of both the oval-shaped and square-shaped faces generated from StyleGANI and StyleGAN2. The distributions of the mem- orability scores using SENet with oval and squared faces are shown in Figure 3 As can be seen, the memorability distributions are very similar for oval and square faces. We further calculated the Kendall rank correlation (Schaeffer & Levit]| 1956) and Spearman's 's rank correlation for the paired oval-shaped and square-shaped images. We observed that SeNet50 and VGGI6 have high-rank score (see Table I These results showed that these models can also be used for the equare-shaper face images that contain background. The benefit of using shaped faces is that have more control on the hair of the person. Please note that the performance of memorability predicting models measured with rank correlation (Khosla et al. 2015).
4

---

Under review as a conference paper at ICLR 2022
1000
6000
0020
5003
5000
5000
Count
4002
Count
4000
s00p
Count
2002
2000
2000
2000
2000
2000
2000
5000
Memorabiltty 1.0
0.3
Homorabiity
Mcmorabilty Score
Memoraalliy Score
(a) StyleGanl (oval)
(b) StyleGanl (square)
(c) StyleGan2 (oval)
(d) StyleGan2 (square)
Distributions of the memorability scores of the generated images of StyleGan and Style- Gan2 based on using an ovalization step or not.
Figure 3:
Table I:
Correlations of memorability scores of qquare-shaped and oval-shaped faces. High corre- lation score suggests the assessor performs well on square faces too.
Assessor
Kendall Tau correlation
Spearman's correlation
ResNe150
0. 0.1606
0,2391
SENet50
0.4419
0.6217
VGG16
0.4720
0.6562
2.4 THE PROPOSED METHOD FOR MODIFYING FACE MEMORABILITY
Next we aimed find a hyperplane to separate the highly-memorabie and low-memorable im- ages. First, we needed to label the faces into highly-memorable and memmrabeeble faces. For this, we used the mean of memorability scores as threshold. We labeled an image as highly or low memorable if its memorability score was higher lower than the mean. In our experiments, we also tried using median of the memorability scores as the threshold for labeling high and low memorable groups. Next, using logistic regression, we attempted to find hyperplane to separate low-melorable and megooyyeemorable images. For this purpose, we used either z E R 1x1222 latent vectors w IR 18x512 extended latent space of StyleGANI and StyleGAN2. After finding the separating hyperplane, we moved the latent vector or the augmented latent vector in the positive or negative direction of the normal vector of that hyperplane
control the image memorability. This hyperplane denotes the moderately memorable images, as we chose it to separate the faces into high- memorable and low-memorable faces based on the mean (or median) of the memorability scores. Memorability of each image is related to the distance of its latent vector (or extended latent vector extended latent vector is used to find the hyperplane) from this hyperplane. Consider image i with corresponding latent vector E R 1xx112 or w; E R 118x512 and its memorability score mem,, We note the normal vector of this hyperplane by 2* E IR1x112 or un* E 1818x512 and we show the distance from the hyperplane by function d. We will have:
mem; & d(z", zi) = z²" .2
mem; x d(w", w:) = w² .w5
Hence, we can change the memorability of each image, by changing the distance of its latent vector
(or extended latent vector) from the separating hyperplane; Zedit = 2 + az" or similarly Wedit = w + aw" and will have:
d(z", Redit) = x" Redit x²" (z az") = z"" x + d(z", z)
+
(3)
d(w", Wedst) = w² Wedst = w" (w
+ oww" = w" ww + a = d(w", w) +
(4)
The results of this classification task for finding the separating hyperplane are presented in Table when different assessors are used for memorability score predictions. In each case, the accuracy was about 10 percent higher when used extended latent space (w), hence showed the results for extended latent space in Table 4 According to the results, we decided to use SENet50 as our assessor and mean memorability the threshold for our further analysis.
5

---

Under review as a conference paper at ICLR 2022
Table 2: Accuracy of the separating hyperplane, based on the method for dividing images into
and low-memorable images, the shape of the images, and the assessor.
Median
Mean
Assessor
Oval
Square
Oval
Square
ResNet50
0.8131
0.7933
0.8149
0.7928
SENet50
0.8157
0.8291
0.8207
0.8317
VGGI6
0.7938
0.8037
0.7952
0.8071
The performance of logistic regression is higher when the extended latent space is used to find the separating hyperplane. Further, space yields better results modifying face memorability. (See A.4 for the latent space results.)
show in the next section working with the extended latent
One of the benefits of our approach is that has been shown that latent vector play an important role in determining different attributes of a face and and smile. Consider the norm vectors of these hyperplanes as A {a1, a2.... ax}. We can fix these attributes by subspace projection follows: Wnew " D (w* (t.Taaaa), .a,)a (See A.5 AS5
fix specific attributes while changing memorability.
find hyperplanes separate faces based those attributes, such glasses, age,
2.5 LATENT VECTOR RECOVERY FOR REAL FACES
The efficiency of our method to modify real human faces depends on how well the latent vector of the real face can be obtained to reconstruct the original image. After acquiring the latent vector of the real face image, we can repeat the process for the synthesized images and modify their memorability. In this work, used image2style (Abdal et al. 2019) to embed the real faces to latent space of StyleGAN1, whose loss function consists of a VGG-16 perceptual loss (Johnson et al. 2016) and pixel-wise MSE loss term. Furthermore, for projecting real faces to StyleGAN2 latent space, employed the same algorithm described by Karras et al. (2020) after using the dlib library (King) 2009) to align 68 face landmarks in the preprocessing step. In this work, we assume that have the latent vector of the generated faces, however, if the latent vector of generated face is not available, similar approach can be employed to retrieve the latent vector.
3
EXPERIMENTS
3.1 CHANGING MEMORABILITY OF SYNTHESIZED FACES
As described in Section 2.1 we created two different datasets with StyleGANI and StyleGAN2.
We randomly sampled 100k = E R1x512 from a standard normal distribution with truncation to produce high-quality synthetic face images and saved their mappings in 18x512 augmented latent space of both GANs. We calculated their memorability scores using SENet50, then labeled them highly low-memorable images when compared the mean memorability finally we used the logistic regression to find the separating hyperplane as described in Section 2.4 After finding the separating hyperplane, we evaluated our method in modifying the memorability of generated faces with StyleGANI and StyleGAN2 using our proposed method. As we discussed earlier in 2.4 we found separating hyperplanes in either latent space or extended latent space of StyleGANT and StyleGAN2. Figure and Figure 5 demonstrates some samples of memorability modification with StyleGANI and StyleGAN2 respectively (See [A.]] for more examples). We ob- serve that increasing the memorability scores causes some decreases in the facial weight, increases the presence of makeup and thickness of the lips, and makes the person look younger. Moreover, will affect the skin tone by making it brighter. makes the face more serious. However, decreasing the memorability score has opposite effects.
in the dataset,
In addition to using our method to evaluate face images qualitatively, we also tested the performance of our method in modifying memorability, quantitatively. For this, we used 10k synthesised faces and tried different weights of memorability modification them. o of memorability scores when shifted by our method as well as the mean of the distributions, which successful in modifying the memorabiliyy scores of the faces.
the distributions
suggests

---

Under review as a conference paper at ICLR 2022
Increasing Memorability
6:9
Decressing Memorsbility
Original Image
Incressin Memorability
BF
Decrsssing Memorability
Figure 4: Modifying memorability of faces generated by StyleGANI. Exemplar faces and their counterparts when our memorability modifying approach is applied to them. The second row depicts images when an extra step of ovalization is applied before feeding to the assessor. The corresponding memorability score is presented the top left corner of each image.
Incrassing Memorshility
64
Decmasing M Memorsbility
Original Image
Incressng Memorabilty
FV
Decressing Memorability
Figure 5: Modifying memorability of faces generated by StyleGAN2. Exemplar faces and their counterparts when our memorability modifying approach is applied them. The second row depicts images when an extra step of ovalization is applied before feeding to the assessor.
3.2 REALNESS OF THE MODIFIRD FACES
In this experiment, we evaluated the realness of the faces when our memorability modification was applied to them. StyleGAN is a ofate-of-the-art model in generating real- real-looking faces. Hence, we considered the realness of the generated images from StyleGAN (before memorability modification) as our baseline and compared them to the modified faces. First generated 10k synthesized faces, then appllied different weights of memorability modification vector to their latent vectors. We uti- lized two well-known measures for this purpose; FID and KID. As demonstrated in Figure 7 the FID and KID scores of the modified images are close to the unmodified faces and therefore. the realness of faces is not affected by method.
7

---

Under review as a conference paper at ICLR 2022
0.4
0. 05 Mamorebilty Score
64
10
02
10
Mamorrsiliyy ScorE
Oval-shape faces
Square faces
Figure 6: The effectiveness of our method for modifying memorability scores tested on 10k gener- ated faces. As depicted the distribution and mean memorability score of images changes with the coefficient used for memorability modification.
rable KD roto
-2 Memorab lity coefficient
Figure 7: The realness of the 10k generated images measured by the FID and KID ratios for dif- ferent memorability modification coefficients. FID and KID score the unmodified images are the respective baselines. A close to one ratio indicates similar level of realness with the baseline.
3.3 CHANGING MEMORABILITY OF REAL FACES
Original Images
Recovered Images
Increasing memorability
Decreasing memorability
CI
OZF
A32
AAP
60
OAT
Figure 8: Memorability modification of real faces. The first three rows were encoded Style- GAN2 latent space and the fourth row was encoded to StyleGANI latent space.
Next evaluated our method on real human faces. First, computed their extended latent vector using the GAN inversion method previously described and then applied our proposed method to them change their memorability. For real human faces, chose work with the extended
8

---

Under review as a conference paper at ICLR 2022
latent vectors, because the regenerated images from the extended latent space similar to the original images (See Figure 8
3.4
LAYERWISE MEMORABILITY MODIFICATION
Decreasing attribute
Original Images
Increasing attribute
630
EX1t
C
6400
00
Moreubd
Layar
Apvenal Ntaee
Layer DOTTOMT 120
Square- shape faces
Oval- Oval-shape faces
Figure 9: The effect of changes in the 7th layer of the extended latent space on memorability score. This layer mostly affects the presence of makeup in female and facial hair in male faces. We observe that when we move this layer in the positive direction of the corresponding layer of the memorability vector (w*), the memorability score will drastically increase and the presence of makeup is highlighted. Additionally, we observed that the lips thickness increases, and the eyebrows shape and the skin tone of the person changes changes.
Next we tried to identify which layers in extended latent vectors contributed the most to face mem- orability, i.e., which layers are most responsible for modifying face memorability score. In each experiment, we only changed one layer and kept other layers the same. We plotted the faces after these layerwise changes to examine what kind of changes these layerwise modifications caused in the faces. We observed that modifications in the first 11 layers were mostly responsible for changes face, and the other layers mostly just affected the color and background of the image. Hence, only focused the first 11 layers. We then repeated the same layerwise changes on 3k syn- thesized faces and calculated the mean memorability of these images before layerwise and after layerwise modifications (See Figure 9 for the other layers see A.2
CONCLUSION
In this work, we proposed a new method for modifying the memorability of face images. Our approach does not suffer from the limitations of the previous methods and is able to modify the memorability of faces (synthetic and real) within an arbitrary continuous range. Moreover, we demonstrated these changes will not affect the realness of the images. However, if a very large weight for the memorability modification vector is chosen, it will affect the face identity and real- ness. We showed that our method is effective by applying it to 10k synthesized faces. Further, employed our method on real human faces and showed it is effective in changing the memorabil- ity of real faces well. Finally, studied how layerwise modifications will affect the face and its memorability score and discussed one of the benefits of the proposed method is modifying the memorability score conditionally by leveraging subspace projection method.

---

Under review as a conference paper at ICLR 2022
REFERENCES
Rameen Ahdal, Yipeng Qin, and Peter Wonka. Image2stylegan: How to embed images into the stylegan latent space? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4432-4441, 2019.
Gal Almog, Saeid Alavi Naeini, Yu Hu, Emma G Duerden, and Yalda Mohsenzadeh. Memoir dataset: Quantifying imgge memorability in adolescents.
Wilma A. Bainbridge, Phillip Isola, and A. Oliva. The intrinsic memorability of face photographs. Journal of experimental psychology. General, 142 4:1323-34, 2013.
Mikotaj Binkowski, Danica J Sutherland, Michael Arbel, and Arthur Gretton. Demystifying mmd gans. arXiv preprint arXiv:1801.0 01 00100,10010000000000001010000000 2018.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint w.iiv:809. 1096, 2018.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE international conference on automatic face & gesture recognition (FG 2018), pp. 67-74. IEEE, 2018.
Yunjey Choi, Minje Choi, Munyoung Kim, Jung-Woo Ha, Sunghun Kim, and Jaegul Choo. Star- gan: Unified generative adversarial networks for multi-domain image-to-imane translation. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 8789-8797, 2018.
Timothy Cootes, Gareth Edwards, and Christopher Taylor. Active appearance models. IEEE Transactions on pattern analysis and machine intelligence, 23(6):681 1-88, 2001.
Navneet Dalal and Bill Triggs. Histograms of oriented gradients for human detection. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR'05), volume 1, pp. 886-893. leee, 2005.
J Fajtl, V Argyriou, D Monekosso, and P Remagnino. Amnet: Memorability estimation with atten- tion, arxiv 2018. arXiv preprint 1
Lore Goetschalckx and Johan Wagemans. Memeat: new category-bane based image set to study image memorability. 05 2019, doi: 10. 191114400..222.11105.5.222
Lore Goetschalekx, Alex Andonian, Aude Oliva, and Phillip Isola. Ganalyze: Toward visual defi- nitions of cognitive image properties. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 5744-5753, 2019.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Erik Härkönen, Aaron Hertzmann, Jaakko Lehtinen, and Sylvain Paris. Ganspace: Discovering interpretable gan controls. arXiv preprint 20X::20004002466. 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 770-778, 2016.
Zhenliang He, Wangmeng Zuo, Meina Kan, Shiguang Shan, and Xilin Chen. Attgan: Facial attribute editing by only changing what you want. IEEE transactions image processing, 28(1 1):5444- 5478, 2019.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by a two time-scale update rule converge local nash equilibrium. Advances in neural information processing systems, 30, 2017.
Jie Hu, Shen, and Gang Sun. Squeeze-and exdexxtttoooiiiiii networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018
10

---

Under review as a conference paper at ICLR 2022
Earl B Hunt, Janet Davidson, and Marey Lansman. Individual differences in long-tern reggtterm memory access. Memory & Cognition, ((6):5996008, 1981.
Phillip Isola, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. What makes an image memorable?
In CVPR 1, pp. 152. IEEE, 2011.
Phillip Isola, Jianxiong Xiao, Devi Parikh, Antonio Torralba, and Aude Oliva. What makes pho- tograph memorable? Pattern Analysis and Machine Intelligence, IEEE Transactions on, 36(7):
1482, 2014.
Justin Johnson, Alexandre Alahi, and Li Fei-Fei. Perceptual losses for real-time style transfer and super-resolution. In European conference on computer vision, pp. 694-711. Springer, 2016.
Tero Karras, Timo Aila, Samuli Laine, and Jaakko Lehtinen. Progressive growing of gans for im- proved quality, stability, and variation. arXiv preprint arXiv: 2017.
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4401-4410, 2019.
Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyz- ing and improving the image quality of stylegan. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp- 81 8110-8119 (118,199, 2020.
Aditya Khosla, Wilma Bainbridge, Antonio Torralba, and Aude Oliva. Modifying the memorability of face photographs. pp. 3200-3207, 12 2013a. doi: 10.1 10.1109/CCC2201333337
Aditya Khosla, Wilma A Bainbridge, Antonio Torralba, and Aude Oliva. Modifying the memora- bility of face photographs. In Proceedings of the IEEE International Conference Computer Vision, pp. 1-000-320,,5 2013b.
Aditya Khosla, Akhil S. Raju, Antonio Torralba, and Aude Oliva. Understanding and predicting image memorability large scale. In 2015 EEEE International Conference Computer Vision (ICCV), pp. 2390-2398, 2015. doi: 10. 10.11099/CCV.2215.27555
Davis E. King. Dlib-ml: A machine learning toolkit. Journal of Machine Learning Research, 10: 1755-1758, 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep volutional neural networks. Advances in neural information processing systems, 25:1097-1105, 2012.
Anders Boesen Lindbo Larsen, Søren Kaae Sønderby, Hugo Larochelle, and Ole Winther. Autoen- coding beyond pixels using a learned similarity metric. In International conference on machine learning, pp. 1558-1566. PMLR, 2016.
David G Lowe. Distinctive image features from iele--invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Coen D Needell and Wilma A Bainbridge. Embracing new techniques in deep learning for estimat- ing image memorability. arXiv preprint arXiv:2 105. 10598, 2021.
Timo Ojala, Matti Pietikainen, and Topi Maenpaa. Multiresolution gray-scale and rotation invari- ant texture classification with local binary patterns. IEEE Transactions on pattern analysis and machine intelligence, 24(7):971-95) 1-987. 2002.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint 11Xiv: 2015.
Maurice S Schaeffer and Eugene Levitt. Concerning kendall's 's tau, a nonparametric correlation coefficient. Psychological Bulletin, 53(4):338, 1956.
Yujun Shen, Ceyuan Yang, Xiaoou Tang, and Bolei Zhou. Interfacegan: Interpreting the disentan- gled face representation learned by gans. IEEE transactions on pattern analysis and machine intelligence, 2020.
11

---

Under review as a conference paper at ICLR 2022
Aliaksandr Siarohin, Gloria Zen, Cveta Majtanovic, Xavier Alameda- Elisa Ricci, and Nicu Sebe. How to make an image more memorable? deep style transfer approach. In Proceedings of the 2017 ACM on International Conference Multimedia Retrieval, pp.
2017.
Oleksii Sidorov. Changing the image memorability: From basic photo editing gans. Proceed- ings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pp. 0-0, 2019.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint 1rXiv:1409.15533 2014.
Hammad Squalli-Houssaiie Ngoc QK Duong, Marquant Gwenaëlle, and Claire-Hélène Demarty- Deep learning for predicting image memorability. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.
IEEE, 2018.
Kaipeng Zhang, Zhampeng Zhang, Zhifeng Li, and Yu Qiao. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 2016.
Jun- Yan Zhu, Taesung Park, Phillip Isola, and Alexei Efros. Unpaired image-to-image translation using cycle-consistent adversarial networks. In Proceedings of the IEEE international conference on computer vision, pp. 2223-2232, 2017.
12

---

Under review as a conference paper at ICLR 2022
A
APPENDIX
A.1
ADDITIONAL EXAMPLES
Decreasing memorability
Original Images
Incressing memorability
AB
R27
028
17
668
A16
821
647
0.22
028
Figure 10: Generated images by StyleGANi with their corresponding memorability scores. Modified images when extended latent space the StyleGANI was used to determine the separating hyperplane. square-shaped faces were fed to the assessor.
Decreasing memorability
Original Images
Increasing memorability
A21
837
83
651
012
8.55
E31
ABE
B31
651
0U23
231
0AT
AS4
335
651
Figure 11: Generated images by StyleGANi with their corresponding memorability scores. Modified images when extended latent space the StyleGAN1 was used to determine the separating hyperplane. Oval-shaped faces were fed to the assessor.
13

---

Under review as a conference paper at ICLR 2022
Decreasing memorability
Original Images
Increasing memorability
318
Eiit
62
622
3.5
x333
623
3.3
028
Figure 12: Generated images by StyleGANI with their corresponding memorability scores. Modified images when latent space of the StyleGAN1 was used to determine the separating hyper- plane. Square-shaped faces were fed to the assessor.
Decreasing memorability
Original Images
Incressing memorability
Ed1
631
8.5
853
653
635
362
6:35
Figure 13: Generated images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when extended latent space of the StyleGAN2 was used to determine the separating hyperplane. Square-shaped faces were fed to the assessor.
14

---

Under review as a conference paper at ICLR 2022
D eecreasing memorability
Original Images
Increasing memorability
A1G
Gd
63B
0.8
631
C41
19
335
633
020
OAT
Figure 14: Generated images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when extended latent space of the StyleGAN2 was used to determine the separating hyperplane. Oval-shaped faces were fed to the assessor.
Decreasing memorability
Original Images
Incressing memorability
115
EM
60
64'
0.19
6:3
6:38
325
63L
Figure 15: Generated images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when latent space of the StyleGAN2 was used to determine the separating hyperplane. Square-shaped faces were fed to the assessor.
15

---

Under review as a conference paper at ICLR 2022
A.2
LAYERWISE MODIFICATIONS
D Decreasing attribute
Original Images
Incressing attribute
E45
0432
Bd1
0.3
220
00
0.2 D5 08 10
Layer
Namoraa Scars
Square shape faces
Oval-shape faces
Figure 16: The effect of the changes in the 1st layer of the extended latent space on memora- bility score. This layer mostly affects shape of the face. Increasing this attribute makes the face smaller, whereas decreasing this attribute makes the face larger, which usually may cause a decrease in the memorability score.
D Decreasing attribute
Original Images
Increasing attribute
E35
AZ2
0.5
1.5
Namondo By SeSTa
Leya www.dor KNNT
square-shape faces
Oval-shape faces
Figure 17: The effect of the changes in the 2nd layer of the extended latent space memora- bility score. This layer mostly affects hair, pose of the face, and the direction of the eyes and shows how these attributes contribute to the memorability score.
16

---

Under review as a conference paper at ICLR 2022
Decreasing attribute
Original Images
Incresing attribute
839
E38
637
061
0.5
0.2
0.6
26 03 100
LayEe
Namorab Scars
Square-shape shape faces
Oval-shape faces
Figure 18: The effect of the changes the 3rd layer of the extended latent space on memora- bility score. This layer mostly affects shape and seriousness of the faces. We observed that moving this layer, in the positive direction of the corresponding layer in the memorability vector (w"), ), will highly increase the memorability score.
D Decrassing attribute
Original Images
Increasing attribute
6:4
60
OUD
B3B
2.25
1.0
0.0 00
02
04 D6
18/08 Coeftdler Pne
00
CO
022
06
80
1.0
Square- faces
Oval-shape faces
Figure 19: The efffec of the changes in the 4th layer of the extended latent space on memora- bility score. This layer mostly affects shape of the face (especially the chin). We observe that changes in this attribute, largely contribute to the memorability of the face and moving this layer in the positive direction of the corresponding layer in the memorability vector (w*), will increase the memorability score.
17

---

Under review as a conference paper at ICLR 2022
D Decreasing attribute
Original Images
Incressigg attribute
0:34
649
E38
AZE
BBC
072
22.5
15
20
06
Layee
Namoraa ScarS
square--hape faces
Oval-shape faces
Figure 20: The effect of the changes the 5th layer of the extended latent space on memora- bility score. This layer mostly affects nose, lips, facial weight, and smile. This is one of the most important layer that plays a role determining the memorability score.
D Dcrrassing attribute
Original Images
Increasing attribute
E35
AIT
030
641
3.0
1.5
MoTrOUD StOTe
Masrorabel S00t
Layer
Square-shape faces
Oval-shape faces
The effect of the changes in the 6th layer of the extended latent space memorabil- ity score. This layer mostly affects the smile and form the lips. We observe that when we move this layer, in the positive direction of the corresponding layer of the memorability vector (w"), the memorability score will increase hugely and the person's 's lips will become thicker. However, when you move it the opposite direction, the person's 's lips will become thinner and the memorability score will decrease.
Figure 21:
18

---

Under review as a conference paper at ICLR 2022
D Decreasing attribute
Original Images
Increasing attribute
E37
Ao
641
2.3
2.0
15
NeTION BY S2E0P
Layer
Namonao By SE5TS
Layer cwffdee km
Square-shape faces
Oval-shape faces
Figure 22: The effect of the changes in the 7th layer of the extended latent space memora- bility score. This layer mostly affects make-ups and facial hair. We observe that when we move this layer, in the positive direction of the corresponding layer of the memorability vector (w"), the memorability score will drastically increase and the makeup starts to appear on the person's ''s face. The lips will become thicker and the eyebrows shape and the person's skin changes.
D Decreasing attribute
Original Images
Increasing attribute
B37
044
AZE
033
222
2.3
30 255
2.0
00
02
04
D6
DA
Lagae CANTRANE
00 Namoral
Square-shape faces
Oval-shape faces
Figure 23: The efffec of the changes the 8th layer of the extended latent space on memora- bility score. This layer mostly affects the eyes and the hair color.
19

---

Under review as a conference paper at ICLR 2022
D ocrreasing attribute
Original Images
Incressigg attribute
642
0.54
683
2.5
0.5
03
M. 0.5
Layer
SULTE
Layer 2oaffdert MAR
Square-shapp faces
Oval-shape faces
Figure 24: The effect of the changes in the 9th layer the extended latent space on memorabil- ity score.This layer mostly affects the facial hair, hair color type and skin color. moving this layer, in the positive direction the corresponding layer of the memorability vector (w=), will make the hair color gold and some shadows and facial hair (if the person is male), will appear on the face.
D Decreasing attribute
Original Images
Increasing attribute
E35
023
622
Layer
Merorah Scare
Layer Coffedett km
Square-shar faces
Oval-shape faces
Figure 25: The effect of the changes in the 10th layer of the extended latent space on orability score. The changes in this layer or mostly responsible for modifications on skin color and eyes. We observe, the changes in this layer are as effective as other layers to modify the memorability score.
20

---

Under review as a conference paper at ICLR 2022
D ecreasing attribute
Original Images
Increasing attribute
61F
E31
825
2.5
Na Tonaaa Soote
Namonaa Bry SCATS
Layer COMESOET MAR
love and
Square-shape faces
Oval-shape faces
Figure 26: The effect of the changes in the 11th layer of the extended latent space memora- bility score. Changes in this layer control the skin color and face brightness. As it shown, this layer does play an important role in modifying memorability score of the face.
A.3 FACE MEMORABILITY ASSESSORS
In order to train the models, split the 10k US Face Database images into train, validation and test split. We used 80 percent of the data as the training samples and used percent of the data for each used as the loss function and the batch size
of the test and validation splits. Euclidean distance was set to 64. Moreover, we leveraged Adam optimizer to train our models. Due to large false alarm rates in human face images, trained models both with by hit rate) and corrected memorability scores (considering false alarm rate). We also tried some simple augmentations on the dataset and found, the score of the models will slightly increase if use a simple augmentation like random horizontal flipping(p = 0.5).
memorability scores (computed
Consistent with Khosla al. Khosla et al.) (2013)), we observed when the corrected hit rate scores are used, all models outperform the case when only hit rate scores are used. That is because false alarm rate introduces noise to memorability scores, therefore. the models perform better when we reduce the noise by correcting for false alarms. We have brought the rank correlation scores using pretrained VGG16, ResNet50 and SENet50 using hit rate and true hit rate values.
Table Memorability scores of the models pre- pre-trained on face recognition (on VGGFaces database) and fine-tuned on 10K US face database. Note that all these computational models, produce larger Spearman's rank correlation score when true hit rate scores are used.
Model
Hit Rate Score
True Hit Rate Score
VGG16
0.445
0.579
ResNe150
0.433
0.607
SENet50
0.448
0.601
A.4 FINDING HYPERPLANE IN THE LATENT SPACE
In this work, have provided the results of the separating hyperplane accuracy in extended latent space. We utilized the latent space to find the separating hypeprlane in the latent space of StyleGAN2 and reported the accuracy of the hyperplane.
21

---

Under review as a conference paper at ICLR 2022
Table Accuracy of the separating hyperplane, based the method for dividing images into high- memorable and low memorable images, the shape of the images, and the assessor. In this case latent space (IR is used to find the separating hyperplane. We can observe that the accuracy of the hyperplane is lower than the case when extended latent space is used.
Median
Mean
Assessor
Oval
Square
Oval
Square
ResNet50
0.699
0.686
0.706
0.683
SENet50
0.696
0.723
0.700
0.733
VGG16
0.689
0.695
0.697
0.705
A.5 MODIFYING THE FACES CONDITIONALLY
As described previously, of the benefits of our work is that it makes it possible to modify the memorability scores of the faces conditionally. Different hyperplanes for different face attributes could be used and then with projection, can try maintain the corresponding attributes un- changed. As there correlation between different face attributes, choosing large weights and attempting change the memorability scores drastically, may affect the attribute. We showed the attempt to maintain smile and age attributes in Figure 27 as an example. However, this method could be applied to variety of face attributes.
Decressing memorability unconditionaaly
Increasing memorability umoonditionally
Origiaal image
Conditional smile
Original image
Conditional to age
0.34
936
06I
042
Figure 27: Modifying memorability scores the faces, with the condition to maintaining smile and age. This figure shows how projection affect memorability and the special attribute that are aiming to maintain unchanged.
A.6 MODIFYING MEMORABILITY OF GENERATED NON-FACE IMAGES STYLEGAN
We extended our experiments and tested our method other non-face images. We leveraged pre- trained StyleGAN2 on churches, cats, horses and cars. In here, we used MemNet as our memo- rability assessor. (See Table 5 for accuracy of the hyperplanes in churches, cats, horses and cars augmented latent space.)
Table 5; Accuracy of the separating hyperplane, based on the weights of the generator.
Weight
Accuracy
Cats
0.7875
Horses
0.8897
Cars
0.8581
Churches
0.8325
22

---

Under review as a conference paper at ICLR 2022
D Decreasing memorability
Original Images
Increasing memorability
A72
BBG
0.79
663
0.83
0:
0.25
0.23
0.24
008
Figure 28: Generated horse images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when extended latent space of the StyleGAN2 was used to determine the separating hyperplane.
D Decreasing memorability
Original Images
Increasing memorability
8B9
OFR
0.70
2.22
651
2.85
668
3.55
8:31
033
Figure 29: Generated cat images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when extended latent space of the StyleGAN2 was used to determine the separating hyperplane.
A.7 MODIFYING MEMORABILITY OF GENERATED OBJECT IMAGES BY BIGGAN
Lastly, we tried to show the effectiveness of our method on generated images by BigGAN. We gen- erated 200k images by 512 x 512 BigGAN-decy (Brock al. 2018), predicted their memorability scores by our assessor, and divided them into low-memorable and memorbbl images. We observed that the effect of the modifications. is similar (iottschalck et al. (019). Increasing the
23

---

Under review as a conference paper at ICLR 2022
D Decreasing memorability
Original Images
Increasing memorability
0.0
665
0.81
Ed
683
0.90
090
Figure 30: Generated car images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when extended latent space of the StyleGAN2 was used to determine the separating hyperplane.
Decreasing memorability
Original Images
Increasing memorability
3.90
EEL
C64
633
6:2
0.96
0/23
c45
E:54
3.55
6:39
0d22
165
EdS
471
817
Figure 31: Generated church images by StyleGAN2 with their corresponding memorability scores. This figure shows the modified images when extended latent space of the StyleGAN2 was used determine the separating hyperplane.
memorability scores, caused the images to become zoomed-in, in some cases the color changed and also in few cases (Cheese burger and snake in Figure 35a made the objects rounder.
24

---

Under review as a conference paper at ICLR 2022
11.05
1..00
11.00
2.75
11.00
2.70
0.80
0.50
0.75
3:10
0.16
0.70
0.6
-6 Mamorabilly
0.50
by
Remorabiily comtitient
Nemrrrbilly ccnflicter.
(a) Cats
(b) Horses
(c) Cars
(d) Churches
Figure 32: The effectiveness of our method for modifying memorability scores tested on 5k gener- ated images for each category. As depicted the distribution and mean memorability score of images changes with the coefficient used for memorability modification
Decreasing memorability
Original Images
Increasing memorability
BB7
A.7
6155
1.9
153
A22
181
154
615
2.77
2.55
87
0121
1.34
Figure 33: Generated images by 512 512 BigGAN-deep with their corresponding memora-
bility scores. This figure shows the modified images when the latent space of the BigGAN (IR 128 used to determine the separating hyperplane.
25

---

Under review as a conference paper at ICLR 2022
Decreasing memorability
Original Images
Increasing memorability
Eds
BB
623
c39
2855
0.06
0.71
3.78
0.4
066
673
3.75
Figure 34: Generated images by 256 x 256 BigGAN with their corresponding memorability scores. This figure shows modified images when the latent space of the BigGAN (R140) was used to determine the separating hyperplane.
0.84
0.90
g 0.52
2
0.85
0.7
0.76
0.75
0.70
0.72
0.70
-3
-2
0.65
-3
-2 -1
Merorabilit scafficient
Mamorabilly conffiiint
(a) 512 BigGAN-deep
(b) 256 256 BigGAN
Figure 35: The effectiveness of our method for modifying memorability scores tested on 20k gener- ated images. As depicted the distribution and mean memorability score images changes with the coefficient used for memorability modification
26


Reviews:
[{"review_id": "SFxfaPHokE5", "reviewer": "Reviewer_xi9k", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "As pros:\n - basic observation (latent space analysis with effect on memorability) is very interesting. it is strong enough to make the paper outstanding. However the paper develops ideas previously published as mentioned in prior work\n- technical bases is solidly built\n- the evaluation is rather strong. The observation, which again is very interesting, needs to be proven by experimental evaluation and the paper does a good job. Furthermore, all the examples  showed in the paper do confirm the basic idea and developments\n- the appendix is useful\n\nAs cons:\n - the paper is limited in scope as it addresses only the problem of face memorability\n- the experimentation is limited to the statistic of UsFaces database. It is not extended with an analysis over gender, age, race.\n- the evaluation does not compare the proposed method with prior art mentioned on pager 3 in paragraph \"Modifying Image Memorability.\"\n\nSmall issues:\n - the amount of images used in experimentation is \"hidden\" in the text. The text should me be made more clear to enhance the quantities used. Also some of the quantities should be added in the figure and tables captions", "summary_of_the_paper": "The paper addresses the problem face image memorability. The proposal is based upon the identification in Style GAN of a hyper axis in the latent space where movement is correlated with memorability", "summary_of_the_review": "Overall the idea of of the paper is nice but the evaluation is barely convincing.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "IVRMsu64FG8", "reviewer": "Reviewer_iYVp", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n1. The experimental results show that the proposed method does change the memorability of images.\n2. The proposed method can be applied to real facial images.\n\nWeaknesses:\n1. Are there experimental comparisons with other existing methods that change image memorability? Can those methods be applied to the images generated by StyleGAN? What're the advantages of the proposed method over the existing methods?\n2. When choosing VGG16, ResNet50, or SENet50, why the accuracy of them on the US 10k Face Database with real labels is not considered?\n3. The accuracy when using the extended latent space (w) is reported in Table 2. It's better to report the accuracy when using the latent space (z) in the appendix.\n4. Fig 4, 5, 6, and 8 lack analysis, such as what can be proved by these figures.  The top-left numbers on the facial images are not explained.", "summary_of_the_paper": "This paper proposes to find the hyperplane to separate images with different memorability in the latent-vector spaces of StyleGAN1 and StyleGAN2, then they can move the latent-vectors along the normal vector of the hyperplane to control the memorability of the generated images. A SENet50 trained on the US 10k Face Database with memorability labels is used to find the hyperplane. The experimental results show that the proposed method can change the memorability of images measured by the scores of the SENet50, and this method can be extended to modify real facial images.", "summary_of_the_review": "The effectiveness of the proposed method is validated by experiments, while the superiority (over existing methods) is not. The experimental results need more detailed descriptions.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "8lD0Rv6zf4", "reviewer": "Reviewer_8EvC", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "## Strengths\n\n- The proposed approach is reasonably designed to modify face memorability.\n\n- The paper shows many example images, and we can observe some interesting cases of appearance changes with respect to memorability scores.\n\n## Weaknesses\n\n- Technical novelty is not clear. As discussed in the paper, the underlying approach based on the attribute-separating hyperplane is mostly equivalent to InterFaceGAN. Shen et al. already demonstrated their approach on face image generation tasks, and this work extends it to memorability modification. The concept of image memorability modification for generated faces has been also discussed in [Goetschalckx et al., 2019], and the contribution is not significant from the technical perspective.\n\n- Experimental validation is not sufficient. Essentially, this paper provides qualitative results and quantitative evaluation in terms of face realness (FID and KID metrics) and memorability (assessor score) without any comparison with baseline approaches. There are some missing aspects such as memorability assessment with human participants and comparison with existing memorability modification approaches. In both senses, it would be required to demonstrate the difference from [Goetschalckx et al., 2019] by showing that the proposed method can control human memory performance on face images better.\n\n- While the paper claims that the proposed technique aims at modifying memorability without affecting identity and other facial attributes, there are some cases in the examples whose identity clearly changed after memorability modification. Compared to other relatively objective facial attributes, it is not a trivial task to judge if the facial identity was maintained during the modification and this would require quantitative evaluation by either machine or human evaluator.", "summary_of_the_paper": "This paper presents a method to control the memorability of face images in the latent subspace of GANs. The basic approach is similar to InterFaceGAN [Shen et al., 2020]. The authors first train a memorability assessor network on the 10k US Adult Faces Database and used the assessor to provide memorability scores on generated face images from StyleGANs pre-trained on the FFHQ dataset. They then find the hyperplane in the StyleGAN latent space to separate low-memorable and high-memorable images according to the memorability score. The memorability of faces is modified by changing the distance from the hyperplane. They demonstrated that their method can successfully control the memorability scores of generated images while keeping the realism as face images and show some qualitative examples and discussions about memorability-modified face images.", "summary_of_the_review": "The approach is reasonable and has the potential to lead to new findings in face memorability studies. However, given that the technical novelty is relatively incremental and the experimental results obtained at this point are still in the early stages, the research contribution is not sufficient.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "4auUANgZ50a", "reviewer": "Reviewer_m2GG", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strength \n\n+ They tackle an interesting problem “What makes an image memorable?”, where the problem is well defined. This work is the first experiment about ‘memorability modification’ that could be applied in various applications. \n\n+ Their approach solves the problem by introducing the concept of distance between the latent vector or the extended latent vector and the normal vector of the hyperplane.\n\n+ The paper is easy to follow and presented the experimental results with StyleGAN and StyleGAN2. \n\nWeakness\n\n- Writing can be sigificantly improved. There are several typos and mistakes in writing (e.g., As the result, → As a result).\n\n- Experiments with more datasets are needed because only two StyeGANs are used for evaluation. More SOTA GAN generated images need to be evaluated to provide the effectiveness and validity of this approach.\n\n- Although they tackle an interesting problem, the proposed method is rather simple, and it is a combination of existing methods. Therefore, the novelty is rather weak.\n\n ", "summary_of_the_paper": "This work presents an approach to modify face memorability as a facial attribute using Generative Adversarial Networks (GANs). They essentially propose “memorability modification vector.” Their approach first determines the hyperplane and we can move the latent vector of each image, in the positive or negative direction of the normal vector of that hyperplane. They control the distance of the latent vector from the separating hyperplane to manipulate the memorability of that image. They conducted experiments with StyleGAN and StylyeGAN2 to demonstrate effectivness of their apporaching using FID and KID metrics. \n", "summary_of_the_review": "Although this work tackles an interesting problem, the novelty is weak. Also, more experiments with different GAN datasets would be helpful. It is limited to StyleGANs. Also, the writing should be improved. Reproducibility is not certain as they did not provide the code. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers raised a number of major concerns including the limited novelty of the proposed, inadequate motivation of the design choices and, most importantly, insufficient and unconvincing experimental evaluation presented. The authors’ rebuttal addressed some of the reviewers’ questions but failed to alleviate all reviewers’ concerns. Hence, I cannot suggest this paper for presentation at ICLR.
Decision: Reject