From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech
Authors: Hyeong-Seok Choi
Changdae Park
Kyogu Lee
Venue:  (2020)
Keywords: Multi-modal learning
Self-supervised learning
Voice profiling
Conditional GANs

Abstract: This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. To this end, we propose a multi-modal learning framework that links the inference stage and generation stage. First, the inference networks are trained to match the speaker identity between the two different modalities. Then the pre-trained inference networks cooperate with the generation network by giving conditional information about the voice. The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our system fully end-to-end. We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of a face image one that comes directly from a speech signal and the other that is not related to it and explore whether the network can learn to generate natural human face image distribution by modeling these factors. Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech. Finally, the correlation between the generated face and the corresponding speech is quantitatively measured to analyze the relationship between the two modalities.

Published as a conference paper at ICLR 2020
FROM INFERENCE TO GENERATION: END-TO-END FULLY SELF-SUPERVISED GENERATION OF HUMAN
FACE FROM SPEECH
"Hyeong-Seon Choi', 'Changdae Park2, Kyogu Lee'
Music and Audio Research Group (MARG), Seoul National University 2 Department of Computer Science and Engineering, Hong Kong University of Science and Technology kekepal snu. ac. kr, cdparkeccnnect.. .ust.uk kgleeêsnu. ac. kr
ABSTRACT
This work seeks the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. To this end, we propose multi-modal learning framework that links the inference stage and generation stage. First, the inference networks are trained to match the speaker identity between the two different modalities. Then the trained inference networks cooperate with the generation network by giving conditional information about the voice. The proposed method exploits the recent development of GANs techniques and generates the human face directly from the speech waveform making our sys- tem fully end-to-end. We analyze the extent to which the network can naturally disentangle two latent factors that contribute to the generation of face image- one that comes directly from a speech signal and the other that is not related to it and explore whether the network can learn to generate natural human face image distribution by modeling these factors. Experimental results show that the pro- posed network not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech. Finally, the correlation between the generated face and the correspond- ing speech is quantitatively measured to analyze the relationship between the two modalities.
INTRODUCTION
[Utilizing audio-visual cues together to recognize person's identity has been studied in various fields from neuroscience (Hasan et al. 2016; Tsantani et al.) 2019) to practical machine learning applications (Nagrani et al.) 201 8b neurological studies have found that in some cortical areas, humans recognize familiar individuals by combining signals from several modalities, such as faces and voices (Hasan al. (0166). In conjunction with the neurological studies, it is also a well known fact that human speech production system is directly related to the shape of the vocal tract (Mermelstein) 1967 Teager & Teager 1990).
Wen et al. 2019a Shon et al. 2019). For example, some
Inspired by the aforementioned scientific evidence, we would like to ask three related questions from the perspective of machine learning: 1) possible match the identity of faces and voices? (inference) 2) If so, is it possible to generate face image from a speech signal? (generation) 3) Can we find the relationship between the two modalities only using cross-modal self-supervision with the data -hild" To answer these questions, we design a two-step approach where the inference and generation stages are trained sequentially. First, the two inference networks for each modality (speech encoder and face encoder) trained to extract the useful features and compute the cross-modal identity matching probability. Then the trained inference networks are transferred the generation stage to pass the information about the speech, which helps the generation network to output the face image from the conditioned speech.
We believe, however, that it impossible to perfectly reconstruct all the attributes in the image of a person's 's face through the characteristics of the voice alone. This is due to factors that are clearly
*Equal contribution

---

Published as a conference paper at ICLR 2020
unrelated one's voice, such as lighting, glasses, and orientation, that also exist in the natural face image. To reflect the diverse characteristics presented in the face images "in-the-wild" we therefore model the generation process by incorporating two latent factors into the neural network. More specifically, we adopted conditional generative adversarial networks (cGANss (Mirza & Osindero 2014 MMyato & Koyama| 2018) so that the generator network can produce face image that is dependent not only on the paired speech condition, but also on the stochastic variable. This allows the latent factors that contribute the overall facial attributes be disentangled into two factors: one that is relevant to the voice and the other that is irrelevant.
Adopting cGANs negligently still leaves few problems. For example, the condition in a cGANs framework is typically provided as embedded conditional vectors through the embedding look-up table for one-hot encoded labels (Brock et al., 2019 Miyato & Koyama 2018). The raw signals such speech, however, cannot be taken directly from the embedding look-up table, so an encoder module is required. Therefore, the trained speech encoder from the inference step is reused to output pseudo conditional label that is used to extract meaningful information relevant to the correspond- ing face. Then the generator and the discriminator are trained in an adversarial way by utilizing the pseudo-embedded conditional vectors obtained from the trained speech encoder in the first step.
Another problem with applying the conventional cGANs for generating faces from voice arises from the fact that the distinction between different speakers can be quite subtle, which calls for need for a more effective conditioning method. To mitigate this problem, we propose a new loss function, relativistic identity cGANs (relidGANs) loss, with modification of the relativistic GANs (Jolicoeur- Martineau 2019). allowing us to generate the face with a more distinct identity.
Each step will be described in greater detail in Section 3
Our contributions can be summarized as follows:
1. We propose simple but effective end-to-end inference networks trained audio-visual data without any labels in a self-supervised manner that perform a cross-modal identity matching task.
2. A cGANs-based generation framework is proposed to generate the face from speech, to be seamlessly integrated with the trained networks from inference stage.
3. A new loss function, so called a relidGANs loss, is designed to preserve a more consistent identity between the voices and the generated images.
4. An extensive analysis is conducted on both inference and generation tasks to validate our proposed approaches.
2 RELATED WORKS
There has been an increasing interest in the self-supervised learning framework within the machine learning research community. While the focus of the many studies has been concentrated appli- cations of matching the audio-visual correspondence such as the presence or absence of objects in a video or temporal alignment between two modalities (Chung et al.) 2017 Afouras al. 2018b a Ephrat et al. (2018), growing attention has come to matching the speaker identity between the human face and voice.
Inference The cross-modal identity matching task between face and voice has been recently studied in machine learning community. Nagrani et al. (2018b) proposed convolutional- neurall--rrkl.l.-te.rr... (CNN) based biometric matching network by concatenating the two embedding vectors from each modality and classifying them with an additional classifier network. Though showing promising results, this style of training is limited as the model is not flexible in that the number of concatenated vectors used in the training cannot be changed in the test phase. Next, Nagtan, et al. (2018a) modeled the concept of personal identity nodes (Bruce & Young 1986) by mapping each modality into the shared embedding space and using triplet loss to train the encoders to allow more flexible inference. Lastly, Wen al. (2019a) proposed DIMNet where two independent encoders speech and face into the shared embedding space, classifying each of them independently utilizing the supervised learning signals from labels such identity, gender and nationality.
trained to map the
2

---

Published as a conference paper at ICLR 2020
Generation There have been few concurrent works that tackled the similar problem of generating a face image from speech, which we think are worth noting here. Duarte et al. (2019) proposed GANs-based framework to generate face image from the speech. But the intention of their work was not to generate a face from unseen speaker identity but more of seeking the possibility of cross- modal generation between the speech and face itself. Oh et al. (2019) proposed similar work called speech2face. The pre-trained face recognition network and the neural parametric decoder were used produce normalized face (Parkhi al., 2015 Cole al. 2017). After that, the speech encoder was trained to estimate the input parameter of the face decoder directly. Because the training process still requires the pre-trained modules trained with supervision, we do not consider this a fully-self- supervised approach. Lastly, the most similar work to ours was recently proposed by Wenn et al. (2019b) where they utilized a pre-trained speech identification network as a speech encoder, and used a GANs-based approach to produce the face conditioned on the speech embedding.
Our proposed method differs from the abovementioned approaches in the following ways. First, none of the previous approaches has tried to model the stochasticity in the generation process, but we address this problem by incorporating the stochasticity in the latent space so that the different face images can be sampled even when the speech condition is fixed. Second, modeling the image more challenging in our work as we aim to train our network to produce larger image size (128 128) compared to other GANs-based works. Also, trained the model on the AVSpeech dataset Epprra et al. (018) which includes extremely diverse dynamics in the images. Finally, the important scope of this work is to seek the possibility of training the whole inference and generation stages only using the self-supervised learning method, which is the first attempt to our knowledge. The whole pipeline of the proposed approach is illustrated Fig.
Proh.
sph7 xpp2
Speaker
Softmax
Facel embedding (fe,)
Inner Product
Speaker
Speech embedding (se)
Speech encoder
Face2 embedding (fez)
(a) Inference stage
Face encoder
Transfer
Transfer
Generator
Fake
Discriminator
D
AdaIN (conditioning)
Speech encoder (fixed)
Real
Inner Product
(b) Generation stage
Figure 1: Overview of the proposed inference and generation stages.
3

---

Published as a conference paper at ICLR 2020
3 SELF-SUPERVISED INFERENCE AND GENERATION
3.1 CROSS-MODAL IDENTITY MATCHING
In order successfully identify the speakeridentity from the two audio-visual modalities, we trained two encoders for each modality. First, a speech encoder is trained to extract the information related to a person's 's identity from a segment of speech. To this end, we use raw- www-wavefrmm based speech encoder that was shown to extract useful features in speaker verification task, outperforming con- ventional feature such as -ref- cooffficentt (MFCC) even when trained using self- The first layer of the speech encoder is SincNet where the parameterized sinc functions act as trainable band-pass filters (Ravanelli & Bengio 2018). The rest of the layers are composed of 7 stacks of Id-CNN, batch normalization (BN), and multi murati-paramttric rectified linear unit (PReL.U) activation. Next, we used a 2d-CNN based face encoder with residual connections in each layer. Note that the network structure is similar to the discriminator network of (Miyato & Koyama 2018). The details of the networks are shown in Appendix C
supervised learning in speech modality (Pascual et al. 2019 [1
Based on the speech encoder SI)) and the face encoder F(.), they are trained to correctly identify whether the given face and speech paired or not. Specifically, as a cross-modal identity matching task, we consider two settings as in (Nagrani et al. 20188). 1. V-F: given a segment of speech select one face out of K different faces, 2. F-V: given an image of face select one speech segment out of K different speech segments. The probability that the j-th face fj is matched to the given speech or vice versa computed by the inner product of embedding vectors from each module followed by the softmax function as follows:
<se.fe,>
1. : ply =
ply = s/sse, fez
DjEk ece,,Je,>>
<oe,se>>
2. F-V: p(U = (1(((((s)), = p(u = fel fel
j=k 2j=1 exoaife>>
where s denotes a speech segment, f denotes a face image, se denotes a speech embedding vector and fe denotes a face embedding vector.
Note that using inner product as means of computing similarity allows more flexible inference. For example, the proposed method enables both F-V and V-F settings, no matter with which method the model is trained. Also, it allows setting a different number of K in the test phase than the training phase.
Finally, the two encoders are trained solely using the self-supervised learning signal from cross- entropy error.
3.2 GENERATING FACE FROM VOICE
Although aim to generate the human face from the speech, it is only natural to think that not all attributes of the face image are correlated to the speech. Hence, we assume the latent space of the face to be broken down into two parts, the deterministic variable from the speech encoder and random variable z sampled from Gaussian distribution.
Such latent space is modeled using cGANs, generative model that allows the sampling of face images conditioned on speech. More specifically, randomly sampled Gaussian noise z E 12128 and speech condition e E IR. 128 are concatenated and used as input for the generator function fake = G(z. c) to sample the face image conditioned on the speech. In addition, adaptive in- stance normalization (AdaIN) technique is applied as a more direct conditioning method for each layer of the generator network (Huang & Belongie 2017 Karras et al. 2019). The details of the network are described in Appendix
In order to generate the face image that are not only close enough to the real face image distribution, but matches with the given condition, the condition information must be properly fed into the dis- criminator. Many studies have suggested such conditioning approaches for the discriminator (Reed
TTe speech encoder be downloaded the following link: http::/ggihub.com/sstttttoo/o
4

---

Published as a conference paper at ICLR 2020
al. 2016 Odena et al. 2017, Miyato & Koyama 2018): among them, adopted a recent condi- tioning method, projection discriminator (Miyato & Koyama 2018), which not only suggests a more principled way of conditioning embeddings into the discriminator, but has also been widely used in many successful GANs related works (Brock et al. 2019) Zhang et al. 2018 Miyato et al. 2018). The study showed that, writing the discriminator function DJJ c) A c)), , the condition information can be effectively provided to the discriminator using the inner-product of two vectors, and Φ as follows:
off.c)= o(f) (o(()),
where A(:) denotes activation function (sigmoid in our case), denotes condition embedding, (ff) denotes output from the inner layer of discriminator and v()) denotes a function that maps input vector scalar value (fully-connected layer in case). Here we focused the fact that the conditioning signals can be used as an inner-prode amer-product of two vectors and replaced it with the same inner-product operation used compute the identity matching probability in the inference framework. Accordingly, we can rewrite the Eq. by providing the condition with the trained speech encoder S() and substituting (()) with trained face encoder F(.) from the subsection 3.1 as follows:
g(f.c) - (()) + (O()) - S(8) F(J) + (F()))
Next, adopted relativistic GANs (reIGANs) loss which was reported to give stable image gener- ation performance. See See Jolicoeur Martineau (219) for more details. Combining the condition term c" ((f) and reIGANs loss. g f c) can be modified to geel (freal f ffkee c) as follows:
grel freat ff ffke
c) = (frraa, c) - fake c)
= c² o(freal) c² (ffake) ((((prea))) - ( (f
(flake, great
c) affaake c) g(freal, c)
offfffakee cº ((frall)) + ((((ffake) (dd(freal)
Eq. is formulated to produce face from the paired speech, but it can catastrophic forget- ting on the trained o(-) because the discriminator is no longer trained to penalize the mismatched face and voice. Thus, we again modify Eq. 4 that the discriminator relativistically penalizes the mismatched face and voice more than a positively paired face and voice as follows:
gelid( ffake c_ gr ffake, ,c,) - c offo(fraa c² (( g freal, c_), g= freal,ct)
(5)
cc) + ( cfo(ffake) - cº ((akak))
where freat and C. denotes the paired face and speech condition from data distribution, s denotes the generated face sample conditioned C+, and denotes the speech condition with mismatched identity to freat using negative sampling.
Finally, utilizing the non-saturating loss (Goodfellow et al. 2014 proposed objective function (reliGGANs loss) for discriminator Lp and generator Lg becomes as follows:
2019). the
Lo = -Begrent .t. fa Paen (Ag (egredd (Freal, flake ,C,, c-))]{
(6)
Lgg -R c+ propatta I fak (og relree preat C.t, c-,,
4 EXPERIMENTS AND RESULTS
Dataset and Sampling Two datasets were used throughout the experiments. The first dataset is the AVSpeech dataset (Fshhrat et al. 22018). consists of 2.8m of YouTube video clips of people actively speaking. Among them, we downloaded about 800k video clips from train set and 140k clips from a test set. Out of 800k training samples, we used 50k of them as a validation set. The face images included in the dataset are extremely diverse since the face images were extracted from the videos "in the wild" (e.g., closing eyes, moving lips, diversity of video quality, and diverse facial expressions) making it challenging train generation model. In addition, since speaker identity information provided, training the model with this dataset is considered fully self sepersuupe Because of the absence of the speaker identity information, we assumed that each audio-visual clip represents
5

---

Published as a conference paper at ICLR 2020
individual identity. Therefore, positive audio-visual pair sampled only within single clip while the negative samples were randomly selected from any clips excluding the ones sampled by the positive pair. Note that each speech and face image included in the positive pair was sampled from different time frames within the clip. This was to ensure that the encoded embeddings of each modality do not contain linguistic related features (e.g., lip movement or phoneme related features).
The second dataset is the intersection of VoxCeleb (Nagrani & Zisserman 2017) and VGGFace (Parkhi al. 2015). VoxCeleb is also large-scale audio-visual dataset collected on YouTube videos. VGGFace is a collection of the images of public figures gathered from the web, which is less diverse and relatively easier to model compared to the images in AVSpeech dataset. Since both VGGFace and VoxCeleb provide speaker identity information, we used the speakers" included in the both datasets resulting 1,251 speakers. We used face images from both VoxCeleb and VGGFace, and used speech audio from Voxceleb. Note that, this dataset provides multiple images and audio for a single speaker; therefore, training model with this dataset cannot be called self-supervised training method in strict sense.
Implementation In every experiment, we used 6 seconds of audio. The speech samples that were shorter or longer than 6 seconds were duplicated or randomly truncated so that they became 6 sec- onds. At the inference stage, we trained the networks with stochastic gradient descent optimizer (SGD). At the generation stage, we trained the networks with Adam optimizer (Kingma & Ba 2015). The discriminator network was regularized using Ri regularizer (Mescheder et al. 2018). More details are described in Appendix B
4.1 CROSS-MODAL INFERENCE EVALUATION RESULTS
Fully self-supervised inference accuracy results: Here we show the evaluation results of the in- ference networks trained on the AVSpeech dataset. Again, the model was trained with no additional information about speaker identity. The evaluations were conducted 5 times by selecting different negative samples each time and reported using the average value of them.
We trained our model in two settings (1. V-F and 2. F-V) and both were trained in 10-way setting (1 positive pair and 9 negative pairs). Then the two trained models were tested on
Table 1: Accuracy (%) of cross- modal identity matching task.
both V-F and F-V settings, each of which were tested on 2- way and 10-way settings. The topl-accuracy (%) results of cross-modal identity matching task is shown in Table [] The results show that our network can perform the task of cross- modal identity matching with a reasonable accuracy despite being trained in a fully self-supervised manner. In addition, our model can perform F-V inference with reasonable accu- racy even when it is trained in V-F setting, and vice versa.
Train
V.F
F.V.
V-F F.V.
89.22
54.33
85.10
44.70 52.55
86.94
49.20
88.47
K-way
2
2
Comparison results: We compared our model with two models SVHFNet (Nagrani al. 2018b) and DIMNet (Wen et al., 2019a). For fair comparisons, we trained our model with the intersection of VexCeeeeb and VGGFace datasets. Our model is trained in two train settings (1. V-F and 2. F-V) with 10-way configuration.
The comparison results are shown in Table The results show that our model performs worse when tested in 2-way setting compared to other models. Note that, SVHFNet used a pre- trained speaker identification network and face recognition network trained in a supervised manner with identity infor- mation and DIMNet trained the network in a supervised man- ner using the labels such as identity, gender, and nationality, and therefore is not trainable without such labels, whereas our model was trained from the scratch and without any such la- bels. Under more challenging situation where K increases from two to ten, however, our model shows significantly bet-
Table 2:
Comparison results in terms of accuracy (%) of cross- modal identity matching task.
Truin
V.F²
Model
SVIINNe DIMNet-IG
81.00
35
79.50
84.12
84.03
Ours
79.90
55.66
80.83
54.84
K-way
2
10
2
10
TThe 10-way results inferred from the graph of the paper DIMNet (Wnn et al.
2019a).

---

Published as a conference paper at ICLR 2020
ter performance, which may indicate the proposed method is capable of extracting informative and reliable features for the cross-modal identity matching task.
4.2 CROSS-MODAL GENERATION EVALUATION RESULTT
We conducted three qualitative analyses (QLA's) and three quantitative analyses (QTA's) to thor- oughly examine the relationship between condition and the generated face samples.
QLA 1. Random samples from (z, c) plane: The generated face images from diversely sampled
and fixed speech condition C shown in Fig. 2
We can observe that each variable shows different
characteristics. For example, it is observable that controls the orientation of head, background color, haircolor, hairstyle, glasses, etc. Alternatively, we observed that e controls gender, age, eth- nicity and the details of the face such the shape of the face and the shape of the nose.
Randomly sampled
Randomly sampled cs
Figure 2: The illustration of generated face images from interpolated speech conditions. Note that the first column consists of the ground truth image of the speakers
QLA Generated samples from interpolated speech condition: Next, we conducted an exper- iment to observe how speech conditions affect facial attributes by exploring the latent variable with z fixed. We sampled the speech segments from the AVSpeech test set and linearly interpolated two speech conditions. After that we generated the face images based on the interpolated condition vectors. The generated face images are shown in Fig. 3 We can observe the smooth transition of speech condition in the latent space and can confirm that it has a control over the parts that are directly correlated to the identity of person such as gender, age, and ethnicity.
QLA Generated samples from interpolated stochastic variable: Lastly, we conducted an ex- periment that shows the role of stochastic variable in the generation process by interpolating two randomly sampled vectors with fixed. The speech segments from the AVSpeech test set were used to make a fixed c for each row. The generated face images are shown in Fig. 4 We can observe that the latent vector has a control over the parts that are not directly correlated to the identity of person such as head orientation, hair color, attire, and glasses.
QTA Correlation between the generated samples and speech conditions: We performed a quantitative analysis to investigate the relationship between the speech condition and the face image generated. To this end, we first sampled two different random variables 21, #2 and two different speech condition vectors C1, C2 of different speakers. Next, we generated two face images f fake (Z21, c1), f/ fake G( 62) using the generator network. Then the two generated samples were encoded using the trained inference network F(-) to extract the face embeddings fe; =ffffa fe, = F(f. akke. We then calculated the cosine distance between the speech condition vectors (CD((1, C2)), and the cosine distance between the two face embeddings (CD(fe1, fez)). Finally, we computed the Pearson correlation between CD(c1, C2) and CD(fe1.fe) This is to see there exists any positive correlation between the embedding spaces of the two different modalities;
Speech samples: htwp:h//ww.....o..oooo../o/////m

---

Published as a conference paper at ICLR 2020
(8) Male
(b) Male
(c) Young to
(d) Young Femaleto to Old Male
(c) Different Ethnicities
The illustration of generated face images from interpolated speech condition vectors while fixing z. Note that the images on the very left and right sides of each row are the ground truth face images of the speakers.
Figure 3:
Head orientation
(b) Hair color
(c) Attire
(d) Glasses
Figure 4: The illustration of generated face images from interpolated random vectors while fixing Note that the images on the very left side of each row are the ground truth face images of the speakers.
that is, to see if closer speech embeddings help generate face images whose embeddings are also closer even when random variable z is perturbed.
Fig. (a) shows the results using the test set of the AVSpeech. We can see positive correlation be- tween the CD(c1, 02) and CD(fe1, fez) meaning that the generated face images are not randomly sampled.
Furthermore, we examined whether the CD between two speech condition vectors gets closer when controlling the gender of the speaker, and also the face images generated from the two speech con- dition vectors. We tested this on VoxCeleb dataset as provides gender labels for each speaker identity. We compared the mean values of C C(C1,, C2) and CD(fe1. fey) in two cases; the gender the two speakers different and the other one setting the gender the same. Fig. (b) shows scatter plot of the CDD when the two sampled speakers have different genders, and Fig. 5
setting
8

---

Published as a conference paper at ICLR 2020
shows a case where the genders set the same. We found out that the mean value of CD(c1, o2) gets smaller (0.46 + 0.28) when we set the gender of two speakers the same and accordingly the Cfefe fez) gets also smaller (0.27 + 0.15).
Correlation: 0,5857
Correlation: 0.4131
Correlation: 0.4514
0.5
0.5
0.5
CD(fe..ee)
0.4
CD(fe,
0.3
0.3
CD
0.3
0.2
0.2
0.2
0.0
0.2
041
0.6
0.8
L.0
0.0
02
0.4
0.6
03
1.0
0.0
0.2
0.4
0.6
0.8
1.0
(D)
CD
CD C2)
CD (C,, C2)
(a) AVSpeech
(b) Voxceleb (diffferent gender)
(c) Voxceleb (samne gender)
Figure 5: The scatter plots of (D((c,,c)) and CD(fer,
QTA Testing the generated samples with the inference networks: Here we conducted two inference experiments in V-F setting. In the first experiment, given a positive pair from the AVSpeech test set (f, 8), two face images (f and [(z, S(s))) are passed to the trained inference networks. The inference networks are then used to determine which of the two images yields greater cross-modal matching probability with a given speech segment s as follows:
N 1 n=1
ply = 21((()))
where I[]] denotes an identity function, denotes index of test sample, fn, denotes the generated sample ((2, (s,)((),)) fn, denotes the ground truth image paired with the speech 8,, and N denotes the total number of test samples. Note that z was randomly sampled from Gaussian distribution for each n.
Surprisingly, we found out that the inference networks tend to choose the generated samples more than the ground truth face images with a chance of 76.65%, meaning that the generator is able to generate a plausible image given the speech condition c. On the other hand, we found out that the chances significantly decreases to 47.2% when do not use the relidGANs loss (using Eq. 4 showing that the proposed loss function helps generate images reflecting the identity of a speaker. Note that, however, this result does not necessarily say that the generator is capable of generating a more plausible image conditioned on one's voice than the paired face image as this experiment is bounded the performance of the inference network. In addition, believe the distribution of real face image is much more diverse than the generated samples, causing the inference network tend towards the generated samples more.
Our second experiment is similar to the first experiment, but this time we compared the generated images from two different generators; one trained without mismatched identity loss (Eq. 4 and the other with the proposed relidGANs loss. We found out that the inference networks selected the generated sample from the generator trained with the proposed relidGANs loss with a chance of 79.68 79.68%, again showing that the proposed loss function helps to generate samples that reflect the identity information encoded in the speech condition.
Next, we conducted one additional experiment in F-V setting. Given generated image G(z, S(x1)) from a speech segment 81, we measured the accuracy of inference network selecting 81 out two audio segments 8 and 82 as follows:
N
Σ 1pp( - (150s
p(u - 2116((,,,)),((6,)))
(8)
where fn denotes a generated face image from a speech segment $n,1, and $n,2 denotes nega- tively selected speech segment. We found out that the inference networks select s1 with chance of 95. 14%. Note that, the accuracy of inference networks in 2-way F-V setting is 88.47%, which means the generator faithfully generate face image according to the given speech segment $1.

---

Published as a conference paper at ICLR 2020
QTA 3. Face image retrieval: Lastly, conducted face retrieval experiment in which the goal was to accurately retrieve a real face image for the speaker using the generated im- age from their speech segment
query. To compose retrieval dataset, randomly sam- pled 100 speakers from the test set the AVSpeech. For each speaker, we extracted 50 face images from video clip resulting in 5,000 images for the retrieval experiment in total. Note that this process of composing the re-
trieval dataset is same as that of Speech2Face
Table 3: Face retrieval performance.
Oh et al. (219). To retrieve the closest face im- age out of the 5,000 samples, the trained face encoder was used to measure the feature dis- tance between the generated image and each of the 5,000 images. We computed three metrics as a feature distance (L1, L2, and cosine dis- tance (CD) and the results are reported in Ta- 3
Top-K
Models
K
K
K
K 10
8.34
24.66
36.22
Speech2Face
8.28
13.66
24.66
35.84
CD
10.92
17.00
30,50
45.82
Ours
L1
7,32
12.81
24.41
38.82
(Wlo mil)
Table B shows that model achieve higher performance than that of Speech2Face all distance metrics. We also measured the performance the generator trained without mismatched identity loss (mil) (Eq. 4). The re- sults show that the proposed relidGANs loss function is crucial generate the face images that reflect the identity of the speakers. Examples of the top-5 retrieval results are shown in Fig.
CD
7.36
13.04
24.78
39.59
Qurs
12.97
20.98
36.56
$2.66
Lg
12.90
21.5
36.84
52.49
(relidGANs))
CD
13.59
21.69
36.94
53.83
Real
Generated
Top-5 nearest face images
Figure 6: Top-5 retrieval examples.
CONCLUSION AND FUTURE WORK
In this work, we proposed a cross- modal inference and generation framework that can be trained in fully self- supe-vuper way. We trained cGANs by transferring the trained networks from the inference stage so that the speech could be successfully encoded as pseudo conditional embedding. We also proposed relidGANs loss to train the discriminator penalize negatively paired face and speech
10

---

Published as a conference paper at ICLR 2020
so that the generator could produce face images with more distinguished identity between different speakers. As a future work, we would like to address a data bias problem (e.g.. ethnicity, gender, age, etc.) that exists in many datasets. This is significant problem as many publicly available datasets have biased demographic statistics, consequently affecting the results of many algorithms (Bullamwini & Gebru, 2018). We believe that this be solved with the of better data sampling strategy in an unsupervised manner such as (Amini et al., 2019). In addition, we would like expand the proposed methods various multi-modal datasets by generalizing the proposed concept to other modalities.
ACKNOWLEDGMENTS
This work was supported partly by Next-Generation Information Computing Development Pro- gram through the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (MSIT; Grant No. NRF-2017MM3CA700N0 & Communications Technology Planning & Evaluation(IITP) grant funded by the Korea govern- ment(MSIT) (N0.2019-0-0166,, Infant-Mimic Neurocognitive Developmental Machine Learning from Interaction Experience with Real World (BabyMind)
and partly by Institute for Information
REFERENCES
Triantafyllos Afouras, Joon Son Chung, Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Deep audio-visual speech recognition. IBEE transactions on pattern analysis and machine intelligence, 2018a.
Triantafyllos Afouras, Joon Son Chung, and Andrew Zisserman. Deep lip reading: comparison of models and an online application. Proc. Interspeech, 2018b.
Alexander Amini, Ava Soleimany, Wilko Schwarting, Sangeeta Bhatia, and Daniela Rus. Uncov- ering and mitigating algorithmic bias through learned latent structure. In Proceedings of the AAAI/ACM Conference on AI, Ethics, and Society, 2019.
Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. International Conference on Learning Representations (ICLR), 2019.
Vicki Bruce and Andy Young. Understanding face recognition. British journal of psychology, 77 (3):005-327, 1986.
Joy Buolamwini and Timnit Gebru. Gender shades: Intersectional accuracy disparities in commer- cial gender classification. In Conference on fairness, accountability and transparency, 2018. 8.
Joon Son Chung. Andrew Senior, Oriol Vinyals, and Andrew Zisserman. Lip reading sentences in the wild. In 2017 IEEE Conference Computer Vision and Pattern Recognition (CVPR). IEEE, 2017.
Forrester Cole, David Belanger, Dilip Krishnan, Aaron Sama,, Inbar Mosseri, and William Free- man. Synthesizing normalized faces from facial identity features. In Proceedings of the IEFE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Amanda Duarte, Francisco Roldan, Miquel Tubau, Janna Escur, Santiago Pascual, Amaia Sal- vador, Eya Mohedano, Kevin McGuinness, Jordi Torres, and Xavier Giro-i Nieto. Wav2pix: speech-conditioned face generation using generative adversarial networks. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2019.
Ariel Ephrat, Inbar Mosseri, Oran Lang, Tali Dekel, Kevin Wilson, Avinatan Hassidim, William Freeman, and Michael Rubinstein. Looking to listen at the cocktail party: A speaker-indeppedent audio-visual model for speech separation. ACM Trans. Graph., 2018
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farke Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 2014.
11

---

Published as a conference paper at ICLR 2020
Bashar Awwad Shiekh Hasan, Mitchell Valdes- Sosa, Joachim Gross, and Pascal Belin. "hearing faces and seeing voices": Amodal coding of person identity in the human brain. Scientific reports, 6:37494, 2016.
Xun Huang and Serge Belongie. Arbitrary style transfer in real-time with adaptive instance nor- malization. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2017.
Alexia Jolicoeur-Marineauu gan. International Conference on Learning Representations (ICLR), 2019.
The relativistic discriminator: a key element missing from standard
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
Davis E. King. Dlib-ml: A machine learning toolkit. J. Mach. Learn. Res., 10:1755-1758. Decem- ber 2009. ISSN 1532-4435.
1755843
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. Conference on Learning Representations (ICLR), 2015.
International
Paul Mermelstein. Determination of the vocal-tract shape from measured formant frequencies. The Journal of the Acoustical Society of America, 41()): (1(55)1283-11)) 123--129,, 1967.
Lars Mescheder, Andreas Geiger, and Sebastian Nowozin. Which training methods for gans do actually converge? Proc. International Conference on Machine Learning (ICML), 2018.
Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint
arXiv:1411.1784 2014.
Takeru Miyato and Masanori Koyama. cGANs with projection discriminator. In International Conference on Learning Representations (ICLR), 2018.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for generative adversarial networks. In International Conference on Learning Representations (ICLR), 2018
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable pins: Cross-modal embeddings for person identity. In Proceedings of the European Conference on Computer Vision (ECCV), 2018a.
Arsha Nagrani, Samuel Albanie. and Andrew Zisserman. Seeing voices and hearing faces; Cross- modal biometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR), 2018b.
Joon Son Chung Nagrani, Arsha and Andrew Zisserman. Voxceleb: a large-scale speaker identifica- tion dataset. In Proc. Interspeech, 2017.
Augustus Odena, Christopher Olah, and Jonathon Shlens. Conditional image synthesis with auxil- iary classifier gans. In Proc. International Conference on Machine Learning (ICML). JMLR. org, 2017.
Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T Freeman, Michael Rubinstein, and Wojeiech Matusik. Speech2face: Learning the face behind voice. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, al. Deep face recognition. In British Machine Vision Conference, 2015.
Santiago Pascual, Mirco Ravanelli, Joan Serrà, Antonio Bonafonte, and Yoshua Bengio. Leam- ing crublem-agnostie speech representations from multiple self-supervised tasks. Interspeech, 2019.
In Proc.
12

---

Published as a conference paper at ICLR 2020
Mirco Ravanelli and Yoshua Bengio. Speaker recognition from waveform with sincnet. In 2018 IEEE Spoken Language Technology Workshop (SLT), 2018.
Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In Proc. International Conference on Machine Learning (ICML), 2016.
Suwon Shon, Tae-Hyun Oh, and James Glass. Noise-tolerant audio-visual online person verification using an attention- based neural network fusion. In IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). IEEE, 2019.
HM Teager and SM Teager. Evidence for nonlinear sound production mechanisms in the vocal tract. In Speech production and speech modelling, pp. 241-261. Springer, 1990.
Maria Tsantani, Nikolaus Kriegeskorte, Carolyn McGettigan, and Lúcia Garrido. Faces and voices in the brain: a modality-generdi person-identity representation in superior temporal sul- cus. Neurolmage, 201: 16004, 2019.
Yandong Wen, Mahmoud Al Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh. Disjoint mapping network for cross-modal matching of voices and faces. International Conference on Learning Representations (ICLR), 2019a.
Yandong Wen, Bhiksha Raj. Changil Kim, and Rita Singh. Face reconstruction from voice using generative adversarial networks. In Advances in neural information processing systems, 2019b.
Han Zhang, Ian Goodfellow, Dimitris Metaxas, and Augustus Odena. Self-attention generative adversarial networks. In Proc. International Conference on Machine Learning (ICML), 2018.
APPENDIX
A DATA PREPROCESSING
Image processing: All video clips downloaded from the AVSpeech dataset were resampled to be 25FPS If more than one face are detected frame, a face closer to the coordinates of the target speaker in the first frame was selected. Note that the coordinates of the target speakers are provided by the AVSpeech dataset. Because the size of the speaker's face on the screen varies from video to video, the image was resized to maintain interocular distance as 55 pixels, as described in Cole al. (2077). We used a publicly available software, Dlib (King) 2009). to detect and crop the face images. The images were cropped to the size of 224 x 224 and resized to 128 x 128 for training. We additionally applied horizontal flip method as a data au augmentation strategy. These procedures of cropping, resize, and flipping are adapted to all images in VGGFace and VoxCeleb, too.
Audio processing: All audio samples were resampled to 16kHz, and stereo audio samples were converted to mono. We used 6 seconds of audio in both the inference and generation model. If the audio is longer than seconds, the audio was randomly truncated. If the audio is shorter than 6 seconds, the entire audio was duplicated until it becomes longer than 6 seconds. After then the duplicated audio sample was randomly truncated to be seconds. We applied root-mean-squan normalization make the overall amplitude of speech signals to be consistent. The reference level was selected as 0.01.
B IMPLEMENTATION DETAILS
We used stochastic gradient descent (SGD) with the momentum of 0.9 and weight decay of 5e 4 to optimize inference networks (speech encoder and face encoder). The learning rate was initialized to 0.001 and decayed by the factor of 10 if validation loss was not decreased for epoch. Training was stopped if learning rate decay occurred three times. Minibatch size was fixed to 32 and 12 for V-F and F-V training configuration, respectively. For both training and test phase, negative samples were randomly selected for every step while pre-defined negative samples for each positive sample were used for validation phase ensure stable learning rate decay scheduling.
13

---

Published as a conference paper at ICLR 2020
To train cGANs, used Adam optimizer using 81 and 31 of 0.9 and 0.9, respectively. The learning rate of generator and discriminator was fixed to 0.0001 and 0.00005 during training. respectively. updated once. Batch size is 24 and
Each time the discriminator was updated twice, the generator we trained the model about 500,000 iterations. Note that we adopted the truncation trick of Brook et al. (2019) in subsection 4.2 In all experiments except QTA and QTA 2, where the truncation trick was not adopted, the truncation threshold was set to 1.0.
C NETWORK STRUCTURES
Inference network consists of a speech encoder and face encoder. The network structure of the speech encoder is based the problem agnostic speech encoder (PASE) (Pascual et al. 2019) followed by an additional time pooling layer and a fully-connected layer (FC). PASE consists of SincNet and 7 stacks convolutional-
(ConyBlock) composed of Id-CNN, batch normalization, and multi-pane parametric rectified linear unit (PReLU) activation. Following PASE, average pooling on time dimension is applied to make the size of embedding time-invariant. The FC layer was used as the last layer of the speech encoder. The details of the speech encoder is depicted in Fig. 7 For face encoder, we used 2d-CNN based residual block (ResBlock) in each layer. Note that the network structure is similar to the discriminator network of (Miyato & Koyama 2018). The details of the architecture shown in Fig.
For the generator, we followed the generator network structure of (Miyato & Koyama 2018) with some modifications, such as concatenating and c as an input and adopting adaptive instance nor- malization (AdaIN) for the direct conditioning method. The details of the discriminator are almost same as the face encoder except that it includes additional FC layer, projection layer, and sig- moid as activation function. The details of the discriminator and generator are shown in Fig. 8 and Fig. 9 respectively.
S
SineNet, BatchNorm Id, PReLU
ConvBlock (64, 20, 10)
ConvBlock (128, 11, 2)
ConvBlock (128, 11,
ConvBlock (256, 11, 2)
ConvBlock (256, 11, 1)
ConvBllock (512, 11, 2)
ConvBlock (512, 11, 2)
Convid (0. K. S)
Conv 1d (100, 1, 1)
BatchNormld
Time pooling, ReLU
PReLU
FC (128)
(a) Speech Encoder
(b) ConvBlock for Speech Encoder
Figure 7: The structure the speech encoder. O, K. indicate the number of output channels, size of the kernel and stride, respectively
14

---

Published as a conference paper at ICLR 2020
lock, Down, 64
ResBlock, Down,
ResBlock, Down,
ResBlock, Down, 512
Face encoder
RexBlock, Down, 1024
ReLU
Conv 3x3
ResBlock, 1024
Conv 3x3
ReLU
Cony 1xl
Downsample
ReLU, Conv3x3,
ReLU
Conv 3x3
ReLU, GSP
Downsample
Conv
Conv 1xl
ReLU
FC
Downsample
Downsample
sigmotd
Discriminator
ResBlock for Discriminator
(c) InitResBlock
Figure 8: The structure of the discriminator network. The blue colored blocks indicate the network structure of the face encoder F which is transferred to the discriminator network at the generation stage. The numbers on each block denote the output channel. The GSP denotes global sum pooling along the spatial dimension.
ConcaL
FC, 1024x4x4
AdaIN
Yu.B
FC
Upsample
ReaBlock, Up. 1024
Upsample
Leaky ReLU
ResBlock, Up. 512
ResBlock, Up. 512
Conv 3x3
ResBlock, Up. 256
AdaIN
Yr.Ba
FC
Conv
ResBlock, Up. 128
Leaky ReLU
ResBlock, 64
Conv 3x3
Conv Conv 3x33
(a) Generator
(b) ResBlock for Generator
Figure 9: The structure of the generator network. The numbers each block denote the output channel.
15

---

Published as a conference paper at ICLR 2020
D SUPPLEMENTARY GENERATED IMAGES
Here we provide some additional uncurated generated face images. Fig. 10 and Fig. ated images with truncation threshold 0.5 and 1.0, respectively
show gener-
Randomly sampled 2:
Randomly sampled
Figure 10: The uncurated generated face images of 19 speakers from the test set of the AVSpeech. The very left column is composed of the ground truth face images of speakers.
Auddo samples are available in the following link:
16

---

Published as a conference paper at ICLR 2020
Randomly sampled
Randomly sampled Es
Figure 11: The uncurated generated face images of 19 speakers from the test set of the AVSpeech. The very left column is composed of the ground truth face images of speakers.
17

---

Published as a conference paper at ICLR 2020
E THE EPFECTIVENESS OF TWOGS TRAINING
In this subsection, we would like to address concerns regarding the validity of the two-stage training method proposed in Fig. 3 Here the effect of the two-stage training method is demonstrated by showing the failure case of training when we skip the inference training stage. Fig. 12 shows that the generator ignores the speech condition vector in the generation process and, consequently, the generated images are only dependent on the random vector 2. This shows that the learned represen- tation using the proposed self-supervised method effective and can be used as an important cue for the conditional generation.
Randomly sampled Zs
Randomly sampled cs
Figure 12: The generated face images when skipping the inference training stage. The very left column is composed of the ground truth face images of speakers.
18


Reviews:
[{"review_id": "rJeQ281nFr", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a multi-modal learning framework that links the inference stage and generation stage for seeking the possibility of generating the human face from voice solely based on the audio-visual data without any human-labeled annotations. Experimental results show that the proposed network can not only match the relationship between the human face and speech, but can also generate the high-quality human face sample conditioned on its speech.\n\nThe writing and presentation are clear.\n\nMy concerns are as below.\n1) What are the training computational complexity and testing time cost of the proposed method?\n2) How can we determine truncation threshold more elegant? Any theoretical analysis and sensitive analysis?\n3) How did the authors handle model collapse during training? \n4) The format of references should be consistent.", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}, {"review_id": "r1lI1bEZcH", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work aims to build one conditional face image generation framework from the audio signal. The whole framework is built on top of cGAN-based projector discriminator framework where the input condition vector is also used in the discriminator stage.  The authors compared with one recent work and demonstrated improvements on face retrieval experiment. \n\n\nQuestions:\n1\tIn terms of differences with previous approaches,  not sure if the self-supervised learning is one of them, since it is also applied in previous Speech2Face framework.\n2\tBased on the listed generated examples in Fig.2, most faces are frontal, especially along Z axis, not sure if the variation of Z can determine the head orientation. Faces in the third row and the forth may not keep the identity that well, especially the comparison between the face at the first column with others.   \n3\tOne typo may need to be addressed in the first paragraph of page 5, there are “the the” before word ‘inference’.\n4\tAs listed in Table 2, it seems the proposed approach achieves better performance when using 10 way training compared to other frameworks, any more analysis  why the proposed framework can achieve better performance in 10 way but obtain less accuracy in 2 way settings?   \n\n5\tDoes the test data of 2 way or 10 way experiments also include the same ratio of positive and negative pairs? If so, how about the performance on the same validation data?\n\n6\tAbout the results listed in Table 2, does the number 35^2 indicate 0.35? \n\n7.    It may be necessary to include these conditional face video generation works in the related work.\n\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "BJeqNfIsqB", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary - \nIn this work the authors propose a two-stage procedure for training a GAN which generates plausible faces conditioned on the raw waveform of a speech signal. In the first stage two embedding functions are trained, one taking as input a frame from a video (of a person speaking), the other taking as input the raw waveform of the same video's audio. These embeddings are trained to maximize the inner product of positively sampled embeddings (frame and audio from same video), and minimize the inner product of negatively sampled embeddings (frame and audio from different videos). In the second stage a GAN is trained where the input to the generator is the a random latent code (z), concatenated with the learned embedding of a speech signal from stage 1 (c). They also initialize the first half (\\phi) of the discriminator using the face embedding network from stage 1, and propose a modification of the relativistic GAN loss to prevent \\phi from losing the ability to produce face embeddings that have low inner product with the wrong speech embedding.\n\nThe authors explore the properties of their learned pipeline in a number of experiments. In 4.1 they compare their self-supervised speaker matching pipeline with prior work, showing that it has competitive performance with prior work that relies on either supervised pretraining, or additional labels such as age and gender. In particular they show that as the number of negative samples increases from 1 to 9, their identity matching performance significantly improves over prior work (in the K=2 regime their method underperforms prior work). Qualitatively they show that their generator has some reasonable success (it is hard to judge what perfect success would look like, and how far they are from reaching it) at disentangling aspects of facial appearence that can and cannot be inferred from the speech signal (QLA1). They also show qualitatively that the output of their generator can be smoothly controlled by interpolating between speech conditioning vectors, producing reasonable faces at each intermediary step (QLA2). In experiment QTA1 they quantitatively validate the conditioning vector c is affecting the generated image. In experiment QTA2 the authors (with a small issue, see weaknesses) show that their proposed modification to the loss function causes their outputs to be better matched to the speech conditioning as measured by the fixed embedding network trained in stage 1. Finally in QT3 they measure how well their generated faces can be used with the original face embedding network to perform image retrieval (I am a bit unclear on the details of this experiment, see questions)\n\n\nStrengths -\n* The authors' proposed pipeline and modified loss function offers a generalized framework for jumpstarting conditional GAN training\n* Their pipeline does not assume human-specified labels, but instead access to paired data from different modalities, which is can easily be obtained for many conditional image generation tasks (inpainting, super-resolution, colorization, etc.)\n* Their pipeline also doesn't seem particularly finetuned for the speech-driven image synthesis task, so it seems reasonable to believe it could be adapted to other tasks\n* Their results are qualitatively compelling, and they make a convincing efforts in experiments QTA1-3 to quantitatively show that the conditioning information is affecting the output\n* The paper is generally well-written and easy to follow\n\nWeaknesses - \n* Without completing the loop, and showing that the second stage of this pipeline helps with identity matching for speakers, I'm not clear on what the motivation for this particular form of conditional image generation is (this is unfortunate, because their framework is quite general, and it seems feasible that the authors could have applied it directly to a task where the output of conditional image generation is directly useful)\n* I interpret the main purpose of experiment QTA 2 as validating the effectiveness of their proposed loss modification, it seems like a natural experiment to make this claim more convincing is to compare a generator trained with Eq 4. against real images (hopefully getting a much lower matching probability than the 76.65% reported in the first experiment of this section)\n* It seems like an important ablation study is testing the effect of jumpstarting the GAN training with the pretrained networks of stage 1, and reporting what happens when one or both of these networks are initialized randomly (assuming that initializing randomly hurts performance significantly, this would be further evidence of their framework's strengths)\n* The novelty of the proposed approach is limited so far as I can tell (slight architectural modifications, and adding a negative sampling term to the discriminator loss)\n\nInitial Rating - Weak Accept\n\nQuestions - \n* In experiment QTA3 I'm confused by how including multiple faces from the same video clip affects measuring retrieval accuracy. Does retrieving any of the 50 faces from the same clip count as correct? \n\nExplanation of Rating - The novelty of the work is limited, and it doesn't seem clearly useful for any practical task. However the stated task is certainly a non-trivial one, and the qualitative results and experiments give compelling evidence that the authors are proposing a powerful framework for conditional image generation. I think that the high quality writing, experiments, and results; coupled with potential impact for other conditional image generation tasks warrant acceptance. However the paper is held back by the lack of novelty and lack of clear motivation.\n\nRevised Rating After Rebuttal: Accept\n\nExplanation of Revised Rating: The authors addressed my concrete concerns about missing experiments in the rebuttal. Despite my concerns about the motivation for this particular task, I think the good results produced by author's methodology indicate this work will be valuable in the context of conditional image generation more broadly. I am upgrading my rating to accept.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors propose a conditional GAN-based approach for generating faces consistent with given input speech.  The technical novelty is not large, as the approach is mainly putting together existing ideas, but the application is a fairly new one and the experiments and results are convincing.  The approach might also have broader applicability beyond this task.
Decision: Accept (Poster)