From Inference to Generation: End-to-end Fully Self-supervised Generation of Human Face from Speech

1 INTRODUCTION . Utilizing audio-visual cues together to recognize a person ’ s identity has been studied in various fields from neuroscience ( Hasan et al. , 2016 ; Tsantani et al. , 2019 ) to practical machine learning applications ( Nagrani et al. , 2018b ; a ; Wen et al. , 2019a ; Shon et al. , 2019 ) . For example , some neurological studies have found that in some cortical areas , humans recognize familiar individuals by combining signals from several modalities , such as faces and voices ( Hasan et al. , 2016 ) . In conjunction with the neurological studies , it is also a well known fact that a human speech production system is directly related to the shape of the vocal tract ( Mermelstein , 1967 ; Teager & Teager , 1990 ) . Inspired by the aforementioned scientific evidence , we would like to ask three related questions from the perspective of machine learning : 1 ) Is it possible to match the identity of faces and voices ? ( inference ) 2 ) If so , is it possible to generate a face image from a speech signal ? ( generation ) 3 ) Can we find the relationship between the two modalities only using cross-modal self-supervision with the data “ in-the-wild ” ? To answer these questions , we design a two-step approach where the inference and generation stages are trained sequentially . First , the two inference networks for each modality ( speech encoder and face encoder ) are trained to extract the useful features and to compute the cross-modal identity matching probability . Then the trained inference networks are transferred to the generation stage to pass the information about the speech , which helps the generation network to output the face image from the conditioned speech . We believe , however , that it is impossible to perfectly reconstruct all the attributes in the image of a person ’ s face through the characteristics of the voice alone . This is due to factors that are clearly * Equal contribution unrelated to one ’ s voice , such as lighting , glasses , and orientation , that also exist in the natural face image . To reflect the diverse characteristics presented in the face images “ in-the-wild ” , we therefore model the generation process by incorporating two latent factors into the neural network . More specifically , we adopted conditional generative adversarial networks ( cGANs ) ( Mirza & Osindero , 2014 ; Miyato & Koyama , 2018 ) so that the generator network can produce a face image that is dependent not only on the paired speech condition , but also on the stochastic variable . This allows the latent factors that contribute to the overall facial attributes to be disentangled into two factors : one that is relevant to the voice and the other that is irrelevant . Adopting cGANs negligently still leaves a few problems . For example , the condition in a cGANs framework is typically provided as embedded conditional vectors through the embedding look-up table for one-hot encoded labels ( Brock et al. , 2019 ; Miyato & Koyama , 2018 ) . The raw signals such as speech , however , can not be taken directly from the embedding look-up table , so an encoder module is required . Therefore , the trained speech encoder from the inference step is reused to output a pseudo conditional label that is used to extract meaningful information relevant to the corresponding face . Then the generator and the discriminator are trained in an adversarial way by utilizing the pseudo-embedded conditional vectors obtained from the trained speech encoder in the first step . Another problem with applying the conventional cGANs for generating faces from voice arises from the fact that the distinction between different speakers can be quite subtle , which calls for a need for a more effective conditioning method . To mitigate this problem , we propose a new loss function , relativistic identity cGANs ( relidGANs ) loss , with modification of the relativistic GANs ( JolicoeurMartineau , 2019 ) , allowing us to generate the face with a more distinct identity . Each step will be described in greater detail in Section 3 . Our contributions can be summarized as follows : 1 . We propose simple but effective end-to-end inference networks trained on audio-visual data without any labels in a self-supervised manner that perform a cross-modal identity matching task . 2 . A cGANs-based generation framework is proposed to generate the face from speech , to be seamlessly integrated with the trained networks from inference stage . 3 . A new loss function , so called a relidGANs loss , is designed to preserve a more consistent identity between the voices and the generated images . 4 . An extensive analysis is conducted on both inference and generation tasks to validate our proposed approaches . 2 RELATED WORKS . There has been an increasing interest in the self-supervised learning framework within the machine learning research community . While the focus of the many studies has been concentrated on applications of matching the audio-visual correspondence such as the presence or absence of objects in a video or a temporal alignment between two modalities ( Chung et al. , 2017 ; Afouras et al. , 2018b ; a ; Ephrat et al. , 2018 ) , growing attention has come to matching the speaker identity between the human face and voice . Inference The cross-modal identity matching task between face and voice has been recently studied in machine learning community . Nagrani et al . ( 2018b ) proposed a convolutional-neural-network ( CNN ) based biometric matching network by concatenating the two embedding vectors from each modality and classifying them with an additional classifier network . Though showing promising results , this style of training is limited as the model is not flexible in that the number of concatenated vectors used in the training can not be changed in the test phase . Next , Nagrani et al . ( 2018a ) modeled the concept of personal identity nodes ( Bruce & Young , 1986 ) by mapping each modality into the shared embedding space and using triplet loss to train the encoders to allow more flexible inference . Lastly , Wen et al . ( 2019a ) proposed DIMNet where two independent encoders are trained to map the speech and face into the shared embedding space , classifying each of them independently utilizing the supervised learning signals from labels such as identity , gender and nationality . Generation There have been a few concurrent works that tackled the similar problem of generating a face image from speech , which we think are worth noting here . Duarte et al . ( 2019 ) proposed a GANs-based framework to generate a face image from the speech . But the intention of their work was not to generate a face from unseen speaker identity but more of seeking the possibility of crossmodal generation between the speech and face itself . Oh et al . ( 2019 ) proposed a similar work called speech2face . The pre-trained face recognition network and the neural parametric decoder were used to produce a normalized face ( Parkhi et al. , 2015 ; Cole et al. , 2017 ) . After that , the speech encoder was trained to estimate the input parameter of the face decoder directly . Because the training process still requires the pre-trained modules trained with supervision , we do not consider this a fully-selfsupervised approach . Lastly , the most similar work to ours was recently proposed by Wen et al . ( 2019b ) where they utilized a pre-trained speech identification network as a speech encoder , and used a GANs-based approach to produce the face conditioned on the speech embedding . Our proposed method differs from the abovementioned approaches in the following ways . First , none of the previous approaches has tried to model the stochasticity in the generation process , but we address this problem by incorporating the stochasticity in the latent space so that the different face images can be sampled even when the speech condition is fixed . Second , modeling the image is more challenging in our work as we aim to train our network to produce larger image size ( 128 × 128 ) compared to other GANs-based works . Also , we trained the model on the AVSpeech dataset ( Ephrat et al. , 2018 ) which includes extremely diverse dynamics in the images . Finally , the important scope of this work is to seek the possibility of training the whole inference and generation stages only using the self-supervised learning method , which is the first attempt to our knowledge . The whole pipeline of the proposed approach is illustrated in Fig . 1 . 3 SELF-SUPERVISED INFERENCE AND GENERATION . 3.1 CROSS-MODAL IDENTITY MATCHING . In order to successfully identify the speaker identity from the two audio-visual modalities , we trained two encoders for each modality . First , a speech encoder is trained to extract the information related to a person ’ s identity from a segment of speech . To this end , we use raw-waveform based speech encoder that was shown to extract useful features in speaker verification task , outperforming conventional feature such as mel-frequnecy-cepstal-coefficient ( MFCC ) even when trained using selfsupervised learning in speech modality ( Pascual et al. , 2019 ) 1 . The first layer of the speech encoder is SincNet where the parameterized sinc functions act as trainable band-pass filters ( Ravanelli & Bengio , 2018 ) . The rest of the layers are composed of 7 stacks of 1d-CNN , batch normalization ( BN ) , and multi-parametric rectified linear unit ( PReLU ) activation . Next , we used a 2d-CNN based face encoder with residual connections in each layer . Note that the network structure is similar to the discriminator network of ( Miyato & Koyama , 2018 ) . The details of the networks are shown in Appendix C. Based on the speech encoder S ( · ) and the face encoder F ( · ) , they are trained to correctly identify whether the given face and speech is paired or not . Specifically , as a cross-modal identity matching task , we consider two settings as in ( Nagrani et al. , 2018b ) , 1 . V-F : given a segment of speech select one face out of K different faces , 2 . F-V : given an image of face select one speech segment out of K different speech segments . The probability that the j-th face fj is matched to the given speech s or vice versa is computed by the inner product of embedding vectors from each module followed by the softmax function as follows : 1 . V -F : p ( y = j| { S ( s ) , F ( fj ) } j=Kj=1 ) = p ( y = j| { se , fej } j=K j=1 ) = e < se , fej > ∑j=K j=1 e < se , fej > , 2 . F -V : p ( y = j| { S ( sj ) , F ( f ) } j=Kj=1 ) = p ( y = j| { sej , fe } j=K j=1 ) = e < sej , fe > ∑j=K j=1 e < sej , fe > , ( 1 ) where s denotes a speech segment , f denotes a face image , se denotes a speech embedding vector and fe denotes a face embedding vector . Note that using inner product as a means of computing similarity allows more flexible inference . For example , the proposed method enables both F-V and V-F settings , no matter with which method the model is trained . Also , it allows setting a different number of K in the test phase than the training phase . Finally , the two encoders are trained solely using the self-supervised learning signal from crossentropy error .

Summary: In this work the authors propose a two-stage procedure for training a GAN which generates plausible faces conditioned on the raw waveform of a speech signal. In the first stage two embedding functions are trained, one taking as input a frame from a video (of a person speaking), the other taking as input the raw waveform of the same video's audio. These embeddings are trained to maximize the inner product of positively sampled embeddings (frame and audio from same video), and minimize the inner product of negatively sampled embeddings (frame and audio from different videos). In the second stage a GAN is trained where the input to the generator is the a random latent code (z), concatenated with the learned embedding of a speech signal from stage 1 (c). They also initialize the first half (\phi) of the discriminator using the face embedding network from stage 1, and propose a modification of the relativistic GAN loss to prevent \phi from losing the ability to produce face embeddings that have low inner product with the wrong speech embedding.