Disjoint Mapping Network for Cross-modal Matching of Voices and Faces

1 INTRODUCTION . A person ’ s face is predictive of their voice . Biologically , the genetic , physical and environmental influences that affect the face also affect the voice . Humans have been shown to be able to associate voices of unknown individuals to pictures of their faces ( Kamachi et al. , 2003 ) . Humans also show improved ability to memorize and recall voices when previously exposed to pictures of the speaker ’ s face , but not imposter faces ( McAllister et al. , 1993 ; Schweinberger et al. , 2007 ; 2011 ) . Cognitively , studies indicate that neuro-cognitive pathways for voices and faces share common structure ( Ellis , 1989 ) , possibly following parallel pathways within a common recognition framework ( Belin et al. , 2004 ; 2011 ) . The above studies lend credence to the hypothesis that it may be possible to find associations between voices and faces algorithmically as well . With this in perspective , this paper focuses on the task of devising computational mechanisms for cross-modal matching of voice recordings and images of the speakers ’ faces . The specific problem we look at is the one wherein we have an existing database of samples of people ’ s voices and images of their faces , and we aim to automatically and accurately determine which voices match to which faces . This problem has seen significant research interest , in particular since the recent introduction of the VoxCeleb corpus ( Nagrani et al. , 2017 ) , which comprises collections of video and audio recordings of a large number of celebrities . The existing approaches ( Nagrani et al. , 2018b ; a ; Kim et al. , 2018 ) have generally attempted to directly relate subjects ’ voice recordings and their face images , in order to find the correspondences between the two . Nagrani et al . ( 2018b ) formulates the mapping as a binary selection task : given a voice recording , one must successfully select the speaker ’ s face from a pair of face images ( or the reverse – given a face image , one must correctly select the subject ’ s voice from a pair of voice recordings ) . They model the mapping as a neural network that is trained through joint presentation of voices and faces to determine if they belong to the same person . In Kim et al . ( 2018 ) ; Nagrani et al . ( 2018a ) , the authors attempt to learn common embeddings ( i.e. , vector representations ) for voices and faces that can be compared to one another to identify associations . The networks that compute the embeddings are also trained through joint presentation of voices and faces , to maximize the similarity of embeddings derived from them if they belong to the same speaker . In all cases , the voice and face are implicitly assumed to directly inform about one another . In reality , though , it is unclear how much these models capture the direct influence of the voice and face on one another , and how much is explained through implicit capture of higher-level variables such as gender , age , ethnicity etc. , which individually predict the two . These higher-level variables , C on tr as tiv e L os s T ri pl et L os s M ul tita sk C la ss ifi ca tio n Multiple Covariates Supervision ( a ) Seeing Faces and Hearing Voices ( b ) Learnable PINs ( c ) Learning Face-Voice Association Gender Identity Nationality ( d ) The Proposed DIMNet Pa ir C on st ru ct io n T r ip le t C on st ru ct io n Positive Pair Positive Pair Negative Pair Negative Pair B in ar y C la ss ifi ca tio n Concat Match or not ? Pa ir C on st ru ct io n Embeddings of Single Face or Voice as Input No Need for Pair or Triplet Construction ... ... ... ... ...... ... ... .. . ...... ... ... Identity 1 Identity 2 Identity 3 Voice Embedding Face Embedding Direct dependence between voices and faces Dependence between different modalities and the covariates Figure 1 : Overview of the proposed DIMNet and its comparison to the existing approaches . ( a ) Seeing faces and hearing voices from Nagrani et al . ( 2018b ) . ( b ) Learnable PINs from Nagrani et al . ( 2018a ) . ( c ) Learning face-voice association from Kim et al . ( 2018 ) . ( d ) Our proposed DIMNets . DIMNets present a joint voice-face embedding framework via multi-task classification and require no pair construction ( i.e. , both voices and faces can be input sequentially without forming pairs ) . which we will refer to as covariates1 can , in fact , explain much of our ability to match voices to faces ( and vice versa ) under the previously mentioned “ select-from-a-pair ” test ( where a voice must be used to distinguish the speaker ’ s face from a randomly-chosen imposter ) . For instance , simply matching the gender of the voice and the face can result in an apparent accuracy of match of up to 75 % in a gender-balanced testing setting . Even in a seemingly less constrained “ verification ” test , where one must only verify if a given voice matches a given face , matching them based on gender alone can result in an equal error rate of 33 % ( Appendix B ) . Even matching the voice and the face by age ( e.g.matching older-looking faces to older-sounding voices ) could result in match accuracy that ’ s significantly better than random . Previous studies ( Nagrani et al. , 2018b ; Kim et al. , 2018 ) attempt to disambiguate the effect of multiple covariates through stratified tests that separate the data by covariate value . The results show that at least some of the learned associations are explained by the covariate , indicating that their learning approaches do utilize the covariate information , albeit only implicitly . In this paper , we propose a novel framework to learn mappings between voices and faces that do not consider any direct dependence between the two , but instead explicitly exploit their individual dependence on the covariates . We define covariate as the identity-sensitive factors that can simultaneously affect voice and face , e.g . nationality , gender , identity ( ID ) , etc . We do not require the value these factors take to be the same between the training and test set , since what we are learning is the nature of the covariation with the variable in general , not merely the covariation with the specific values the variable takes in the training set . In contrast to existing methods where supervision is provided through the correspondence of voices and faces , our learning framework , Disjoint Mapping Network ( DIMNet ) , obtains supervision from common covariates , applied separately to voices and faces , to learn common embeddings for the two . The comparison between the existing approaches and DIMNets are illustrated in Fig . 1 . DIMNet comprises individual feature learning modules which learn identically-dimensioned features for data from each modality , and a unified input-modality-agnostic classifier that attempts to predict covariates from the learned feature . Data from each modality are presented separately during learning ; however the unified classifier forces the feature representations learned from the individual modalities to be comparable . Once trained , the classifier can be removed and the learned feature representations are used to compare data across modalities . The proposed approach greatly simplifies the learning process and , by considering the modalities individually rather than as coupled pairs , makes much more effective use of the data . Moreover , if multiple covariates are known , they can be simultaneously used for the training through multi-task learning in our framework ( see Fig . 2 ) . Compared to current methods ( Nagrani et al. , 2018b ; a ; Kim et al. , 2018 ) , DIMNets achieve consistently better performance , indicating that direct supervision through covariates is more effective in 1To be clear , these are covariates , factors that vary jointly with voice and face , possibly due to some other common causative factors such as genetics , environment , etc . They are usually not claimed to be causative factors themselves . these settings . We find that of all the covariates , ID provides the strongest supervision . The results obtained from supervision through other covariates also match what may be expected . Our contributions are summarized as follows : • We propose DIMNets , a framework that formulates the problem of cross-modal matching of voices and faces as learning common embeddings for the two through individual supervision from one or more covariates , in contrast to current approaches that attempt to map voices to faces directly . An overview of our framework is given in Fig . 2 . • In this framework , we can make full use of multiple kinds of label information ( provided by covariates ) with a multi-task objective function . • We achieve the state-of-the-art results on multiple tasks . We are also able to isolate and analyze the effect of the individual covariate on the performance . Moreover , we note that the proposed framework is applicable in any setting where matching of different types of data which have common covariates is required . 2 THE PROPOSED FRAMEWORK . Our goal is to learn common vector representations for both voices and faces , that permit them to be compared to one another . In the following sections we first describe how we learn them from their relationship to common covariates . Subsequently , we describe how we will use them for comparison of voices to faces . 2.1 LEVERAGING COVARIATES TO LEARN EMBEDDINGS . The relationship between voices and faces is largely predicted by covariates – factors that individually relate to both the voice and the face . To cite a trivial example , a person ’ s gender relates their voice to their face : male subjects will have male voices and faces , while female subjects will have female voices and faces . More generally , many covariates may be found that relate to both voice and face ( Lippert et al. , 2017 ) . Our model attempts to find common representations for both face images and voice recordings by leveraging their relationship to these covariates ( rather than to each other ) . We will do so by attempting to predict covariates from voice and face data in a common embedding space , such that the derived embeddings from the two types of data can be compared to one another . Let V represent a set of voice recordings , and F represent a set of face images . Let C be the set of covariates we consider . For the purpose of this paper , we assume that all covariates are discrete valued ( although this is not necessary ) . Every voice recording in V and every face in F can be related to each of the covariates in C. For every covariate C ∈ C we represent the value of that covariate for any voice recording v as C ( v ) , and similarly the value of the covariate for any face f as C ( f ) . For example , C could be ID , gender , or nationality . When C is ID , C ( v ) and C ( f ) are the ID of voice v and face f , respectively . Let Fv ( v ; θv ) : v 7→ Rd be a voice embedding function with parameters θv that maps any voice recording v into a d-dimensional vector . Similarly , let Ff ( f ; θf ) be a face embedding function that maps any face f into a d-dimensional vector . We aim to learn θv and θf such that the embeddings of the voice and face for any person are comparable . For each covariate C ∈ C we define a classifier HC ( x ; φC ) with parameter φC , which assigns any input x ∈ Rd to one of the values taken by C. The classifier HC ( · ) is agnostic to which modality its input x was derived from ; thus , given an input voice v , it operates on features Fv ( v ; θv ) derived from the voice , whereas given a face f , it operates on Ff ( f ; θf ) . For each v ( or f ) and each covariate C , we define a loss L ( HC ( Fv ( v ; θv ) ; φC ) , C ( v ) ) between the covariate predicted by HC ( . ) and the true value of the covariate for v , C ( v ) . We can now define a total loss L over the set of all voices V and the set of all faces F , over all covariates as L ( θv , θf , { φC } ) = ∑ C∈C λC ( ∑ v∈V L ( HC ( Fv ( v ; θv ) ; φC ) , C ( v ) ) + ∑ f∈F L ( HC ( Ff ( f ; θf ) ; φC ) , C ( f ) ) ) ( 1 ) In order to learn the parameters of the embedding functions , θf and θv , we perform the following optimization . θ∗v , θ ∗ f = arg min θv , θf min { φC } L ( θv , θf , { φC } ) ( 2 )

Summary: This paper aims at matching people's voices to the images of their faces. It describes a method to train shared embeddings of voices and face images. The speech and image features go through separate neural networks until a shared embedding layer. Then a classification network is built on top of the embeddings from both networks.  The classification network predicts various combinations of covariates of faces and voices: gender, nationality, and identity.  The input to the classification network is then used as a shared representation for performing retrieval and matching tasks.