Title: DISJOINT MAPPING NETWORK FOR CROSS-MODAL MATCHING OF VOICES AND FACES

Abstract: We propose a novel framework, called Disjoint Mapping Network (DIMNet), for cross-modal biometric matching, in particular of voices and faces. Different from the existing methods, DIMNet does not explicitly learn the joint relationship between the modalities. Instead, DIMNet learns a shared representation for different modalities by mapping them individually to their common covariates. These shared representations can then be used to find the correspondences between the modalities. We show empirically that DIMNet is able to achieve better performance than the current state-of-the-art methods, with the additional benefits of being conceptually simpler and less data-intensive. The code is made available at https://github.com/ydwen/DIMNet.

1. INTRODUCTION:

A person's face is predictive of their voice. Biologically, the genetic, physical and environmental influences that affect the face also affect the voice. Humans have been shown to be able to associate voices of unknown individuals to pictures of their faces (Kamachi et al., 2003) . Humans also show improved ability to memorize and recall voices when previously exposed to pictures of the speaker's face, but not imposter faces (McAllister et al., 1993; Schweinberger et al., 2007; 2011) . Cognitively, studies indicate that neuro-cognitive pathways for voices and faces share common structure (Ellis, 1989) , possibly following parallel pathways within a common recognition framework (Belin et al., 2004; 2011) . The above studies lend credence to the hypothesis that it may be possible to find associations between voices and faces algorithmically as well. With this in perspective, this paper focuses on the task of devising computational mechanisms for cross-modal matching of voice recordings and images of the speakers' faces.

The specific problem we look at is the one wherein we have an existing database of samples of people's voices and images of their faces, and we aim to automatically and accurately determine which voices match to which faces. This problem has seen significant research interest, in particular since the recent introduction of the VoxCeleb corpus (Nagrani et al., 2017) , which comprises collections of video and audio recordings of a large number of celebrities. The existing approaches (Nagrani et al., 2018b ;a; Kim et al., 2018) have generally attempted to directly relate subjects' voice recordings and their face images, in order to find the correspondences between the two. Nagrani et al. (2018b) formulates the mapping as a binary selection task: given a voice recording, one must successfully select the speaker's face from a pair of face images (or the reverse -given a face image, one must correctly select the subject's voice from a pair of voice recordings). They model the mapping as a neural network that is trained through joint presentation of voices and faces to determine if they belong to the same person. In Kim et al. (2018) ; Nagrani et al. (2018a) , the authors attempt to learn common embeddings (i.e., vector representations) for voices and faces that can be compared to one another to identify associations. The networks that compute the embeddings are also trained through joint presentation of voices and faces, to maximize the similarity of embeddings derived from them if they belong to the same speaker. In all cases, the voice and face are implicitly assumed to directly inform about one another.

In reality, though, it is unclear how much these models capture the direct influence of the voice and face on one another, and how much is explained through implicit capture of higher-level variables such as gender, age, ethnicity etc., which individually predict the two. These higher-level variables, which we will refer to as covariates1 can, in fact, explain much of our ability to match voices to faces (and vice versa) under the previously mentioned "select-from-a-pair" test (where a voice must be used to distinguish the speaker's face from a randomly-chosen imposter). For instance, simply matching the gender of the voice and the face can result in an apparent accuracy of match of up to 75% in a gender-balanced testing setting. Even in a seemingly less constrained "verification" test, where one must only verify if a given voice matches a given face, matching them based on gender alone can result in an equal error rate of 33% (Appendix B). Even matching the voice and the face by age (e.g.matching older-looking faces to older-sounding voices) could result in match accuracy that's significantly better than random.

Previous studies (Nagrani et al., 2018b; Kim et al., 2018) attempt to disambiguate the effect of multiple covariates through stratified tests that separate the data by covariate value. The results show that at least some of the learned associations are explained by the covariate, indicating that their learning approaches do utilize the covariate information, albeit only implicitly.

In this paper, we propose a novel framework to learn mappings between voices and faces that do not consider any direct dependence between the two, but instead explicitly exploit their individual dependence on the covariates. We define covariate as the identity-sensitive factors that can simultaneously affect voice and face, e.g. nationality, gender, identity (ID), etc. We do not require the value these factors take to be the same between the training and test set, since what we are learning is the nature of the covariation with the variable in general, not merely the covariation with the specific values the variable takes in the training set. In contrast to existing methods where supervision is provided through the correspondence of voices and faces, our learning framework, Disjoint Mapping Network (DIMNet), obtains supervision from common covariates, applied separately to voices and faces, to learn common embeddings for the two. The comparison between the existing approaches and DIMNets are illustrated in Fig. 1 .

DIMNet comprises individual feature learning modules which learn identically-dimensioned features for data from each modality, and a unified input-modality-agnostic classifier that attempts to predict covariates from the learned feature. Data from each modality are presented separately during learning; however the unified classifier forces the feature representations learned from the individual modalities to be comparable. Once trained, the classifier can be removed and the learned feature representations are used to compare data across modalities.

The proposed approach greatly simplifies the learning process and, by considering the modalities individually rather than as coupled pairs, makes much more effective use of the data. Moreover, if multiple covariates are known, they can be simultaneously used for the training through multi-task learning in our framework (see Fig. 2 ). ...

Compared to current methods (:



Modality Switch:

Figure 2 : Our DIMNet framework. The input training data can be either voice or face, and there is no need for voices and faces to form pairs. Modality switch is to control which embedding network (voice or face) to process the data. While the embeddings are obtained, a multi-task classification network is applied to supervise the learning.

these settings. We find that of all the covariates, ID provides the strongest supervision. The results obtained from supervision through other covariates also match what may be expected.

Our contributions are summarized as follows:

• We propose DIMNets, a framework that formulates the problem of cross-modal matching of voices and faces as learning common embeddings for the two through individual supervision from one or more covariates, in contrast to current approaches that attempt to map voices to faces directly. An overview of our framework is given in Fig. 2 . • In this framework, we can make full use of multiple kinds of label information (provided by covariates) with a multi-task objective function. • We achieve the state-of-the-art results on multiple tasks. We are also able to isolate and analyze the effect of the individual covariate on the performance.

Moreover, we note that the proposed framework is applicable in any setting where matching of different types of data which have common covariates is required.

2. THE PROPOSED FRAMEWORK:

Our goal is to learn common vector representations for both voices and faces, that permit them to be compared to one another. In the following sections we first describe how we learn them from their relationship to common covariates. Subsequently, we describe how we will use them for comparison of voices to faces.

2.1. LEVERAGING COVARIATES TO LEARN EMBEDDINGS:

The relationship between voices and faces is largely predicted by covariates -factors that individually relate to both the voice and the face. To cite a trivial example, a person's gender relates their voice to their face: male subjects will have male voices and faces, while female subjects will have female voices and faces. More generally, many covariates may be found that relate to both voice and face (Lippert et al., 2017) .

Our model attempts to find common representations for both face images and voice recordings by leveraging their relationship to these covariates (rather than to each other). We will do so by attempting to predict covariates from voice and face data in a common embedding space, such that the derived embeddings from the two types of data can be compared to one another.

Let V represent a set of voice recordings, and F represent a set of face images. Let C be the set of covariates we consider. For the purpose of this paper, we assume that all covariates are discrete valued (although this is not necessary). Every voice recording in V and every face in F can be related to each of the covariates in C. For every covariate C ∈ C we represent the value of that covariate for any voice recording v as C(v), and similarly the value of the covariate for any face f as C(f ). For example, C could be ID, gender, or nationality. When C is ID, C(v) and C(f ) are the ID of voice v and face f , respectively.

Let F v (v; θ v ) : v → R d be a voice embedding function with parameters θ v that maps any voice recording v into a d-dimensional vector. Similarly, let F f (f ; θ f ) be a face embedding function that maps any face f into a d-dimensional vector. We aim to learn θ v and θ f such that the embeddings of the voice and face for any person are comparable.

For each covariate C ∈ C we define a classifier H C (x; φ C ) with parameter φ C , which assigns any input x ∈ R d to one of the values taken by C. The classifier H C (•) is agnostic to which modality its input x was derived from; thus, given an input voice v, it operates on features F v (v; θ v ) derived from the voice, whereas given a face f , it operates on F f (f ; θ f ).

For each v (or f ) and each covariate C, we define a loss

L(H C (F v (v; θ v ); φ C ), C(v))

between the covariate predicted by H C (.) and the true value of the covariate for v, C(v). We can now define a total loss L over the set of all voices V and the set of all faces F, over all covariates as

EQUATION

In order to learn the parameters of the embedding functions, θ f and θ v , we perform the following optimization. The training data comprise voice recordings and face images. Voice recordings are sent to the voiceembedding network, while face images are sent to the face-embedding network. This switching operation is illustrated by the switch at the input in Fig. 2 . In either case, the output of the embedding network is sent to the covariate classifiers.

θ * v , θ * f = arg min θv ,θ f min {φ C } L(θv, θ f , {φC }) (2) 2.2 DISJOINT MAPPING NETWORKS In DIMNet, we instantiate F v (v; θ v ), F f (f ; θ f ) and H C (x; φ C ) as neural networks.

As can be seen, at any time the system either operates on a voice, or on a face, i.e.the operations on voices and faces are disjoint. During the learning phase too, the updates of the two networks are disjoint -loss gradients computed when the input is voice only update the voice network, while loss gradients derived from face inputs update the face network, while both contribute to updates of the classification networks.

In our implementation, specifically, F v (•) is a convolutional neural network that operates on Mel-Spectrographic representations of the speech signal. The output of the final layer is pooled over time to obtain a final d-dimensional representation. F f (•) is also a convolutional network with a pooled output at the final layer that produces a d-dimensional representation of input images. The classifiers H C (•) are all simple multi-class logistic-regression classifiers comprising a single softmax layer.

Finally, in keeping with the standard paradigms for training neural network systems, we use the cross-entropy loss to optimize the networks. Also, instead of the optimization in Eq. 2, the actual optimization performed is the one below. The difference is inconsequential.

EQUATION

2.3. TRAINING THE DIMNET:

All parameters of the network are trained through backpropagation, using stochastic gradient descent. During training, we construct the minibatches with a mixture of speech segments and face images, as the network learns more robust cross-modal features with mixed inputs. Taking voice as an example, we compute the voice embeddings using F v (v; θ v ), and obtain the losses using classifiers H C (•) for all the covariates. We back-propagate the loss gradient to update the voice network as well as the covariate classifiers. The same procedure is also applied to face data: the backpropagated loss gradients are used to update the face network and the covariate classifiers. Thus, the embedding functions are learned using the data from their modalities individually, while the classifiers are learned using data from all modalities.

2.4. USING THE EMBEDDINGS:

Once trained, the embedding networks F v (v; θ v ) and F f (f ; θ f ) can be used to extract embeddings from any voice recording or face image. Given a voice recording v and a face image f , we can now compute a similarity between the two through the cosine similarity S(v, f ) =

F v F f Fv 2 F f 2 .

We can employ this similarity to evaluate the match of any face image to any voice recording. This enables us, for instance, to attempt to rank a collection of faces f 1 , • • • , f K in order of estimated match to a given voice recording v, according to S(v, f i ), or conversely, to rank a collection of voices v 1 , • • • , v K according to their match to a face f , on order of decreasing S(v i , f ).

3. EXPERIMENTS:

We ran experiments on matching voices to faces, to evaluate the embeddings derived by DIMNets. The details of the experiments are given below and Appendix A. Training. The detailed network configurations are elaborated in appendix A.3. Note that the classification networks are single-layer softmax units with as many outputs as the number of unique values the class can take (2 for gender, 32 for nationalities, and 924 for IDs in our case). The networks are trained to minimize the cross entropy loss, following the typical settings of stochastic gradient descent (SGD) in appendix A.3

Testing. We use the following protocols for evaluation:

• 1:2 Matching. Here, we are given a probe input from one modality (voice or face), and a gallery of two inputs from the other modality (face or voice), including one that belongs to the same subject as the probe, and another of an "imposter" that does not match the probe. The task is to identify which entry in the gallery matches the probe. We report performance in terms of matching accuracy -namely what fraction of the time we correctly identify the right instance in the gallery.

To minimize the influence of random selection, we construct as many testing instances as possible through exhaustive enumeration all positive matched pairs (of voice and face). To each pair, we include a randomly drawn imposter in the gallery. We thus have a total of 4,678,897 trials in the validation set, and 6,780,750 trials in the test set.

• 1:N Matching. This is the same as the 1:2 matching, except that the gallery now includes N -1 imposters. Thus, we must now identify which of the N entries in the gallery matches the probe.

Here too results are reported in terms of matching accuracy. We use the same validation and test sets as the 1:2 case, by augmenting each trial with N -2 additional imposters. So the number of trials in validation and test sets is the same as earlier.

• Verification. We are given two inputs, one a face, and another a voice. The task is to determine if they are matched, i.e.both belong to the same subject. In this problem setting the similarity between the two is compared to a threshold to decide a match. The threshold can be adjusted to trade off false rejections (F R ), i.e.wrongly rejecting true matches, with false alarms (F A ), i.e.wrongly accepting mismatches. We report results in terms of equal error rate, i.e.when F R = F A . We construct our validation and test sets from those used for the 1:2 matching tests, by separating each trial into two, one comprising a matched pair, and the other a mismatched pair. Thus, our validation and test sets are exactly twice as large as those for the 1:2 test.

• Retrieval. The gallery comprises a large number of instances, one or more of which might match the probe. The task is to order the gallery such that the entries in the gallery that match the probe lie at the top of the ordering. Here, we report performance in terms of Mean Average Precision (MAP) (Manning et al., 2008 ). Here we use the entire collection of 58,420 test faces as the gallery for each of our 21,799 test voices, when retrieving faces from voices. For the reverse (retrieving voices from faces), the numbers are reversed.

Each result is obtained by averaging the performances of 5 models, which are individually trained.

Covariates in Training and Testing. We use the three covariates provided in the dataset, namely identity (I), gender (G), and nationality (N) for our experiments. The treatment of covariates differs for training and test.

• Training. For training, supervision may be provided by any set of (one two or three) covariates.

We consider all combinations of covariates, I, G, N, (I,G), (I,N), (G,N) and (I,G,N). Increasing the number of covariates effectively increases the supervision provided to training. All chosen covariates were assigned a weight of 1.0. • Testing. As explained in Appendix B, simply recognizing a covariate such as gender can result in seemingly significant matching performance. For instance, just recognizing the subjects' gender from their voice and images can result in a 33% EER for verification, and 25% error in matching for the 1 : 2 tests. In order to isolate the effect of covariates on performance hence we also stratify our test data by them. Thus we construct 4 testing groups based on the covariates, including the unstratified (U) group, stratified by gender (G), stratified by nationality (N), and stratified by gender and nationality (G, N). In each group the test set itself is separated into multiple strata, such that for all instances within any stratum the covariate values are the same. In this section we report results on the 1:2 and 1:N matching tests. In order to ensure that the embedding networks do indeed leverage on accurate modelling of covariates, we first evaluate the classification accuracy of the classification networks for the covariates themselves. Table 1 shows the results.

3.1. CROSS-MODAL MATCHING:

The rows of the table show the covariates used to supervise the learning. Thus, for instance, the row labelled "DIMNet-I" shows results obtained when the networks have been trained using ID alone as covariate, the row labelled "DIMNet-G" shows results when supervision is provided by gender, "DIMNet-IG" has been trained using ID and gender, etc.

The columns of the table show the specific covariate being evaluated. Since the identities of subjects in the training and test set do not overlap, we are unable to evaluate the accuracy of ID classification. Note that we can only test the accuracy of the classification network for a covariate if it has been used in the training. Thus, classification accuracy for gender can be evaluated for DIMNet-G, DIMNet-GN and DIMNet-IGN, while that for nationality can be evaluated for DIMNet-N, DIMNet-GN and DIMNet-IGN.

The results in Table 1 show that gender is learned very well, and in all cases gender recognition accuracy is quite high. Nationality, on the other hand, is not a well-learned classifier, presumably because the distribution of nationalities in the data set is highly skewed (Nagrani et al., 2018b) , with nearly 65% of all subjects belonging to the USA. It is to be expected therefore that nationality as a covariate will not provide sufficient supervision to learn good embeddings.

1:2 matching. Table 2 shows the results for the 1:2 matching tests. In the table, the row labelled "SVHF-Net" gives results obtained with the model of Nagrani et al. (2018b) .

The columns are segregated into two groups, one labelled "voice → face" and the other labelled "face → voice". In the former, the probe is a voice recording, while the gallery comprises faces.

In the later the modalities are reversed. Within each group the columns represent the stratification of the test set. "U" represents test sets that are not stratified, and include the various covariates in the same proportion that they occur in the overall test set. The columns labelled "G" and "N" have been stratified by gender and nationality, respectively, while the column "G, N" represents data that have been stratified by both gender and nationality. In the stratified tests, we have ensured that all data within a test instance have the same value for the chosen covariate. Thus, for instance, in a test instance for voice → face in the "G" column, the voice and both faces belong to the same gender. This does not reduce the overall number of test instances, since it only requires ensuring that the gender of the imposter matches that of the probe instance.

We make several observations. First, DIMNet-I performs better than SVHF-Net, improving the accuracies by 2.45%-4.02% for the U group, and 7.01%-8.38% for the G group. It shows that mapping voices and faces to their common covariates is an effective strategy to learn representations for cross-modal matching.

Second, DIMNet-I produces significantly better embeddings that DIMNet-G and DIMNet-N, highlighting the rather unsurprising fact that ID provides the more useful information than the other two covariates. In particular, DIMNet-G respectively achieves 72.90% and 72.47% for voice to face and face to voice matching using only gender as a covariate. This verifies our hypothesis that we can achieve almost 75% matching accuracy by only using the gender. These numbers also agree with the performance expected from the numbers in Table 1 and the analysis in Appendix B. As expected, nationality as a covariate does not provide as good supervision as gender. DIMNet-IG is marginally better than DIMNet-I, indicating that gender supervision provides additional support over ID alone.

Third, we note that while DIMNet-I is able to achieve good performance on the dataset stratified by gender, DIMNet-G only achieves random performance. The performance achieved by DIMNet-G on the U dataset is hence completely explained by gender matching. Once again, the numbers match our expectations (Appendix B). All the results in Fig. 3 are consistent with Table 2 . As expected, the performance of all methods degrades with increasing N . In general, DIMNets that use ID as supervision outperform SVHF-Net by a considerable margin, showing that DIMNets are able to make best use of the ID information. We obtain the best results when both ID and gender are used as supervision covariates. However, The results obtained using only gender information as covariate is much worse, which is also consistent with our analysis in Appendix B. For verification, we need to determine whether an audio segment and a face image are from the same ID or not. We report the equal error rate (EER) for verification in Table 3 .

In general, DIMNets that use ID as a covariate achieve an EER of about 25%, which is considerably lower than the 33% expected if the verification were based on gender matching alone. The results in Table 3 show that using both gender and ID information as covariates can further improve the performance over using ID alone, well validating the superiority of our multi-task learning framework. Using proper combination of covariates is crucial to the performance. ID is arguably the most effective covariate supervision. More interestingly, nationality is seen to be an ineffective covariate, while gender alone as a covariate produces results that well matches our expectation. We also perform retrieval experiments using voice or face as query. Table 4 lists the mean average precision (mAP) of the retrieval for various models.

3.3. CROSS-MODAL RETRIEVAL:

The columns in the table represent the covariate being retrieved. Thus, for example, in the "ID" column, the objective is to retrieve gallery items with the same ID as the query, whereas in the "gender" column the objective is to retrieve the same gender.

We note that ID-based DIMNets produce the best features for retrieval, with the best performance obtained with DIMNet-IG. Also, as may be expected, the covariates used in training result in the best retrieval of that covariate. Thus, DIMNet-G achieves an mAP of nearly 98% on gender, though on retrieval of ID it is very poor. As in other experiments, nationality remains a poor covariate in general. Compared to gender (2 classes) and nationality (unbalanced 28 classes), retrieving ID is a challenging problem given the large amount of identities (182 classes). The significant and consistent improvements over chance-level results show that the DIMNet models do learn some useful associations between voices and faces. We compare DIMNet with the state of the art (Nagrani et al., 2018a) . The results are reported in Table 5 . Note that it is fair comparison because the DIMNet models in this section are trained with and evaluated on the same released datasets in Nagrani et al. (2018a) . Detailed statistics and splits of the dataset can be found in Appendix A.1. There are two evaluation protocols, including Seen-Heard and Unseen-Unheard scenarios. The identities of the training and testing set have overlaps in Seen-Heard scenario (closed-set), while they are fully disjoint in Unseen-Unheard scenario (openset). For each scenario, there are 5 testing groups based on the covariates, including the unstratified group (U), group, stratified by gender (G), stratified by nationality (N), stratified by age (A), and stratified by (G, N, A). We compute the area under the curve (AUC) for different testing groups.

3.4. COMPARISONS TO THE CURRENT STATE-OF-THE-ART:

Seen-Heard Unseen-Unheard U G N A G, N, A U G N A G,

It is clear that DIMNets produce better embeddings than Nagrani et al. (2018a) for pair-wise verification on both seen-heard and unseen-unheard scenarios. Specifically, DIMNets achieve 8%-15% absolute and 3%-10% absolute improvements on seen-heard and unseen-unheard test set, respectively.

Compared to DIMNet-IG, DIMNet-I performs better on the seen-heard test set while DIMNet-IG is better on the unseen-unheard test set. It implies that introducing useful covariates improves the generalization capability of DIMNet.

4. DISCUSSIONS AND CONCLUDING REMARKS:

We have proposed that it is possible to learn common embeddings for multi-modal inputs, particularly voices and faces, by mapping them individually to common covariates. In particular, the proposed DIMNet architecture is able to extract embeddings for both modalities that achieves consistently better performance than the methods that directly map faces to voices.

The approach also provides us the ability to tease out the influence of each of the covariates of voice and face data, in determining their relation. The results show that the strongest covariate, not unexpectedly, is ID. The results also indicate that prior results by other researchers who have attempted to directly match voices to faces may perhaps not be learning any direct relation between the two, but implicitly learning about the common covariates, such as ID, gender, etc.

Our experiments also show that although we have achieved possibly the best reported performance on this task, thus far, the performance is not anywhere close to prime-time. In the 1 : N matching task, performance degrades rapidly with increasing N , indicating a rather poor degree of true match.

To better understand the problem, we have visualized the learned embeddings from DIMNet-I in Fig. 4 to provide more insights. The visualization method we used is multi-dimensional scaling (MDS) (Wickelmaier, 2003) , rather than the currently more popular t-SNE (van der Maaten & Hinton, 2008). This is because MDS tends to preserve distances and global structure, while t-SNE attempts to retain statistical properties and highlights clusters, but does not preserve distances.

From Fig. 4 , we immediately notice that the voice and face data for a subject are only weakly proximate. While voice and face embeddings for a speaker are generally relatively close to each other, they are often closer to other subjects. Interestingly, the genders separate (even though gender has not been used as a covariate for this particular network), showing that at least some of the natural structure of the data is learned. Fig. 4 shows embeddings obtained from both training and test data. We can observe similar behaviors in both, showing that the the general characteristics observed are not just the outcome of overfitting to training data. The visualization in Fig. 4 also shows that there is still significant room for improvement. For example, it may be possible to force compactness of the distributions of voice and face embeddings through modified loss functions such as the center loss (Wen et al., 2016) or angular softmax loss (Liu et al., 2016; 2017a; b) , or through an appropriately designed loss function that is specific to this task.

B.1.1 PERFECT GENDER IDENTIFICATION:

Consider the situation where we are able to identify the gender of the subject of the data (face or voice) perfectly.

There are two possibilities: (a) both probe instances are the same gender, and (b) they are different genders. Each of the two possibilities occurs with a probability of 0.5

We employ the following simple strategy: If the two gallery instances are different genders, then we select the instance whose gender matches the probe. In this case, clearly, the probability of error is 0. If the two instances are the same gender, we select one of them randomly with a probability of 0.5. The probability of error here is 0.5.

Thus, the overall probability of error is P rob(error) = 0.5 × 0 + 0.5 × 0.5 = 0.25.

B.1.2 IMPERFECT GENDER IDENTIFICATION:

Now let us consider the situation where gender identification itself is imperfect, and we have error rates e f and e v in identifying the gender of faces and voices, respectively. Assume the error rates are known. We will assume below that gallery entries are faces, and probe entries are voices. (The equations are trivially flipped to handle the converse case).

Since we are aware that we sometimes make mistakes in identifying gender, we modify our strategy as follows: when the two gallery items are found to have different genders, we select the entry with the same gender as the probe P of the time (so that if the gender classification was correct, we would have a match error rate of (1-P )). When both gallery items are found to be the same gender, we choose randomly.

The actual error can now be computed as follows. The gallery items are both of the same gender in 0.5 of the trials, and of mismatched gender in the remaining 0.5 of the trials.

When both gallery items have the same gender, regardless of the strategy chosen, the probability of error is 0.5 (by symmetry).

When both gallery items are of mismatched gender, we have 8 combinations of correctness of gender-classification. Table 9 lists all eight, along with the probability of matching error (in the final column). Taking type 1 as an example, we have probability (1 -e v )(1 -e f ) 2 that the gender type probe gallery1 gallery2 P rob(error, type) 1

(1 -ev)(1 -e f ) 2 of both probe and galleries are correctly classified. In this case, our strategy gives us an error of (1 -P ). For type 2, the gender of probe and one of the gallery items is correctly classified, while the other gallery item is misclassified, we have an error of 0.5. If we go through all the cases, the total error P rob(error) can be computed as P rob(error) = 0.25 + 0.5 1 -α of these as gender mismatched, and the error on these instances is (1 -Q). So the overall false rejection rate is given by

F R = α(1 -P ) + (1 -α)(1 -Q) = 1 -αP -(1 -α)Q

Among the negative pairs, half are gender matched, and half are gender mismatched. Using the same logic as above, the error on the gender-matched negative pairs is αP + (1 -α)Q. Among the gender mismatched pairs the error is αQ + (1 -α)P . The overall false acceptance rate is given by F A = 0.5(αP + (1 -α)Q) + 0.5(αQ + (1 -α)P ) = 0.5(P + Q).

Equating F A and F R as the condition for EER, we obtain

1 -αP -(1 -α)Q = 0.5(P + Q) =⇒ (3 -2α)Q + (1 + 2α)P = 2.

Since at EER, the EER equals F A , and we would like to minimize it, we obtain the following solution to determine the optimal P and Q: For α > 0.5 it is easy to see that the solution to this is obtained at

Q = 0 P = 2 1 + 2α .

For α < 0.5 the optimal solution is at

P = 0 Q = 2 3 -2α .

That is, when the probe and gallery classifiers are likely to make the same error more than half the time, the optimal solution is to always reject pairs detected as having mismatched genders, and to accept matched-gender pairs 2 1+2α of the time. The optimal EER is 1 1+2α . When they are more likely to make different errors, the optimal solution is to always reject pairs detected as having matched genders, and to accept mismatched-gender pairs 2 3-2α of the time. The optimal EER now is 1 3-2α . Note that if an operating point other than EER were chosen to quantify performance (e.g.F A = βF R for β = 1, or for some fixed F A or F R ), the above analysis can be modified to accommodate it, provided a feasible solution exists.

:

To be clear, these are covariates, factors that vary jointly with voice and face, possibly due to some other common causative factors such as genetics, environment, etc. They are usually not claimed to be causative factors themselves.

Figures and Tables:

Figure 1: Overview of the proposed DIMNet and its comparison to the existing approaches. (a) Seeing faces and hearing voices from Nagrani et al. (2018b). (b) Learnable PINs from Nagrani et al. (2018a). (c) Learning face-voice association from Kim et al. (2018). (d) Our proposed DIMNets. DIMNets present a joint voice-face embedding framework via multi-task classification and require no pair construction (i.e., both voices and faces can be input sequentially without forming pairs).

shows the network architecture we use to train our embeddings. It comprises three components. The first, labelled Voice Network in the figure, represents F v (v; θ v ) and is a neural network that extracts ddimensional embeddings of the voice recordings. The second, labelled Face Network in the figure, represents F f (f ; θ f ) and is a network that extracts d-dimensional embeddings of face recordings. The third component, labelled Classification Networks in the figure, is a bank of one or more classification networks, one per covariate considered. Each of the classification networks operates on the d-dimensional features output by the embedding networks to classify one covariate, e.g.gender.

Datasets. Our experiments were conducted on the Voxceleb(Nagrani et al., 2017) and VGGFace (Parkhi et al., 2015) datasets, which are specified in appendix A.1. We use the intersection of the two datasets, i.e.subjects who figure in both corpora, for our final corpus, which thus includes 1,225 IDs with 667 males and 558 females from 36 nationalities. The data are split into train/validation/test sets, following the settings inNagrani et al. (2018b). Details can be found in Appendix A.1. We use ID, gender and nationality as our covariates, all of which are provided by the datasets. Separated data preprocessing pipelines are employed to audio segments and face images (see Appendix A.2).

Figure 3: Performance of 1:N matching

Figure 4: Visualization of voice and face embeddings using multi-dimensional scaling Wickelmaier (2003) . The left panel shows subjects from the training set, while the right panel is from the test set.

rob(error, type) = 0.25 + 0.5(2e f e v -e v -e f + 1 +P (2e f + 2e v -4e f e v -1))

s.t.1 ≥ P, Q ≥ 0, (3 -2α)Q + (1 + 2α)P = 2.



Table content: <table/>



Table content: <table><tr><td>method</td><td colspan="4">gender classification nationality classification voice face voice face</td></tr><tr><td>DIMNet-I</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>DIMNet-G</td><td>97.48</td><td>99.22</td><td>-</td><td>-</td></tr><tr><td>DIMNet-N</td><td>-</td><td>-</td><td>74.86</td><td>60.13</td></tr><tr><td>DIMNet-IG</td><td>97.70</td><td>99.42</td><td>-</td><td>-</td></tr><tr><td>DIMNet-IN</td><td>-</td><td>-</td><td>74.17</td><td>60.27</td></tr><tr><td>DIMNet-GN</td><td>97.59</td><td>99.06</td><td>74.62</td><td>60.50</td></tr><tr><td colspan="2">DIMNet-IGN 97.69</td><td>99.15</td><td>74.37</td><td>59.88</td></tr></table>

Performance comparison of 1:2 matching for models trained using different sets of covariates.

Table content: <table><tr><td>method</td><td>U</td><td colspan="2">voice → face (ACC %) G N</td><td>G, N</td><td>U</td><td colspan="2">face → voice (ACC %) G N</td><td>G, N</td></tr><tr><td>SVHF-Net</td><td>81.00</td><td>63.90</td><td>-</td><td>-</td><td>79.50</td><td>63.40</td><td>-</td><td>-</td></tr><tr><td>DIMNet-I</td><td colspan="8">83.45± 0.42 70.91±0.56 81.97±0.51 69.89±0.78 83.52±0.45 71.78±0.55 82.41±0.48 70.90±0.81</td></tr><tr><td>DIMNet-G</td><td colspan="8">72.90±0.55 50.32±0.70 71.92±0.51 50.21±0.65 72.47±0.54 50.48±0.71 72.15±0.54 50.61±0.68</td></tr><tr><td>DIMNet-N</td><td colspan="8">57.53±0.45 55.33±0.67 53.04±0.43 51.96±0.59 56.20±0.43 54.34±0.61 53.90±0.44 51.97±0.57</td></tr><tr><td>DIMNet-IG</td><td colspan="8">84.12±0.44 71.32±0.60 82.65±0.57 70.39±0.80 84.03±0.39 71.65±0.60 82.96±0.49 70.78±0.47</td></tr><tr><td>DIMNet-IN</td><td colspan="8">82.95±0.40 70.04±0.67 81.04±0.55 68.59±0.76 82.86±0.35 70.91±0.59 81.91±0.52 70.22±0.77</td></tr><tr><td>DIMNet-GN</td><td colspan="8">75.92±0.42 56.66±0.55 72.94±0.48 53.48±0.73 73.78±0.69 54.90±0.54 72.63±0.48 53.45±0.85</td></tr><tr><td colspan="9">DIMNet-IGN 83.73±0.53 70.76±0.34 81.75±0.48 69.17±0.71 83.63±0.66 71.42±0.49 82.50±0.43 70.46±0.62</td></tr></table>

Verification results.

Table content: <table><tr><td>3.2 CROSS-MODAL VERIFICATION</td><td/><td/><td/></tr><tr><td>method</td><td>U</td><td>verification (EER %) G N</td><td>G, N</td></tr><tr><td>DIMNet-I</td><td colspan="3">24.95±0.20 34.95±0.45 25.92±0.68 35.74±0.87</td></tr><tr><td>DIMNet-G</td><td colspan="3">34.86±0.11 49.69±0.24 35.13±0.36 49.67±0.51</td></tr><tr><td>DIMNet-N</td><td colspan="3">45.89±0.39 46.97±0.55 47.89±0.82 48.87±1.14</td></tr><tr><td>DIMNet-IG</td><td colspan="3">24.56±0.23 34.84±0.41 25.54±0.65 35.73±0.79</td></tr><tr><td>DIMNet-IN</td><td colspan="3">25.54±0.18 36.22±0.40 27.25±0.72 37.39±0.79</td></tr><tr><td>DIMNet-GN</td><td colspan="3">33.28±0.52 46.65±0.16 34.77±0.26 48.08±0.52</td></tr><tr><td colspan="4">DIMNet-IGN 25.00±0.19 35.76±0.36 26.80±0.69 37.30±0.74</td></tr></table>

GN 1.89±0.11 95.89±0.44 45.20±0.63 1.64±0.11 93.95±0.43 48.39±0.54 DIMNet-IGN 4.07±0.09 92.30±0.57 44.10±0.62 4.05±0.09 91.31±0.61 45.82±0.59 Retrieval performance (mAP).

Table content: <table><tr><td>method</td><td>ID</td><td colspan="2">voice → face (mAP %) gender nationality</td><td>ID</td><td colspan="2">face → voice (mAP %) gender nationality</td></tr><tr><td>Random</td><td>0.58</td><td>52.61</td><td>40.70</td><td>0.55</td><td>52.60</td><td>40.69</td></tr><tr><td>DIMNet-I</td><td colspan="6">4.25±0.11 89.57±0.35 43.26±0.16 4.17±0.10 88.50±0.37 43.68±0.20</td></tr><tr><td>DIMNet-G</td><td colspan="6">1.07±0.07 97.84±0.58 41.56±0.23 1.15±0.12 97.15±0.62 41.97±0.20</td></tr><tr><td>DIMNet-N</td><td colspan="6">1.24±0.13 56.99±0.32 45.69±0.65 1.03±0.10 56.90±0.34 49.30±0.57</td></tr><tr><td>DIMNet-IG</td><td colspan="6">4.42±0.12 93.10±0.45 43.22±0.14 4.23±0.09 92.16±0.42 43.86±0.17</td></tr><tr><td>DIMNet-IN</td><td colspan="6">3.94±0.11 89.72±0.39 43.95±0.69 3.99±0.14 88.39±0.39 45.93±0.66</td></tr><tr><td>DIMNet-</td><td/><td/><td/><td/><td/><td/></tr></table>

AUCs (%) of DIMNets under different testing groups.

Table content: <table><tr><td>N, A</td></tr></table>

the possible error types with probabilities.

Table content: <table><tr><td>5</td></tr></table>

Peer review (iclr2019):
 # Summary The article proposes a deep learning-based approach aimed at matching face images to voice recordings belonging to the same person. To this end, the authors use independently parametrized neural networks to map face images and audio recordings -- represented as spectrograms -- to embeddings of fixed and equal dimensionality. Key to the proposed approach, unlike related prior work, these modules are not directly trained on some particular form of the cross-modal matching task. Instead, the resulting embeddings are fed to a modality-agnostic, multiclass logistic regression classifier that aims to predict simple covariates such as gender, nationality or identity. The whole system is trained jointly to maximise the performance of these classifiers. Given that (face image, voice recording) pairs belonging to the same person must share equal for these covariates, the neural networks embedding face images and audio recordings are thus indirectly encouraged to map face images and voice recordings belonging to the same person to similar embeddings. The article concludes with an exhaustive set of experiments using the VGGFace and VoxCeleb datasets that demonstrates improvements over prior work on the same set of tasks. # Originality and significance The article follows-up on recent work [1, 2], building on their original application, experimental setup and model architecture. The key innovation of the article, compared to the aforementioned papers, lies on the idea of learning face/voice embeddings to maximise their ability to predict covariates, rather than by explicitly trying to optimise an objective related to cross-modal matching. While the fact that these covariates are strongly associated to face images and audio recordings had already been discussed in [1, 2], the idea of actually using them to drive the learning process is novel in this particular task. While the article does not present substantial, general-purpose methodological innovations in machine learning, I believe it constitutes a solid application of existing techniques. Empirically, the proposed covariate-driven architecture is demonstrated to lead to better performance in the (VGGFace, VoxCeleb) dataset in a comprehensive set of experiments. As a result, I believe the article might be of interest to practitioners interested in solving related cross-modal matching tasks. # Clarity The descriptions of the approach, related work and the different experiments carried out are written clearly and precisely. Overall, the paper is rather easy to read and is presented using a logical, easy-to-follow structure. In my opinion, perhaps the only exception to that claim lies in Section 3.4. If possible, I believe the Seen-Heard and Unseen-Unheard scenarios should be introduced in order to make the article self-contained. # Quality The experimental section is rather exhaustive. Despite essentially consisting of a single dataset, it builds on [1, 2] and presents a solid study that rigorously accounts for many factors, such as potential confounding due to gender and/or nationality driving prediction performance in the test set. Multiple variations of the cross-modal matching task are studied. While, in absolute terms, no approach seems to have satisfactory performance yet, the experimental results seem to indicate that the proposed approach outperforms prior work. Given that the authors claimed to have run 5 repetitions of the experiment, I believe reporting some form of uncertainty estimates around the reported performance values would strengthen the results. However, I believe that the success of the experimental results, more precisely, of the variants trained to predict the "covariate" identity, call into question the very premise of the article. Unlike gender or nationality, I believe that identity is not a "covariate" per se. In fact, as argued in Section 3.1, the prediction task for this covariate is not well-defined, as the set of identities in the training, validation and test sets are disjoint. In my opinion, this calls into question the hypothesis that what drives the improved performance is the fact that these models are trained to predict the covariates. Rather, I wonder if the advantages are instead a "fortunate" byproduct of the more efficient usage of the data during the training process, thanks to not requiring (face image, audio recording) pairs as input. # Typos Section 2.4 1) "... image.mGiven ..." 2) Cosine similarity written using absolute value |f| rather than L2-norm ||f||_{2} 3) "Here we are give a probe input ..." # References [1] Nagrani, Arsha, Samuel Albanie, and Andrew Zisserman. "Learnable PINs: Cross-Modal Embeddings for Person Identity." arXiv preprint arXiv:1805.00833 (2018). [2] Nagrani, Arsha, Samuel Albanie, and Andrew Zisserman. "Seeing voices and hearing faces: Cross-modal biometric matching." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2018.