A Benchmark for Voice-Face Cross-Modal Matching and  Retrieval
Authors: Chuyuan Xiong
Deyuan Zhang
Tao Liu
Xiaoyong Du
Jiankun Tian
Songyan Xue
Venue:  (2021)
Keywords: Cross-Modal Learning
Voice-Face Matching
Voice-Face Retrieval

Abstract: Cross-modal associations between a person's voice and face can be learned algorithmically, and this is a useful functionality in many audio and visual applications. The problem can be defined as two tasks: voice-face matching and retrieval. Recently, this topic has attracted much research attention, but it is still in its early stages of development, and evaluation protocols and test schemes need to be more standardized. Performance metrics for different subtasks are also scarce, and a benchmark for this problem needs to be established. In this paper, a baseline evaluation framework is proposed for voice-face matching and retrieval tasks. Test confidence is analyzed, and a confidence interval for estimated accuracy is proposed. Various state-of-the-art performances with high test confidence are achieved on a series of subtasks using the baseline method  (called TriNet) included in this framework. The source code will be published along with the paper. The results of this study can provide a basis for future research on voice-face cross-modal learning.

Under review as a conference paper at ICLR 2021
A BENCHMARK FOR VOICE-FACE CROSS-MODAL MATCHING AND RETRIEVAL
Anonymous authors Paper under double-blind review
ABSTRACT
Cross-modal associations between a person's voice and face can learned algo- rithmically, and this is useful functionality in many audio and visual applications. The problem can be defined as two tasks: voice-face matching and retrieval. Re- cently, this topic has attracted much research attention, but it is still in its early stages of development, and evaluation protocols and test schemes need to be more standardized. Performance metrics for different subtasks are also scarce, and a benchmark for this problem needs to be established. In this paper, baseline evalu- ation framework is proposed for voice-face matching and retrieval tasks. Test con- fidence is analyzed, and confidence interval for estimated accuracy is proposed. Various state-of of-the-art performances with high test confidence are achieved on a series of subtasks using the baseline method (called TriNet) included in this frame- work. The source code will be published along with the paper. The results of this study can provide basis for future research voice-face cross ross-modal learning.
INTRODUCTION
Studies in biology and neuroscience have shown that person's appearance associated with his or her voice (Smith et al.) (2016bla] Mavica & Barenholtz 2013). Both the facial features and voice-controlling organs of individuals are affected by hormones and genetic information (Hollien & Moore, 1960 Thornhill & Moller 1997| Kamachi et al. 2003 Wells et al. 2013). and human beings have the ability recognize this association. For example, when speaking on the phone, can guess the gender and approximate age of the person on the other end of the line. When watching a show without sound, we can also imagine the approximate voice of the protagonist by observing his her face movements. With the recent advances in deep learning, face recognition models (Wen et al. 20166] Wu et al. 2018 Liu et al. 2017) and speaker recognition models (Wang then natural wonder
al. 2018 Li al. 2017) have achieved extremely high precision. the associations between voices and faces could be discovered algorithmically by machines. The research on this problem could benefit many applications such the synchronization of video faces with talking voices and the generation of faces according to voice.
In recent years, much research attention (Wen al., 2018 Horiguchi et al. 2018 Nagrani al., 2018a Kim et al., 20188 Nagrani et al., 2018b) has been paid to voice-face cross-modal learning tasks, which has shown the feasibility recognizing voice-face associations. This problem gen- erally formulated as a voice-face matching task and voice-face retrieval task. The research on this problem is still at an early stage, and benchmark for this problem still needs be established. In this paper, we address this issue with the following contributions: 1) Existing methods are all eval- uated on a single dataset about 200 identities with limited tasks. The estimated accuracy always has great deviation due to the high sampling risk existed in cross-modal learning problem. Test confidence interval is proposed for qualifying the statistical significance of experimental results. 2) A solid baseline framework for voice-face matching and retrieval is also proposed. State-of the-art performances on various voice-face matching and retrieval tasks are achieved on large-scale datasets with high test confidence.

---

Under review as a conference paper at ICLR 2021
Table 1:
Statistics of voice-face cross-modal datasets. fox-VGG-n represents the combined dataset
of the VoxCeleb (Nagrani et al., 2017; Chung et al. 2018) and VGGFace (Cao et al. 2018 Parkhi al. 2015) and n denotes the version. The number of images refers to the number of all images remaining after MTCNN (Zhang et al. 2016) face detection.
Datuset
#ldentitics #Utterances #lmages
Vox- WGG-2
5,994
1,092,009
1,905,016
VoX-VGG-T
1,251
153,516 573,283
2 RELATED WORKS
The existing methods for voice-face cross-modal learning can be classified as classification -based methods and pair-wise loss based methods, as shown in Figure II CNN-based networks are normally used to embed the voices and faces into feature vectors. SVHF (Nagrani al. 2018b) is prior study on voice-face cross-modal learning that investigated the performance of a CNN-based deep network on this problem. The human baseline for the voice-face matching task is also presented in this paper. DIMNet (Wen et al. (2018) learns a common representation for faces and voices by leveraging their relationships to some covariates such as gender and nationality. For pair-wise loss based methods, a pair or triplct of vectors is embedded by voice and face network, and contrastive loss (Hadsell al. 2006) or triplet loss (Schroff et al.) 2015) is used to supervise the learning of the embeddings. Horiguchi et al.'s 's's method (Horiguchi et al. 2018)) Pins (Nagrani et al. 2018a). Kim et al.'s methods (Kim al. 2018) are all these kind of methods. The aim of pair-w wise loss based methods to make the embeddings of positive pairs closer and the embeddings of negative pairs farther apart. In contrast, the aim of classification-bet
methods is to separate the embeddings of different classes. Of these two approaches, pair-wo wise loss based methods are better at distinguishing hard examples because of the characteristics of this approach.
There is still no related work which presents benchmark for voice-face cross moosl learning tasks, which is addressed in detail as follows:
1) As for evaluation metrics, the reliability of experiments has not been addressed by all previous research. Test confidence is proposed in this paper. With the guidance of test confidence, reliable evaluations can be conducted.
2) As for tasks, joint matching and joint retrieval tasks established in this paper are not noticed by previous research. Though these tasks direct extensions of traditional tasks, these very simple extensions can improve the performance of voice-face cross-modal learning dramatically.
3) As for models, the most similar work to TriNet of this paper is Kim et al.'s method (Kim al.) 2018). Both models use the triplet loss function. The main difference is that TriNet uses L2 normal- ization and voice-anchored embedding learning to constrain the feature space, because it is difficult to obtain satisfactory results by training directly in huge Euclidean space. Though L2 normaliza- tion is a normal technique, hasn' 't been introduced the current problem.
4) As for datasets, currently available voice-face datasets are the data generated by the common speakers of VGGFace (Cao et al. 2018 Parkhi al., 2015) face recognition dataset and VoxCeleb (Nagrani et al.) 2017] [Chung et al. 2018) speaker recognition dataset. As shown in Table IT the voice-face datasets have two versions, Vox- VGG-1 and Vox- VGG-2, which include 1,251 and 5,994 identities, respectively. To the best of our knowledge, only Vox-VGG- is used in previous research. Both Vox-VGG-1 and Vox-VGG-2 are used to evaluate the proposed baseline method, TriNet.
TASKS AND EVALUATION
3.1
TASKS
1:2 Matching and 1:n Matching. Given an audio and two face candidates (only one of which is from the speaker of the audio), the goal is to find the face that belongs to the speaker. The more difficult Im matching task is an extension of the 1:2 matching task that increases the number of candidate faces from 2 to N.
2

---

Under review as a conference paper at ICLR 2021
Classfication Baved Methods
Pair-wise Rased Methods
Fal kptt
Tarlip
CSN
VGG-M
Enveetor
VGG-M
VGG-M
Coxcat
Comuzc Verken
Vertors
Par Yestor
Triple Venters
Classification
Clasafication
Cosine Similarity
Contrastive Loss
Triplet Loss
Horiguchi's
Pins
Kim's
Figure 1: Comparison of existing methods.
Retrieval. Given a "query" voice, the goal of voice-face retrieval to rank face images accord- ing to their relevance with respect to the voice query. This task is supplement to the matching task, the position-relatee information of all retrieved faces also effective for analyzing the model performance.
Joint Matching and Joint Retrieval. Instead of single audio segment or single face for one identity, multiple audio segments and faces can provide more information. Matching and retrieval can be conducted on the mean embeddings of multiple audios or images. This is the simplest way to improve the performance of current voice-face matching and retrieval methods. Widespread video resources imply that the use of multiple faces and voices is feasible.
3.2 TEST CONFIDENCE
The evaluation criteria for matching and retrieval tasks are accuracy and mAPpChristoppe et al, 2008) respectively. All previous studies (Wen et al. 2018 Nagrani al.) 2018 Kim et al.) 2018 Nagrani et al. 2018b) evaluated their methods on a single dataset of about 200 identities. As shown the experiment (Section (5.4). the 1:2 matching accuracy tested on multiple datasets with 189 identities varies significantly, from 81% to 87%. So the results of all related works that used Vox- VGG-1 for training and testing are unretiable. Testing a model on a single small dataset may lead to large deviation in the accuracy.
In 1:2 matching task, the accuracy is estimated on the sampled data, to represent the accuracy on the overall population. The estimated accuracy always has a large deviation due to the high sampling risk in the triplet sampling scenario.
Essentially, our aim is to obtain the correct matching probability of single independent triplet. When the dataset and the model are determined, a single independent sampling conforms to the Bemoulli distribution B(p). The results of n samplings fit the binomial distribution (n,,)) Interval estimation a binomial distribution can be used for quantifying the deviation of the estimated accuracy. Suppose dataset D generate up N triplets, and the number of sampled triplets used for testing is n. Among the sampled triplets, there are m correctly matched triplets. Suppose the sample rate is p. where p
and the population P. When n is sufficiently large, p can be approximated as normal distribution p N(P, P(I-P), By converting it standard normal distribution, we obtain u
of correctly matched N triplets is
NO, 1). For
PLL
significance level a, the confidence interval of p (p ug model on multiple datasets is strongly recommended when the dataset is very small. The test can be performed multiple times datasets with similar scale, and the results regarded conforming to the normal distribution. The t-test can then used to estimate the confidence interval of the accuracy.
p(1-)) p ug
p(1-2)). Testing "
3

---

Under review as a conference paper at ICLR 2021
4 TRINET BASELINE METHOD
As shown Figure the baseline method in the proposed framework consists of three steps: ex- tracting voice and face features, constraining embeddings to spherical space, and computing the triplet loss. After training, the face embedding and voice embedding form their own regions, and the distance between positive samples tends to be smaller.
Iaput triplet
Network Backbome
Embedding Constraints
Triplet Loss on Embedding
M1=-
I
Metric Space
]
v emb
ooo
v
"emb
b1a=-
boo... ooo
ooo
I
f 100
f()
000-
(0)
1 -
embi
femb
f()
Coustralnts
Tmbb
Figure 2: Overview TriNet. 1) A feature extraction module extracts voice and face features. 2) Embeddings constrained to lie spherical space using I.2 normalization. In this step, the audio network is frozen. 3) The triplet loss function is used to learn the embeddings.
4.1 TRIPLET MINING
The input triplets for the embedding network need to be mined from the datasets, the number of which extremely large. previous research (Nagrani et al., 20T8b Kim al.] [008]] Wen al.) 2018 Horiguchi et al. 2018). discrete triplets are randomly mined to create single input each time, which will lead to training and test inefficiency. Identity based sampling named as online mining is adopted in this paper, which greatly improve the training and testing efficiency. In the identity based sampling, batch of identities is randomly selected first, and then certain number of face images and audios for each identity of the batch are sampled. Triplets each batch of identities. Triplet Loss is susceptible to noise which means the direction of network convergence is easy to be changed by few noise samples. Identity based training effectively handle the disadvantage of Triplet Loss.
generated based
4.2 EMBEDDING CONSTRAINTS AND THE LOSS FUNCTION
For specific triplet (()), ((), (())) (0), and f() are from the same identity, and ((() and f(6) are from different identities. The feature extraction functions for voice and face defined as Feature, (v) and Feature, (f), respectively, and fully connected layer is added to form the embedding vectors as emb, (v) = s x W, Feature, + B and embr(s) = s x [W, x Feature (f) Brl LResNet50 (He al. 2015) and Thin ResNet34 (Xie al., 2019) with NetVL.D (Arandjelovic et al. 2017) are networks that perform well on face recognition and speaker recognition tasks, respectively. These two networks are used in this paper for face feature extraction and voice feature extraction.
As illustrated Figure 3a| embedded vectors from the same person will appear in Euclidean space after long period of training. Because there are billions of input triples, it is difficult to obtain satisfactory results by directly training in huge Euclidean space. To deal with this problem, two strategies adopted in the method proposed in this paper. First, L2 normalization is added to constrain the embedding vectors to spherical space (Figure 3b). Second, voice-anchored em- bedding learning is adopted. By freezing the pre-trained voice embedding network, feature vectors

---

Under review as a conference paper at ICLR 2021
from voice serve anchors, and the goal of the model is make positive instances approach each other while keeping the negative instances away (Figure Bc). Examples tend to be distinguished much better and faster in the voice-anchores embedding learning process when used with the L2 constrained space. Triplet loss is adopted this paper for embedding learning. Suppose d(x, y) indicates Euclidean distance; the loss function is defined as
Loss
Σ
max max (d(emb,(v emb
emb, (())((()))) + m,
where m is margin to control the distance between positive and negative pairs.
(3) Original
L2-Norm
(c) Voice Anchored L2-Norm
Figure 3: Visualization of three metrie learning methods.
5 EXPERIMENT
5.1 BASELINE MODEL SETUP
Training Details. TriNet was trained on Vox- VGG-2 with 5,994 identities. Face detection based on MTCNN (Zhang et al. 2016) was conducted and all face images were then rescaled to 112 x 112 3 to form the input for the face embedding networks. Audio preprocessing consisted 512-point fast Fourier transform, short-time Fourier transform (Benesty et al., 2211) for each frame, and normalization. The audio segments used for training were uniformly trimmed to 2.5 s for training efficiency, and the test audio segments 257 x (100 x k) x 1. The voice embedding network and face embedding network were pre-trained by VoxCeleb2 and VGGFace2, respectively. Margin, m, for the triplet loss was to and scale, 8, for the 1.2 normalization was set to 128. The Adam optimizer was adopted in these experiments, and the total number of learning steps was 70k. The learning rates of the final fully connected layer for step < 20k. 20k < step < 40k, 40k < step < 60k, and step > 60k were 10-3, 1004, 10-5, and 10 respectively. The learning rate of the face embedding network was fixed 10 10-6
not clipped. The input shape for k-s audio clip is
Testing Details. I) For the 1:2 matching task, a total of 10,000 steps were tested on the baseline, which implies that total of 30.72 million triples were tested. Note that the gender of the test triples in the 1:2 matching task was balanced. 2) For the I:n matching task, the number of tuples to be sampled will much higher than the number of triples 1:2 matching; therefore, we performed this test directly on the 10k tuples. This will lower the confidence level, but the results are still useful for comparisons. 3) For the retrieval task, a face database of 500 pictures was constructed from 100 randomly selected identities, and 40 audio queries were constructed for each identity.
Table 2: Comparison of the proposed method with other models the 1:2 matching task and retrieval task. "ACC" is accuracy.
Method
Ideatities Triplets
ACC(%)
TriNet
1,251
SVHP Nagrani
20188
81.00 1.01%
1:2 Matching
DIMNATIG
Nen
2013
189
678M
S4 12%
Kim's
20
250
7% 20%5
Horiguchi's
Horigachf
01 2013
216
38B
77. 80%
Method
Test Edentitics Chance (mAP%)
mAP
Retrieval
DIMNa-IG
TrNNl
2018
1251
2.15
IT:48
Horigachi's
Horgeaii
5

---

Under review as a conference paper at ICLR 2021
SVHF DIMNett-GG TriNet
75
accurscy of 1:2 metching
0.05
70
0.84
03-
0.83
60-
0.82
i0D-idenbity tatasets,Pupo a
0.81
189 piso
0.50%
500 eob-ideenityyy datasets pizoo -4.23 0.64 6.64%
0.80
1000-identity datasets,Puees 84.48 2 0.25%
15
20
random
(a) Comparisons on the Im matching task.
(b) Changes in accuracy experiments on four differ- ent sizes of datasets.
Figure 4: (a) Comparisons the 1:n matching task. (b) Accuracy curves.
Voice
Top 10 retrieved faces
Figure 5: Qualitative analysis of retrieval results produced by TriNet.
5.2 COMPARISONS ON MATCHING AND RETRIEVAL TASKS
Comparisons of TriNet and related works on the 1:2 matching task and retrieval task shown in Table 2 TriNet achieves state-of-the-an art performance on these two main tasks. As shown in Figure increases. This task is still significantly difficult.. Some TriNet retrieval results that fit p@1 = 1 are illustrated in Figure The top ranked faces in each sequence are very similar to the target face.
4al on the 1:n task, the performances of all the methods decrease rapidly
5.3 JOINT TASKS PERFORMANCE
The results of 1:2 joint matching using mean voice and mean face are shown in Table S Two variables, m and m. are introduced to represent the number of faces and audios used to compute the mean embedding. Various values of my and m, are tested for 1:2 matching. For retrieval, m, was set to 20 and m was set to 5, This simple strategy of using multiple faces and voices can further improve the matching accuracy and retrieval mAP. Specifically, when m, = m, = 10, an accuracy of 89.66 + 0.80% can be obtained for TriNet on the 1:2 matching task, which is 5% higher than that of single voice and single face matching. This improvement reveals a broad prospect for future research on using video data for cross-modal learning.

---

Under review as a conference paper at ICLR 2021
Table 3: Performance of 1:2 joint matching and joint retrieval using mean face and mean voice embeddings. Here, m and m, denote the number of face images and audio segments, respectively, used calculate the averages. The values of my and my are by default.
Joint Matching Using Mean Voices and Mean Faces (TriNet)
y (m,
1) ACC(S) (ml 1) ACC(%) my
ACC(%)
5
85.55
5
85.13
1+1
84.48
10
84. 56
10
86.01
5+5
86.42
15
15
85.49
10+10
89.66
20
86.53
20
86,16
20+20
89.54
85.63
Joint Retrieval Using Mean Voice and
Mean Face
mean voice mAP(%) %)
mean face mAP(%)
TriNet Chance
12.53
21.65
2.15
5.25
Table 4: Effect of training set scale. "TriNet" used the default configuration. was trained 1,000 and tested 189 identities.
"TriNet-train1000
Method
#Train Identities #lest Identities #lest Triplets
ACC(%)
mAP(%)
TriNet
5994
1251
30M
84.48
0.01%
11.48%
TriNet-train1000
1000
189
3M
83.55 = 0.55%
9.61%
5.4 TEST CONFIDENCE OF DATASETS WITH DIFFERENT SCALES
Figure 4b shows the fluctuations in the estimated accuracies of TriNet on the 1:2 matching task when 30 repeated random tests were conducted. The numbers of sampled identities for each curve are 100, 189, 500, and 1,000. For determined dataset scale (such as 100 identities), instead of testing the model on single dataset with 100 identities, 30 randomly sampled sets of data with 100 identities were used for testing. As shown in the figure, when small-scale dataset is used, the accuracy of different runs fluctuates substantially. For large datasets, fluctuations test accuracies also exist; however, in general, the test results more generalized; therefore, large datasets strongly recommended for evaluation.
5.5 ABLATION STUDY
There are various options in the baseline model. To determine the option that has a greater impact on performance, we conducted more detailed ablation study.
5.5.1 TRAINING SCALE
The size of the training dataset identities used by the baseline is five times that of most related studies. To demonstrate the effects of larger training dataset the results, TriNet was also trained on a dataset of 1,000 identities and tested on a dataset with 189 identities. As shown in Table 4 the improvement of training on large scale dataset near 1%. The upper limit of the results similar to those of DIMNet-IG. Adding more identities can increase the performance by 0.5%. It is difficult further improve the performance by increasing the size of the dataset. In contrast, integrating multiple faces and voices is an effective way to further improve the performance.
5.5.2 PREPROCESSING
We need study whether face detection should be used and how big the detection box should be. As the results in reveal, without the use of face detection, large amount of noise is introduced along with a few useful features, and the performance of the baseline model on all matching and retrieval tasks decreases. When the size of the default detection box is increased by 1.1 times, better performance obtained.
5.5.3 NETWORK STRUCTURE
As shown in Table 5 deeper CNN structures such as SE-ResNet50 et al. 2018) and the structure used in DIMNet outperform traditional shallow structures such as VGG-M. An SE- Nes5500 with
7

---

Under review as a conference paper at ICLR 2021
denote accuracy for 1:2 and 1:3 matching Table 5: Ablation study of TriNet. 1:2(%) and 1:3(%)
tasks, and Male(%) and Female(% Jenote accuracy all male triplets and female triplets, respec- tively.
Settings
1:2(%)
1:3(%)
Ma)e(()
Female(
mAP(%)
TriNet (Default)
84.48
73.50
69.85
71.04
11.48
Defect Scale
Andio Crop
Without Scule
Whole
83,04
71.13
66.99
69.69
8.22
1.1 Scale
Whole
84.27
73.28
70.03
71.65
10.42
1.0 Scale
Crop 2.5
83.11
70.49
61.41
69.45
Face Backhone
VGG-M
81.30
68.95
64.90
64,84
8.06
Face Backhone DIMNet
83.88
72.06
62.93
68.98
10.05
SE-ResNer50
84.0
72.62
69.22
69.49
Vace Frozen
Voice Frazen
N
84,43
73.56
68.27
68.85
9.77
81.68
68.52
64.87
66.76
8.33
Metric Space
Metric Scale
Euclidean
128
82.19
69.29
69.38
64.97
8.44
1.2
1
81.71
69.09
62.62
67.49
7.58
512
84.27
72.48
69.43
71.77
10.02
MS-IB Guo 2016
VOX2
84.13
63.81
74.70
None
VOX2
73.29
58.20
54.26
58.90
5.56
None
None
70.64
54.11
52.21
51.47
3.91
squeeze-and-excitttion module does not produce better results than the original ResNet50 structure used in the baseline model.
5.5.4 EMBEDDING CONSTRAINTS
The effects of using L2 normalization and freezing the pre-trained networks are analyzed here. As presented in Table 5 using L2 normalization improves the performance of 1:2 matching accuracy by 2% and mAP by 2%. In the default configuration, the size of the metric space is 128. The model performance decreases when the scale is set to 1, which indicates that is necessary to properly increase the size of the metric space. Freezing the face embedding network reduces the performance, whereas freezing the voice embedding network improves the performance slightly. This is because human voices only related to some local features of human faces, and similar faces in traditional face recognition tasks do not necessarily have similar voices. Therefore, voice- anchored embedding learning outperforms face-anchored embedding learning. Training efficiency improved substantially by freezing the voice network.
5.5.5 PRE-TRAINING
As shown Table 5 when TriNet is pre-trained on the large dataset MS-IB (Guo et al. 2016), its performance is not improved. However, without pre-training the model's performance substan- tially reduced. (Note that in this case, the voice network was not frozen.)
CONCLUSION
In this study, a benchmark was established for voice-face matching and retrieval. The contributions of this paper are follows. A solid voice-face matching and retrieval baseline method (TriNet) was proposed, which was tested on large-scale dataset with comprehensive ablation studies. The test confidence was proposed
metric for qualifying the statistical significance of the experiments. On the 1:2 matching and retrieval tasks, TriNet achieved an accuracy of 84.48% and mAP of 11 1. Compared with the best results published so far, there is 7% improvement in mAP. Using mean face and mean voice embeddings, the matching accuracy and retrieval mAP can be further improved by approximately 5% and 10%. respectively. This improvement implies broad prospect for future research using video data for cross-modal learning.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
R Arandjelovic. P Gronat, A Torii. T Pajdla, and Sivic. Netvlad: Cnn architecture for weakly supervised place recognition. IEEE Transactions on Pattern Analysis & Machine Intelligence, PP (99):1-1, 2017.
Jacob Benesty, Jingdong Chen, and Emanuël AP Habets. Speech enhancement in the SIFT domain. Springer Science & Business Media, 2011.
Qiong Cao, Li Shen, Weidi Xie, Omkar M Parkhi, and Andrew Zisserman. Vggface2: A dataset for recognising faces across pose and age. In 2018 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2018), pp. 67-74. IEEE, 2018.
D Manning Christopher, Raghavan Prabhakar, and Schüizee Hinrich. Introduction to information retrieval. An Introduction To Information Retrieval, 151(177):5, 2008.
Joon Son Chung, Arsha Nagrani, and Andrew Zisserman. Voxceleb2: Deep speaker recognition. 2018.
Yandong Guo, Lei Zhang, Yuxiao Hu, Xiaodong He, and Jianfeng Gao. Ms-celeb- Im: A dataset and benchmark for large-scale face recognition. In European Conference on Computer Vision, pp. 87-102. Springer, 2016.
Raia Hadsell, Sumit Chopra, and Yann LeCun. Dimensionality reduction by learning an invariant mapping. In 2006 IEEE Computer Society Conference on Computer Vision and Pattern Recogni- tion (CVPR 06), volume 2, pp. 1735-1742. IEEE, 2006.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. 2015.
Harry Hollien and G. Paul Moore. Mcasurements of the vocal folds during changes in pitch. Journal of Speech and Hearing Research, 3(2):157-165, 1960.
Shota Horiguchi, Naoyuki Kanda, and Kenji Nagamatsu. Face-voice matching using cross-modal embeddings. In 2018 ACM Multimedia Conference on Multimedia Conference, pp. 1011-1019. ACM, 2018
Jie Hu, Li Shen, and Gang Sun. Squeeze-and-excitttte networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 7132-7141, 2018.
Miyuki Kamachi, Harold Hill, Karen Lander, and Bric Vatikiotis-Bateson. Putting the face to the voice': Matching identity across modality. Current Biology, 17019)110091174, 2003.
Changil Kim, Hijung Valentina Shin, Tae-Hyun Oh, Alexandre Kaspar, Mohamed Elgharib, and Wojciech Matusik. On learning associations of faces and voices. In Asian Conference on Com- puter Vision, pp. 276-292. Springer, 2018.
Chao Li, Xiaokong Ma, Bing Jiang, Xiangang Li, Xuewei Zhang, Xiao Liu, Ying Cao, Ajay Kannan, and Zhenyao Zhu. Deep speaker: an end-to-end neural speaker embedding system. arXiv preprint arXiv: 2017.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 212-220, 2017.
Lauren W Mavica and Elan Barenholtz. Matching voice and face identity from static images. Jour- nal of Experimental Psychology: Human Perception and Performance, 39(2):307, 2013.
Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Voxceleb: a large-scale speaker identifi- cation dataset. 2017.
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Learnable pins: moosl embeddings for person identity. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 71-88, 2018a.

---

Under review as a conference paper at ICLR 2021
Arsha Nagrani, Samuel Albanie, and Andrew Zisserman. Seeing voices and hearing faces: Cross- modal biometric matching. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 8427-8436, 2018b.
Omkar M Parkhi, Andrea Vedaldi, Andrew Zisserman, al. Deep face recognition. In bmvc, volume 1, pp. 6, 2015.
Florian Schroff, Dmitry Kalenichenko, and James Philbin. Facenet: A unified embedding for face the IEEE conference on computer vision and pattern
recognition and clustering. In Proceedings the recognition, pp. 815-823, 2015.
Harriet MJ Smith, Andrew K Dunn, Thom Baguley, and Paula C Staccy. Concordant in faces and voices: Testing the backup signal hypothesis. Evolutionary Psychology, 14(1): 1474704916630317, 2016a.
Harriet MJ Smith, Andrew K Dunn, Thom Baguley, and Paula C Stacey. Matching novel face and voice identity using static and dynamic facial images. Attention, Perception, & Psychophysics, 78 (3):868 179868-879.08 2016b.
Randy Thornhill and Anders Pape Møller. Developmental stability, disease and medicine. Biological Reviews, 72(4):497-548, 1997.
Quan Wang. Carlton Downey, Li Wan, Philip Andrew Mansfield, and Ignacio Lopz Moreno. Speaker diarization with Istm. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5239-5243. IEEE, 2018.
Timothy Wells, Thom Baguley, Mark Sergeant, and Andrew Dunn. Perceptions of human attrac- tiveness comprising face and voice cues. Archives of sexual behavior, 42(5):805 -1180, 2013.
Yandong Wen, Kaipeng Zhang, Zhifeng Li, and Yu Qiao. A discriminative feature learning approach for deep face recognition. European conference on computer vision, pp. 499-515. Springer, 2016.
Yandong Wen, Mahmoud AI Ismail, Weiyang Liu, Bhiksha Raj, and Rita Singh. Disjoint mapping network for cross-modal matching of voices and faces. 2018.
Xiang Wu, Ran He, Zhenan Sun, and Tieniu Tan. A light enn for deep face representation with noisy labels. IEEE Transactions on Information Forensics and Security, 13(11):2884-298 2896, 2018.
Weidi Xie, Arsha Nagrani, Joon Son Chung, and Andrew Zisserman. Utterance-level aggregation for speaker recognition in the wild. In ICASSP 2019-2019 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 5791-5795. IEEE, 2019.
Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Qiao Yu. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE Signal Processing Letters, 2016.
10


Reviews:
[{"review_id": "c6T5skOdMhE", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The goal of this paper is to learn cross-modal associations between a person’s face and a voice. The authors use a standard three stream network trained with a triplet loss, and evaluate on the VoxCeleb-VGGFace datasets.   \n\nStrengths: \n- The authors train on VoxCeleb-VGGFace2, and evaluate on VoxCeleb1, which is a larger set of identities, and perform a t-test to show statistical significance. \n\n- The ablation study showing that performance saturates with more training data is interesting. \n\n\nWeaknesses: \n- There is limited novelty in the method. The authors use the triplet loss which has been widely used for this problem before (Kim et al. 2018, Cheng et al. 20- https://dl.acm.org/doi/pdf/10.1145/3394171.3413710). The authors also claim that anchoring the voice subnetwork with frozen weights is a novel contribution but don’t discuss that this teacher-student style model was already tried in the Learnable PINs paper (where the face subnetwork is frozen). L2 normalisation of embeddings has also been used widely with the triplet loss, and has also been used in Learnable PINs (and hence been applied to this problem as well). \n\n- Table 2 is not a fair comparison. The authors have compared the performance of different models on completely different test sets, with a different number of speakers! To assess the performance of TriNet, it must be compared to SOTA methods on the same test set of 189 speakers, or if the authors prefer - on all of the 1,251 speakers in VoxCeleb1. In this case the other SOTA methods must be re-evaluated on this new test set.  \n\n\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "9d1QRjRWsTv", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work focuses on the problem of cross-modal matching and retrieval for face and voice modalities. The paper suggests a new benchmark for the evaluation of both matching and retrieval tasks. It also proposes a confidence margin computation to verify the statistical significance of the results. The results reported on Vox (voice) and VGG (face) dataset using the suggested benchmark are encouraging. I have summarized my comments below which will help in improving the quality of this manuscript:  \n\n1. I believe the technical contribution of this paper is very limited for ICLR. This work builds upon the previous works in the area of cross-modal retrieval. The contribution is very incremental and most of the conclusions are very well known. The multi-stream N/W architecture is very similar to the previous works and triplet loss has been used extensively for face and voice tasks in the past. The suggested protocol just uses a different combination of training and evaluation set from Vox and VGG sets.\n\n2. In section 3.2, the authors mentioned \"So the results of all related works that used VoxVGG-1 for training and testing are unreliable.\" I feel this is a very strong statement and may not always be true. First, the Vox dataset is collected from YouTube videos and represents a wide variety of conditions and results on the careful partition (no overlapping identities and conditions) of training and test may be very helpful. Second, in cases where evaluation conditions or use cases are known and similar to the VoxVGG-1 this may not hold true. \n\n3. In speaker recognition, data augmentation plays a huge role in learning a noise-robust representation. It is not clear if the authors applied data augmentation to the Vox2 training set. These works highlight the importance of data augmentation:\nSnyder, D., Garcia-Romero, D., Sell, G., Povey, D. and Khudanpur, S., 2018, April. X-vectors: Robust DNN embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5329-5333). IEEE.\nZeinali, H., Wang, S., Silnova, A., Matějka, P. and Plchot, O., 2019. But system description to voxceleb speaker recognition challenge 2019. arXiv preprint arXiv:1910.12592.\n\n4. Did the authors use any voice activity detection system to remove silence in the audio? \n\n5. In section 5.4, the authors claim \"Therefore, voice anchored embedding learning outperforms face-anchored embedding learning.\" It would be better to clarify these results in Table 5.\n\n6. The current state-of-the-art in speaker recognition is TDNN based x-vector. I would request authors to add a reference to the following paper in the introduction:\nSnyder, D., Garcia-Romero, D., Sell, G., Povey, D. and Khudanpur, S., 2018, April. X-vectors: Robust DNN embeddings for speaker recognition. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP) (pp. 5329-5333). IEEE.", "extra_scores": {}, "extra_text": {}}, {"review_id": "0kbe8eNg2nf", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary:\nThis paper aims to propose a benchmark for voce-face matching and retrieval problem. As shown by the test confidence analysis, the model is suggested to be evaluated on a large dataset or multiple datasets to avoid the large deviation in the accuracy. A baseline method TriNet and joint matching & retrieval are proposed. Improved results are reported in the experiment section.\n\nMy biggest concern is the unclear contribution of this paper.\nTest confidence is proposed, but has not been used to measure any results reported in this paper. It seems a bit disconnected. The only conclusion from test confidence analysis is to test model on multiple dataset, which is obvious.\n\nThe TriNet uses L2 normalized triplet loss, which is also not new and can be found in many previous work, e.g. [1]. Simply applying this normalized triplet loss to cross-modal matching is not a significant contribution.\n[1] Schroff, Florian, Dmitry Kalenichenko, and James Philbin. \"Facenet: A unified embedding for face recognition and clustering.\" Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.\n\nAlso, the results in Table 2 are not fair comparison. As the paper points out, the training and testing data of TriNet are different from other methods, which means the results are not comparable. So I couldn't see any insight from the results. \n\n\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers pointed out several opportunities for improvements and concurred that the paper needs significant work before it is ready for publication.  The authors did not provide a rebuttal. We hope the review process was useful to the authors. 
Decision: Reject