CN-CELEB: a challenging Chinese speaker recognition dataset

Abstract: Recently, researchers set an ambitious goal of conducting speaker recognition in unconstrained conditions where the variations on ambient, channel and emotion could be arbitrary. However, most publicly available datasets are collected under constrained environments, i.e., with little noise and limited channel variation. These datasets tend to deliver over optimistic performance and do not meet the request of research on speaker recognition in unconstrained conditions. In this paper, we present CN-Celeb, a large-scale speaker recognition dataset collected `in the wild'. This dataset contains more than 130,000 utterances from 1,000 Chinese celebrities, and covers 11 different genres in real world. Experiments conducted with two state-of-the-art speaker recognition approaches (i-vector and x-vector) show that the performance on CN-Celeb is far inferior to the one obtained on VoxCeleb, a widely used speaker recognition dataset. This result demonstrates that in real-life conditions, the performance of existing techniques might be much worse than it was thought. Our database is free for researchers and can be downloaded from this http URL.

Speaker recognition including identification and verification, aims to recognize claimed identities of speakers. After decades of research, performance of speaker recognition systems has been vastly improved, and the technique has been deployed to a wide range of practical applications. Nevertheless, the present speaker recognition approaches are still far from reliable in unconstrained conditions where uncertainties within the speech recordings could be arbitrary. These uncertainties might be caused by multiple factors, including free text, multiple channels, environmental noises, speaking styles, and physiological status. These uncertainties make the speaker recognition task highly challenging BIBREF0, BIBREF1.
Researchers have devoted much effort to address the difficulties in unconstrained conditions. Early methods are based on probabilistic models that treat these uncertainties as an additive Gaussian noise. JFA BIBREF2, BIBREF3 and PLDA BIBREF4 are the most famous among such models. These models, however, are shallow and linear, and therefore cannot deal with the complexity of real-life applications. Recent advance in deep learning methods offers a new opportunity BIBREF5, BIBREF6, BIBREF7, BIBREF8. Resorting to the power of deep neural networks (DNNs) in representation learning, these methods can remove unwanted uncertainties by propagating speech signals through the DNN layer by layer and retain speaker-relevant features only BIBREF9. Significant improvement in robustness has been achieved by the DNN-based approach BIBREF10, which makes it more suitable for applications in unconstrained conditions.
The success of DNN-based methods, however, largely relies on a large amount of data, in particular data that involve the true complexity in unconstrained conditions. Unfortunately, most existing datasets for speaker recognition are collected in constrained conditions, where the acoustic environment, channel and speaking style do not change significantly for each speaker BIBREF11, BIBREF12, BIBREF13. These datasets tend to deliver over optimistic performance and do not meet the request of research on speaker recognition in unconstrained conditions.
To address this shortage in datasets, researchers have started to collect data `in the wild'. The most successful `wild' dataset may be VoxCeleb BIBREF14, BIBREF15, which contains millions of utterances from over thousands of speakers. The utterances were collected from open-source media using a fully automated pipeline based on computer vision techniques, in particular face detection, tracking and recognition, plus video-audio synchronization. The automated pipeline is almost costless, and thus greatly improves the efficiency of data collection.
In this paper, we re-implement the automated pipeline of VoxCeleb and collect a new large-scale speaker dataset, named CN-Celeb. Compared with VoxCeleb, CN-Celeb has three distinct features:
CN-Celeb specially focuses on Chinese celebrities, and contains more than $130,000$ utterances from $1,000$ persons.
CN-Celeb covers more genres of speech. We intentionally collected data from 11 genres, including entertainment, interview, singing, play, movie, vlog, live broadcast, speech, drama, recitation and advertisement. The speech of a particular speaker may be in more than 5 genres. As a comparison, most of the utterances in VoxCeleb were extracted from interview videos. The diversity in genres makes our database more representative for the true scenarios in unconstrained conditions, but also more challenging.
CN-Celeb is not fully automated, but involves human check. We found that more complex the genre is, more errors the automated pipeline tends to produce. Ironically, the error-pron segments could be highly valuable as they tend to be boundary samples. We therefore choose a two-stage strategy that employs the automated pipeline to perform pre-selection, and then perform human check.
The rest of the paper is organized as follows. Section SECREF2 presents a detailed description for CN-Celeb, and Section SECREF3 presents more quantitative comparisons between CN-Celeb and VoxCeleb on the speaker recognition task. Section SECREF4 concludes the entire paper.
The original purpose of the CN-Celeb dataset is to investigate the true difficulties of speaker recognition techniques in unconstrained conditions, and provide a resource for researchers to build prototype systems and evaluate the performance. Ideally, it can be used as a standalone data source, and can be also used with other datasets together, in particular VoxCeleb which is free and large. For this reason, CN-Celeb tries to be distinguished from but also complementary to VoxCeleb from the beginning of the design. This leads to three features that we have discussed in the previous section: Chinese focused, complex genres, and quality guarantee by human check.
In summary, CN-Celeb contains over $130,000$ utterances from $1,000$ Chinese celebrities. It covers 11 genres and the total amount of speech waveforms is 274 hours. Table TABREF5 gives the data distribution over the genres, and Table TABREF6 presents the data distribution over the length of utterances.
Table TABREF13 summarizes the main difference between CN-Celeb and VoxCeleb. Compared to VoxCeleb, CN-Celeb is a more complex dataset and more challenging for speaker recognition research. More details of these challenges are as follows.
Most of the utterances involve real-world noise, including ambient noise, background babbling, music, cheers and laugh.
A certain amount of utterances involve strong and overlapped background speakers, especially in the dram and movie genres.
Most of speakers have different genres of utterances, which results in significant variation in speaking styles.
The utterances of the same speaker may be recorded at different time and with different devices, leading to serious cross-time and cross-channel problems.
Most of the utterances are short, which meets the scenarios of most real applications but leads to unreliable decision.
CN-Celeb was collected following a two-stage strategy: firstly we used an automated pipeline to extract potential segments of the Person of Interest (POI), and then applied a human check to remove incorrect segments. This process is much faster than purely human-based segmentation, and reduces errors caused by a purely automated process.
Briefly, the automated pipeline we used is similar to the one used to collect VoxCeleb1 BIBREF14 and VoxCeleb2 BIBREF15, though we made some modification to increase efficiency and precision. Especially, we introduced a new face-speaker double check step that fused the information from both the image and speech signals to increase the recall rate while maintaining the precision.
The detailed steps of the collection process are summarized as follows.
STEP 1. POI list design. We manually selected $1,000$ Chinese celebrities as our target speakers. These speakers were mostly from the entertainment sector, such as singers, drama actors/actrees, news reporters, interviewers. Region diversity was also taken into account so that variation in accent was covered.
STEP 2. Pictures and videos download. Pictures and videos of the $1,000$ POIs were downloaded from the data source (https://www.bilibili.com/) by searching for the names of the persons. In order to specify that we were searching for POI names, the word `human' was added in the search queries. The downloaded videos were manually examined and were categorized into the 11 genres.
STEP 3. Face detection and tracking. For each POI, we first obtained the portrait of the person. This was achieved by detecting and clipping the face images from all pictures of that person. The RetinaFace algorithm was used to perform the detection and clipping BIBREF16. Afterwards, video segments that contain the target person were extracted. This was achieved by three steps: (1) For each frame, detect all the faces appearing in the frame using RetinaFace; (2) Determine if the target person appears by comparing the POI portrait and the faces detected in the frame. We used the ArcFace face recognition system BIBREF17 to perform the comparison; (3) Apply the MOSSE face tracking system BIBREF18 to produce face streams.
STEP 4. Active speaker verification. As in BIBREF14, an active speaker verification system was employed to verify if the speech was really spoken by the target person. This is necessary as it is possible that the target person appears in the video but the speech is from other persons. We used the SyncNet model BIBREF19 as in BIBREF14 to perform the task. This model was trained to detect if a stream of mouth movement and a stream of speech are synchronized. In our implementation, the stream of mouth movement was derived from the face stream produced by the MOSSE system.
STEP 5. Double check by speaker recognition.
Although SyncNet worked well for videos in simple genres, it failed for videos of complex genres such as movie and vlog. A possible reason is that the video content of these genres may change dramatically in time, which leads to unreliable estimation for the stream of the mouth movement, hence unreliable synchronization detection. In order to improve the robustness of the active speaker verification in complex genres, we introduced a double check procedure based on speaker recognition. The idea is simple: whenever the speaker recognition system states a very low confidence for the target speaker, the segment will be discarded even if the confidence from SyncNet is high; vice versa, if the speaker recognition system states a very high confidence, the segment will be retained. We used an off-the-shelf speaker recognition system BIBREF20 to perform this double check. In our study, this double check improved the recall rate by 30% absolutely.
STEP 6. Human check.
The segments produced by the above automated pipeline were finally checked by human. According to our experience, this human check is rather efficient: one could check 1 hour of speech in 1 hour. As a comparison, if we do not apply the automated pre-selection, checking 1 hour of speech requires 4 hours.
In this section, we present a series of experiments on speaker recognition using VoxCeleb and CN-Celeb, to compare the complexity of the two datasets.
VoxCeleb: The entire dataset involves two parts: VoxCeleb1 and VoxCeleb2. We used SITW BIBREF21, a subset of VoxCeleb1 as the evaluation set. The rest of VoxCeleb1 was merged with VoxCeleb2 to form the training set (simply denoted by VoxCeleb). The training set involves $1,236,567$ utterances from $7,185$ speakers, and the evaluation set involves $6,445$ utterances from 299 speakers (precisely, this is the Eval. Core set within SITW).
CN-Celeb: The entire dataset was split into two parts: the first part CN-Celeb(T) involves $111,260$ utterances from 800 speakers and was used as the training set; the second part CN-Celeb(E) involves $18,849$ utterances from 200 speakers and was used as the evaluation set.
Two state-of-the-art baseline systems were built following the Kaldi SITW recipe BIBREF22: an i-vector system BIBREF3 and an x-vector system BIBREF10.
For the i-vector system, the acoustic feature involved 24-dimensional MFCCs plus the log energy, augmented by the first- and second-order derivatives. We also applied the cepstral mean normalization (CMN) and the energy-based voice active detection (VAD). The universal background model (UBM) consisted of $2,048$ Gaussian components, and the dimensionality of the i-vector space was 400. LDA was applied to reduce the dimensionality of the i-vectors to 150. The PLDA model was used for scoring BIBREF4.
For the x-vector system, the feature-learning component was a 5-layer time-delay neural network (TDNN). The slicing parameters for the five time-delay layers were: {$t$-2, $t$-1, $t$, $t$+1, $t$+2}, {$t$-2, $t$, $t$+2}, {$t$-3, $t$, $t$+3}, {$t$}, {$t$}. The statistic pooling layer computed the mean and standard deviation of the frame-level features from a speech segment. The size of the output layer was consistent with the number of speakers in the training set. Once trained, the activations of the penultimate hidden layer were read out as x-vectors. In our experiments, the dimension of the x-vectors trained on VoxCeleb was set to 512, while for CN-Celeb, it was set to 256, considering the less number of speakers in the training set. Afterwards, the x-vectors were projected to 150-dimensional vectors by LDA, and finally the PLDA model was employed to score the trials. Refer to BIBREF10 for more details.
We first present the basic results evaluated on SITW and CN-Celeb(E). Both the front-end (i-vector or x-vector models) and back-end (LDA-PLDA) models were trained with the VoxCeleb training set. Note that for SITW, the averaged length of the utterances is more than 80 seconds, while this number is about 8 seconds for CN-Celeb(E). For a better comparison, we resegmented the data of SITW and created a new dataset denoted by SITW(S), where the averaged lengths of the enrollment and test utterances are 28 and 8 seconds, respectively. These numbers are similar to the statistics of CN-Celeb(E).
The results in terms of the equal error rate (EER) are reported in Table TABREF24. It can be observed that for both the i-vector system and the x-vector system, the performance on CN-Celeb(E) is much worse than the performance on SITW and SITW(S). This indicates that there is big difference between these two datasets. From another perspective, it demonstrates that the model trained with VoxCeleb does not generalize well, although it has achieved reasonable performance on data from a similar source (SITW).
To further compare CN-Celeb and VoxCeleb in a quantitative way, we built systems based on CN-Celeb and VoxCeleb, respectively. For a fair comparison, we randomly sampled 800 speakers from VoxCeleb and built a new dataset VoxCeleb(L) whose size is comparable to CN-Celeb(T). This data set was used for back-end (LDA-PLDA) training.
The experimental results are shown in Table TABREF26. Note that the performance of all the comparative experiments show the same trend with the i-vector system and the x-vector system, we therefore only analyze the i-vector results.
Firstly, it can be seen that the system trained purely on VoxCeleb obtained good performance on SITW(S) (1st row). This is understandable as VoxCeleb and SITW(S) were collected from the same source. For the pure CN-Celeb system (2nd row), although CN-Celeb(T) and CN-Celeb(E) are from the same source, the performance is still poor (14.24%). More importantly, with re-training the back-end model with VoxCeleb(L) (4th row), the performance on SITW becomes better than the same-source result on CN-Celeb(E) (11.34% vs 14.24%). All these results reconfirmed the significant difference between the two datasets, and indicates that CN-Celeb is more challenging than VoxCeleb.
We introduced a free dataset CN-Celeb for speaker recognition research. The dataset contains more than $130k$ utterances from $1,000$ Chinese celebrities, and covers 11 different genres in real world. We compared CN-Celeb and VoxCeleb, a widely used dataset in speaker recognition, by setting up a series of experiments based on two state-of-the-art speaker recognition models. Experimental results demonstrated that CN-Celeb is significantly different from VoxCeleb, and it is more challenging for speaker recognition research. The EER performance we obtained in this paper suggests that in unconstrained conditions, the performance of the current speaker recognition techniques might be much worse than it was thought.

Figure and table captions:
Table 2. The distribution over utterance length.
Table 1. The distribution over genres.
Table 3. Comparison between CN-Celeb and VoxCeleb.
Table 4. EER(%) results of the i-vector and x-vector systems trained on VoxCeleb and evaluated on three evaluation sets.
Table 5. EER(%) results with different data settings.

Questions about this paper:
What was the performance of both approaches on their dataset?
What genres are covered?
Which of the two speech recognition models works better overall on CN-Celeb?
By how much is performance on CN-Celeb inferior to performance on VoxCeleb?

Answers:
ERR of 19.05 with i-vectors and 15.52 with x-vectors
genre, entertainment, interview, singing, play, movie, vlog, live broadcast, speech, drama, recitation and advertisement
x-vector
For i-vector system, performances are 11.75% inferior to voxceleb. For x-vector system, performances are 10.74% inferior to voxceleb