Introduction
One of the challenges of processing real-world spoken content, such as media broadcasts, is the potential presence of different dialects of a language in the material. Dialect identification can be a useful capability to identify which dialect is being spoken during a recording. Dialect identification can be regarded as a special case of language recognition, requiring an ability to discriminate between different members within the same language family, as opposed to across language families (i.e., for language recognition). The dominant approach, based on i-vector extraction, has proven to be very effective for both language and speaker recognition BIBREF0 . Recently, phonetically aware deep neural models have also been found to be effective in combination with i-vectors BIBREF1 , BIBREF2 , BIBREF3 . Phonetically aware models could be beneficial for dialect identification, since they provide a mechanism to focus attention on small phonetic differences between dialects with predominantly common phonetic inventories.
Since 2015, the Arabic Multi-Genre Broadcast (MGB) Challenge tasks have provided a valuable resource for researchers interested in processing multi-dialectal Arabic speech. For the ASRU 2017 MGB-3 Challenge, there were two possible tasks. The first task was aimed at developing an automatic speech recognition system for Arabic dialectal speech based on a multi-genre broadcast audio dataset. The second task was aimed at developing an Arabic Dialect Identification (ADI) capability for five major Arabic dialects. This paper reports our experimentation efforts for the ADI task.
While the MGB-3 Arabic ASR task included seven different genres from the broadcast domain, the ADI task focused solely on broadcast news. Participants were provided high-quality Aljazeera news broadcasts as well as transcriptions generated by a multi-dialect ASR system created from the MGB-2 dataset BIBREF4 . The biggest difference from previous MGB challenges is that only a relatively small development set of in-domain data is provided for adaptation to the test set (i.e., the training data is mismatched with the test data). For the ADI baseline, participants were also provided with i-vector features from the audio dataset, and lexical features from the transcripts. Evaluation software was shared with all participants using baseline features available via Github.
The evaluation scenario for the MGB-3 ADI task can be viewed as channel and domain mismatch because the recording environment of the training data is different from the development and test data. In general, channel or domain mismatch between training and test data can be a significant factor affecting system performance. Differences in channel, genre, language, topic etc. produce shifts in low-dimensional projections of the corresponding speech and ultimately cause performance degradations on evaluation data.
In order to address performance degradation of speaker and language recognition systems due to domain mismatches, researchers have proposed various approaches to compensate for, and to adapt to the mismatch BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 , BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 , BIBREF14 , BIBREF15 , BIBREF16 , BIBREF17 . For the MGB-3 ADI task, we utilized the development data to adapt to the test data recording domain, and investigated approaches to improve ADI performance both on the domain mismatched scenario, and the matching scenario, by using a recursive whitening transformation, a weighted dialect i-vector model, and a Siamese Neural Network.
In contrast to the language recognition scenario, where there are different linguistic units across languages, language dialects typically share a common phonetic inventory and written language. Thus, we can potentially use ASR outputs such as phones, characters, and lexicons as features. N-gram histograms of phonemes, characters and lexicons can be used as feature vectors directly, and indeed, a lexicon-based n-gram feature vector was provided for the MGB-3 ADI baseline. The linguistic feature space is, naturally, completely different to the audio feature space, so a fusion of the results from both feature representations has been previously shown to be beneficial BIBREF18 , BIBREF19 , BIBREF20 , BIBREF21 , BIBREF22 . Moreover, the linguistic feature has an advantage in channel domain mismatch situations because the transcription itself does not reflect the recording environment, and only contains linguistic information.
In this paper, we describe our work for the MGB-3 ADI Challenge. The final MIT-QCRI submitted system is a combination of audio and linguistic feature-based systems, and includes multiple approaches to address the challenging mismatched conditions. From the official results, this system achieved the best performance among all participants. The following sections describe our research in greater detail.
MGB-3 Arabic Dialect Identification
For the MGB-3 ADI task, the challenge organizers provided 13,825 utterances (53.6 hours) for the training (TRN) set, 1,524 utterances (10 hours) for a development (DEV) set, and 1,492 utterances (10.1 hours) for a test (TST) set. Each dataset consisted of five Arabic dialects: Egyptian (EGY), Levantine (LEV), Gulf (GLF), North African (NOR), and Modern Standard Arabic (MSA). Detailed statistics of the ADI dataset can be found in BIBREF23 . Table TABREF3 shows some facts about the evaluation conditions and data properties. Note that the development set is relatively small compared to the training set. However, it is matched with the test set channel domain. Thus, the development set provides valuable information to adapt or compensate the channel (recording) domain mismatch between the train and test sets.
Dialect Identification Task & System
The MGB-3 ADI task asks participants to classify speech as one of five dialects, by specifying one dialect for each audio file for their submission. Performance is evaluated via three indices: overall accuracy, average precision, and average recall for the five dialects.
Baseline ADI System
The challenge organizers provided features and code for a baseline ADI system. The features consisted of 400 dimensional i-vector features for each audio file (based on bottleneck feature inputs for their frame-level acoustic representation), as well as lexical features using bigrams generated from transcriptions BIBREF23 . For baseline dialect identification, a multi-class Support Vector Machine (SVM) was used. The baseline i-vector performance was 57.3%, 60.8%, and 58.0% for accuracy, precision and recall respectively. Lexical features achieved 48.4%, 51.0%, and 49.3%, respectively. While the audio-based features achieved better performance than the lexical features, both systems only obtained approximately 50% accuracy, indicating that this ADI task is difficult, considering that there are only five classes to choose from.
Siamese Neural Network-based ADI
To further distinguish speech from different Arabic dialects, while making speech from the same dialect more similar, we adopted a Siamese neural network architecture BIBREF24 based on an i-vector feature space. The Siamese neural network has two parallel convolutional networks, INLINEFORM0 , that share the same set of weights, INLINEFORM1 , as shown in Figure FIGREF5 (a). Let INLINEFORM2 and INLINEFORM3 be a pair of i-vectors for which we wish to compute a distance. Let INLINEFORM4 be the label for the pair, where INLINEFORM5 = 1 if the i-vectors INLINEFORM6 and INLINEFORM7 belong to same dialect, and INLINEFORM8 otherwise. To optimize the network, we use a Euclidean distance loss function between the label and the cosine distance, INLINEFORM9 , where INLINEFORM10
For training, i-vector pairs and their corresponding labels can be processed by combinations of i-vectors from the training dataset. The trained convolutional network INLINEFORM0 transforms an i-vector INLINEFORM1 to a low-dimensional subspace that is more robust for distinguishing dialects. A detailed illustration of the convolutional network INLINEFORM2 is shown in Figure FIGREF5 (b). The final transformed i-vector, INLINEFORM3 , is a 200-dimensional vector. No nonlinear activation function was used on the fully connected layer. A cosine distance is used for scoring.
i-vector Post-Processing
In this section we describe the domain adaptation techniques we investigated using the development set to help adapt our models to the test set.
Although the baseline system used an SVM classifier, Cosine Distance Scoring (CDS) is a fast, simple, and effective method to measure the similarity between an enrolled i-vector dialect model, and a test utterance i-vector. Under CDS, ZT-norm or S-norm can be also applied for score normalization BIBREF25 . Dialect enrollment can be obtained by means of i-vectors for each dialect, and is called the i-vector dialect model: INLINEFORM0 , where INLINEFORM1 is the number of utterances for each dialect INLINEFORM2 . Since we have two datasets for dialect enrollment, INLINEFORM3 for the training set, and INLINEFORM4 for the development set, we use an interpolation approach with parameter INLINEFORM5 , where INLINEFORM6
We observed that the mismatched training set is useful when combined with matched development set. Figure FIGREF7 shows the performance evaluation by parameter INLINEFORM0 on the same experimental conditions of System 2 in Section 4.3. This approach can be thought of as exactly the same as score fusion for different system. However, score fusion is usually performed at the system score level, while this approach uses a combination of knowledge of in-domain and out-of-domain i-vectors with a gamma weight on a single system.
For i-vector-based speaker and language recognition approaches, a whitening transformation and length normalization is considered essential BIBREF26 . Since length normalization is inherently a nonlinear, non-whitening operation, recently, a recursive whitening transformation has been proposed to reduce residual un-whitened components in the i-vector space, as illustrated in Figure FIGREF10 BIBREF14 . In this approach, the data subset that best matches the test data is used at each iteration to calculate the whitening transformation. In our ADI experiments, we applied 1 to 3 levels of recursive whitening transformation using the training and development data.
Phoneme Features
Phoneme feature extraction consists of extracting the phone sequence, and phone duration statistics using four different speech recognizers: Czech, Hungarian, and Russian using narrowband model, and English using a broadband model BIBREF27 . We evaluated the four systems using a Support Vector Machine (SVM). The hyper-parameters for the SVM are distance from the hyperplane (C is 0.01), and penalty l2. We used the training data for training the SVM and the development data for testing. Table TABREF13 shows the results for the four phoneme recognizers. The Hungarian phoneme recognition obtained the best results, so we used it for the final system combination.
Character Features
Word sequences are extracted using a state-of-the-art Arabic speech-to-text transcription system built as part of the MGB-2 BIBREF28 . The system is a combination of a Time Delayed Neural Network (TDNN), a Long Short-Term Memory Recurrent Neural Network (LSTM) and Bidirectional LSTM acoustic models, followed by 4-gram and Recurrent Neural Network (RNN) language model rescoring. Our system uses a grapheme lexicon during both training and decoding. The acoustic models are trained on 1,200 hours of Arabic broadcast speech. We also perform data augmentation (speed and volume perturbation) which gives us three times the original training data. For more details see the system description paper BIBREF4 . We kept the <UNK> from the ASR system, which indicates out-of-vocabulary (OOV) words, we replaced it with special symbol. Space was inserted between all characters including the word boundaries. An SVM classifier was trained similarly to the one used for the phoneme ASR systems, and we achieved 52% accuracy, 51.2% precision and 51.8% recall. The confusion matrix is different between the phoneme classifier and the character classifier systems, which motivates us to use both of them in the final system combination.
Score Calibration
All scores are calibrated to be between 0 and 1. A linear calibration is done by the Bosaris toolkit BIBREF29 . Fusion is also done in a linear manner.
ADI Experiments
For experiments and evaluation, we use i-vectors and transcriptions that are provided by the challenge organizers. Please refer to BIBREF23 for descriptions of i-vector extraction and Arabic speech-to-text configuration.
Using Training Data for Training
The first experiment we conducted used only the training data for developing the ADI system. Thus, the interpolated i-vector dialect model cannot be used for this experimental condition. Table TABREF14 shows the performance on dimension reduced i-vectors using the Siamese network (Siam i-vector), and Linear Discriminant Analysis (LDA i-vector), as compared to the baseline i-vector system. LDA reduces the 400-dimension i-vector to 4, while the Siamese network reduces it from 400 to 200. Since the Siamese network used a cosine distance for the loss function, the Siam i-vector showed better performance with the CDS scoring method, while others achieved better performance with an SVM. The best system using Siam i-vector showed overall 10% better performance accuracy, as compared to the baseline.
Using Training and Development Data for Training
For our second experiment, both the training and development data were used for training. For phoneme and character features, we show development set experimental results in Table TABREF15 . For i-vector experiments, we show results in Table TABREF16 . In the table we see that the interpolated dialect model gave significant improvements in all three metrics. The recursive whitening transformation gave slight improvements on the original i-vector, but not after LDA and the Siamese network. The best system is the original i-vector with recursive whitening, and an interpolated i-vector dialect model, which achieves over 20% accuracy improvement over the baseline.
While the Siamese i-vector network helped in the training data only experiments, it does not show any advantage over the baseline i-vector for this condition. We suspect this result is due to the composition of the data used for training the Siamese network. To train the network, i-vector pairs are chosen from from training dataset. We selected the pairs using both the training and development datasets. However, if we could put more emphasis on the development data, we suspect the Siamese i-vector network would be more robust on the test data. We plan to further examine the performances due to different compositions of data in the future.
Performance Evaluation of Submission
Tables TABREF21 and TABREF22 show detailed performance evaluations of our three submitted systems. System 1 was trained using only the training data as shown in Table TABREF21 . Systems 2 and 3 were trained using both the training and development sets as shown in Table TABREF22 . We found the best linear fusion weight based on System 1 to prevent over-fitting was 0.7, 0.2 and 0.1 for i-vector, character, and phonetic based scores respectively. We applied the same weights to Systems 2 and 3 for fusion.
From Table TABREF21 , we see that the Siamese network demonstrates its effectiveness on both the development and test sets without using any information of the test domain. The interpolated i-vector dialect model also demonstrates that it reflects test domain information well as shown by Systems 2 and 3 in Table TABREF22 . Although we expected that the linguistic features would not affected by the domain mismatch, character and phoneme features show useful contributions for all systems. We believe the reason for the performance degradation of Systems 2 and 3 after fusion on the development data can be seen in the fusion rule. We applied the fusion rule derived from System 1 which was not optimal for Systems 2 and 3, considering the development set evaluation. By including the development data as part of their training, Systems 2 and 3 are subsequently overfit on the development data, which was why we used the fusion rule of System 1. From the excellent fusion performance on the test data for Systems 2 and 3, we believe that the fusion rule from System 1 prevented an over-fitted result.
Conclusion
In this paper, we describe the MIT-QCRI ADI system using both audio and linguistic features for the MGB-3 challenge. We studied several approaches to address dialect variability and domain mismatches between the training and test sets. Without knowledge of the test domain where the system will be applied, i-vector dimensionality reduction using a Siamese network was found to be useful, while an interpolated i-vector dialect model showed effectiveness with relatively small amounts of test domain information from the development data. On both conditions, fusion of audio and linguistic feature guarantees substantial improvements on dialect identification. As these approaches are not limited to dialect identification, we plan to explore their utility on other speaker and language recognition problems in the future.

Question: What is the architecture of the siamese neural network?
Answer: two parallel convolutional networks, INLINEFORM0 , that share the same set of weights

Question: How do they explore domain mismatch?
Answer: Unanswerable

Question: How do they explore dialect variability?
Answer: Unanswerable

Question: Which are the four Arabic dialects?
Answer: Egyptian (EGY), Levantine (LEV), Gulf (GLF), North African (NOR)