## 4 Experimental Results and Analysis
### Verification for Automatic Language Identification Using R-Vector
#### 4.3.1 Evaluation and Analysis of Automatic LID
Next, we investigated the performance of three-class LID discriminating English, Spanish, and Chinese. As shown in Table 7, the accuracy was significantly degraded in comparison with two-class LID. However, the experiment derived a common tendency between LID approaches that was observed in Table 6. A notable result is that the R-vector employing i-vector achieved the same performance as the CNN-based approach, making the best accuracy. In order to observe the efficiency of the proposed R-vector-based approach in a more sophisticated manner, we analyzed the results of the confusion matrix shown in Figure 10. Among the matrices, the experiment on the SiTEC corpus provided more notable LID accuracy compared to that on the Mozilla corpus, showing stable LID performance in both English and Korean. In a three-class LID experiment, our system identified China better than other languages, achieving about 68% accuracy. Meanwhile, English and Spanish indicated similar recognition results. \begin{table}
\begin{tabular}{c c c c c} \hline \hline  & **Baseline Approaches** & **ER (\%)** & **Proposed Approaches** & **ER (\%)** \\ \hline \multirow{2}{*}{CN vs. EN} & CNN & 31.14 & R-vector with SVM & 36.05 \\  & i-vector & 32.26 & R-vector with i-vector & 31.78 \\ \hline \multirow{2}{*}{CN vs. SP} & CNN & 30.19 & R-vector with SVM & 32.62 \\  & i-vector & 23.71 & R-vector with i-vector & 23.50 \\ \hline \multirow{2}{*}{EN vs. SP} & CNN & 37.08 & R-vector with SVM & 43.88 \\  & i-vector & 38.58 & R-vector with i-vector & 38.32 \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Performance (Error Rate (ER)) of two-class LID experiments on Mozilla corpus. CN: Chinese, EN: English, SP: Spanish. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Baseline Approaches** & **ER (\%)** & **Proposed Approaches** & **ER (\%)** \\ \hline \multirow{2}{*}{CNN} & CNN & 47.38 & R-vector with SVM & 53.27 \\  & i-vector & 48.59 & R-vector with i-vector & 47.38 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Performance (Error Rate (ER)) of three-class LID experiments on Mozilla corpus. Finally, we attempted to investigate the efficiency of the proposed approach on an evaluation dataset specialized for the LID task. We selected the '2011 National Institute of Standards and Technology (NIST) Language Recognition Evaluation (LRE) Test Set' published by the Linguistic Data Consortium (LDC), as LRE datasets are known as representative evaluation data for language identification. The dataset is characterized as conversational telephone speech recorded in real-life environments, which is similar to the Mozilla corpus. For this reason, we used classification models trained with the Mozilla data to evaluate the LRE test set in consideration of cross-corpus evaluation. To compare with the Mozilla results, we conducted two-class and three-class LID experiments using English, Spanish, and Chinese. Tables 8 and 9 demonstrate the results. Due to the difference between the training data and test data, the LRE dataset provided lower accuracy compared to the Mozilla results, while showing a similar performance tendency to the Mozilla data. In two-class experiments, the EN-SP pair also represented the worst performance among the three language pairs. Another similar tendency was shown in the CN-SP pair, in which the i-vector approach provided higher performance than that of the CNN. Outstanding results were observed in the performance of the proposed R-vector. Although the proposed R-vector-based approaches indicated the worst performance in the Mozilla results, the approaches demonstrated better or similar accuracy in comparison with the conventional approaches in the LRE evaluation. Next, in three-class experiments, the proposed approaches employing R-vector demonstrated an LID performance comparable to the conventional approaches  
In several LID experiments, the proposed R-vector-based approaches demonstrated slightly different performance according to speech corpora.