# Under-Representated Speech Dataset from Open Data: Case Study on the Romanian Language
## 6 Evaluation  
The recognition performance of several existing Romanian ASR systems is evaluated against the newly created dataset. RO-DS2 [33] and RO-DS2-ROBIN [34] are based on the DeepSpeech2 [35] architecture, with RO-DS2-ROBIN being trained with additional data and using an improved language model. RO-WAV2 VEC2 [36] is based on the Wav2 Vec2 architecture [37]. RO-Whisper [38] is based on the OpenAI Whisper [39] architecture. All the systems are fine-tuned on publicly available Romanian language speech data which do not cover the under-represented categories from the USPDATRO dataset, since such data were not available at the time of training of the models (for the purpose of this work, we do not retrain any models). Evaluating the performance of state-of-the-art pre-trained models on under-represented speech is relevant in order to understand the necessity of including more such data in the training of new models or in the fine-tuning process. If the model's performance on the USPDATRO dataset is significantly lower compared to regular data, this is a strong indication that a larger under-represented speech dataset is needed to be included in the model's training. Results are given in Table 6. The baseline values are the best scores reported in the corresponding system papers.  
The baseline results presented in Table 6 are computed on slightly different combinations of Romanian language corpora, which are described in Section 2. The Whisper baselines are an average of the results given on individual corpora by the authors. This accounts for the slight decrease in performance. Nevertheless, all models give significantly worse results on the USPDATRO dataset. Even for the best-performing system on USPDATRO (the RO-Whisper medium), the WER is double compared to the baseline. Unexpectedly, the medium variant of the Whisper model perfor  
\begin{table}
\begin{tabular}{r c c c c} \hline \hline  & \multicolumn{2}{c}{**Baseline**} & \multicolumn{2}{c}{**USPDATRO**} \\
**System** & **WER** & **CER** & **WER** & **CER** \\ \hline RO-DS2 & 0.0991 & 0.0280 & 0.5714 & 0.3638 \\ RO-DS2-ROBIN & 0.0991 & - & 0.6491 & 0.3381 \\ RO-WAV2 VEC2 & 0.1393 & 0.0983 & 0.9115 & 0.6675 \\ RO-Whisper medium & 0.1379 & - & **0.2800** & **0.1319** \\ RO-Whisper large-v2 & 0.1261 & - & 0.4330 & 0.2875 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Evaluation of several Romanian language ASR systems.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Indicator** & **Value** & **Indicator** & **Value** \\ \hline Text files & 2637 & UPOS Noun & 8471 \\ Sentences & 6652 & UPOS Verb & 5793 \\ Tokens & 48,530 & UPOS Adp & 4009 \\ Unique tokens & 8221 & UPOS Adv & 3717 \\ Unique lemmas & 5509 & UPOS Adj & 1952 \\ Hapax legomena & 5055 & UPOS Num & 615 \\ Avg. Sentence Length & 7.30 & UPOS PropN & 851 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Characteristics of the text part of the dataset.  
the large-v2 version. This is likely due to the few-shot training scenario employed for the model, where the total number of training epochs is limited to 10. Further training is expected to improve over the large-v2 results.  
We further expand on the work by [38] and evaluate Whisper models (without fine-tuning) on the test part of the Representative Corpus of Contemporary Romanian Language (CoRoLa) corpus and on the USPDATRO corpus. Results are given in Table 7. For each model, we report results with and without beam search. When the beam search algorithm is activated, a beam with size 32 is used. As expected, the error of the prediction decreases with the increase in model size. Thus, the best performing model on both CoRoLa and USPDATRO is the _large-v2_ variant of the model. Furthermore, the use of the beam search algorithm has an impact on the predictions, improving performance. However, even the best results on the USPDATRO dataset are below the CoRoLa-based results by 8% WER. This indicates the need to include under-represented speech during model training or fine-tuning.