Performance Analysis of Speech Encoders for Low-Resource SLU and ASR in Tunisian Dialect (2024)

Abstract: Speech encoders pretrained through selfsupervised learning (SSL) have demonstrated remarkable performance in various downstream tasks, including Spoken Language Understanding (SLU) and Automatic Speech Recognition (ASR). For instance, fine-tuning SSL models for such tasks has shown significant potential, leading to improvements in the SOTA performance across challenging datasets. In contrast to existing research, this paper contributes by comparing the effectiveness of SSL approaches in the context of (i) the low-resource spoken Tunisian Arabic dialect and (ii) its combination with a low-resource SLU and ASR scenario, where only a few semantic annotations are available for fine-tuning. We conduct experiments using many SSL speech encoders on the TARIC-SLU dataset. We use speech encoders that were pre-trained on either monolingual or multilingual speech data. Some of them have also been refined without in-domain nor Tunisian data through multimodal supervised teacher-student paradigm. This study yields numerous significant findings that we are discussing in this paper.

heading: 1 Introduction
text: Self-supervised learning methods aim to train a representational model, also called upstream model, that benefits a collection of downstream tasks. SSL models are trained by using information extracted from the input data itself as the label to target. Various techniques have been introduced in the literature in order to learn powerful representations from the speech signal, including those based on autoregressive predictive coding (Chung et al., 2019), contrastive losses (Schneider et al., 2019; Baevski et al., 2020) and masked prediction (Liu et al., 2021; Chen et al., 2022; Hsu et al., 2021). Other works explored the combination of contrastive learning and masked language modeling
(Chung et al., 2021). The learned SSL models like wav2vec 2.0 (Baevski et al., 2020), wavLM (Chen et al., 2022), data2vec (Baevski et al., 2022b), data2vec 2.0 (Baevski et al., 2022a), ArTST (Toyin et al., 2023), w2v-BERT (Chung et al., 2021) have been proven effective for a wide range of tasks as they considerably lighten the amount of annotated speech data normally required for downstream tasks, while exploiting very large amounts of unlabeled data. Some models have been extended to a cross-lingual setting through XLS-R-128 (Babu et al., 2021), MMS (Pratap et al., 2023) and the recently released w2v-BERT 2.0 model (Barrault et al., 2023). Several efforts went further to enrich the frame-level speech representations with textual semantic information. In 2022, Khurana et al. (2022) proposed such a model called SAMUXLSR allowing the obtention of a better semantic encoding in the speech representations of a pretrained SSL model. SAMU-XLSR approach follows a teacher-student supervised learning framework that uses multilingual text/audio paired data and the Language-agnostic BERT Sentence Embedding (LaBSE) model (Feng et al., 2022). This teacher-student approach is used to refine an SSL pre-trained model dedicated to speech processing to make it able to produce sentence-level embeddings from speech similar to the embbedings given by the LaBSE model from the speech transcriptions. They use the CommonVoice version 8 corpora applied to 53 languages to refine the XLS-R128 model (Babu et al., 2021).
More Recently, a multilingual sentence embedding space for 200 text and 37 speech languages called SONAR (Duquenne et al., 2023) was introduced by META AI. The authors adopted a twostep training strategy: They first build a sentence embedding space for the multilingual text before extending this to the speech modality via a teacherstudent approach. Applied to many downstream tasks, SSL speech encoders have shown their great
130
potential by improving the state-of-the-art performances on challenging benchmark datasets. However, understanding speech encoder capabilities requires a benchmarking effort to compare and draw insights across the techniques.
Indeed, there has been a considerable amount of work and effort in order to benchmark such SSL models. SUPERB (Yang et al., 2021) is a good example of this effort. It provides a comprehensive speech SSL benchmark including tasks such as phoneme detection, ASR, slot filling, intent detection, keyword spotting, etc. The SUPERB benchmark has been extended to evaluate multilingual speech systems in the ML-SUPERB benchmark (Shi et al., 2023). XTREME-S (Conneau et al., 2022) is another example that focuses only on SSL models trained with multiple languages. Despite the large number of high-quality benchmarks that evaluate SSL models on various downstream tasks, a limited number of studies have probed their effectiveness for downstream tasks in spoken Arabic dialects.
In this paper, we propose to broaden the SSL benchmarking effort to the Automatic Speech Recognition (ASR) and Spoken Language Understanding (SLU) of spoken Arabic Dialects. The contributions of this work are as follows:
1. We benchmark the effectiveness of various state-of-the-art SSL speech encoders in the very challenging context of a low-resource spoken Arabic dialect (Tunisian dialect) with a limited training data;
2. We compare the performances of mono vs. multilingual/bilingual SSL models, and the impact of a semantic encoding refinement through a multimodal supervised teacherstudent approach;
3. We explore the use of recently released SSL models (w2v-BERT 2.0 and SONAR) for both ASR and SLU. To our knowledge, this is the first work in the literature to evaluate w2vBERT 2.0 and SONAR for an SLU task;
4. We release our code and models1 for reproducibility and to encourage future research on ASR and SLU of Arabic dialects.
1https://github.com/speechbrain/speechbrain/ tree/develop/recipes/TARIC

heading: 2 Benchmarking protocol
text: In this section, we formulate the downstream tasks and describe the Tunisian dialect dataset together with the speech encoders used in our study.

heading: 2.1 Downstream tasks
text: In this work, we address two tasks: Automatic Speech Recognition and Spoken Language Understanding of the Tunisian Arabic spoken dialect.
(a) ASR: Automatic Speech Recognition has the ultimate goal of providing the correct transcription given spoken utterance. It is used to assess the ability of SSL models to extract content information from audio inputs. The ASR task will be evaluated in terms of Word Error Rate (WER).
(b) SLU: Spoken language understanding refers to natural language processing tasks that aim to extract semantic information from speech (Tur and De Mori, 2011). Different tasks can be addressed as SLU tasks, such as named entity recognition from speech, dialog state tracking, intent recognition, slot filling, etc. . . In the context of a conventional dialogue system, information is typically represented through a semantic frame structure. For each utterance, constructing the semantic representation primarily involves (i) classifying the user’s utterance in terms of ‘speech acts’ (SA) (Searle, 1969) or ‘intents’ and (ii) slot filling (Wang et al., 2005). We consider these two SLU tasks following the available annotations: (1) Speech Act classification classifies speech utterances into predefined classes to determine the intent of the speaker; (2) Slot filling is a natural language processing and information extraction technique that entails the identification and extraction of specific pieces of information or attributes, referred to as ‘slots’, from unstructured text or spoken language. These slots are typically associated with predefined categories or entities. The SLU speech act recognition will be evaluated in terms of Speech Act Error Rate (SAER), which is a standard classification error rate. The SLU slot filling task will be evaluated in terms of Concept Error Rate (COER) and Concept/Value Error Rate (CVER). COER is
computed similarly to WER by taking into account only the semantic labels in the reference and hypothesis annotations. The CVER computation is identical, but the occurrences of concept/value pairs are taken into account instead of the concept alone. CVER implies that if any character within the word’s support prediction or the concept tag prediction differs from the reference, the entire prediction for that concept is considered as an error. For the calculation of these metrics, we have drawn on the detailed description in Laperrière et al. (2022).

heading: 2.2 Dataset
text: The ASR dataset TARIC (Masmoudi et al., 2014) has been used for this work along with its recent SLU enrichment, TARIC-SLU (Mdhaffar et al., 2024). The acquisition of the TARIC dataset was carried out in train stations in Tunisia. The dataset is made of human-human recordings with their manual transcriptions and semantic annotations. It is composed of more than 2,000 dialogues from 109 different speakers. The dataset2 is split into three parts (train, dev and test) as described in Table 1.
TARIC-SLU was annotated using 62 semantic concept tags such as city name arrival, departure time, ticket price, etc. and 3 speech acts (directivesanswer, directives-query and politeness). The example below shows the original sentence (a) together with its English translation (b) and the corresponding semantic annotation (c).
As for the annotation tags, the first word of each sentence represents the speech act tag (Directivesquery in this example). ‘<city-name-departure ’ is an opening tag starting the support word sequence ‘Sfax’ and expressing that this word sequence is associated with the city-name-departure semantic concept. The character ‘>’ represents the closing tag, and it is used to close all concept tags.
2https://demo-lia.univ-avignon.fr/ taric-dataset/
The same schema is used for the other semantic concepts. For speech act tags, there is no closing tag, since each sentence has a single speech act.
(a) ̄A ® áÓ HAK A¾ K Pð P ú æJ ¢«
@ ù ëCËAK.  ð éJ JÒ JË @ ©Ó ñ JË
(b) Please give me two tickets from Sfax to Tunis at eight thirty
(c) Directives-query please <command-task give me > <number-of-tickets two > <object tickets >
from <city-name-departure Sfax > to <city-name-arrival Tunis > at
<departure-time eight thirty >

heading: 2.3 SSL speech encoders
text: For our study, we used different types of speech encoders. We considered monolingual SSL speech encoders (French: wav2vec 2.0 LeBenchmark-7K; English: wav2vec 2.0 LV60, HuBERT, wavLM, data2vec 2.0 and SONAR-ENG) as well as crosslingual SSL speech encoders (wav2vec 2.0 VP100K, XLS-R-128, MMS, MMS-1B, w2v-BERT 2.0 and SAMU-XLSR) and two bilingual speech encoders (SONAR-ARB and SONAR-FRA). As explained in the introduction, SAMU-XLSR is a modified version of XLS-R-128 and SONAR models are a modified version of w2v-BERT 3. We put the URLs for all these models in the Appendix A.
SAMU-XLSR is based on the pre-trained multilingual XLS-R-128. SAMU-XLSR will process audio and text paired data. The XLS-R-128 model used in this approach was designed to generate speech representations for short 20 milliseconds speech frames. To make use of this model, SAMUXLSR performs pooling and projection to create a single sentence-level representation. In parallel, LaBSE (Feng et al., 2022) sentence-level textual representations are simply extracted. Both representations being on the same semantic space, SAMU- XLSR’s is then being pulled towards LaBSE’s with the help of a cosine similarity loss function. This means the parameters of all SAMUXLSR’s components are optimized to predict the textual representations generated by the frozen
3Notice that w2v-BERT and w2v-BERT 2.0 are two different models. Both of them are trained using the same architecture, but w2v-BERT is an English model and w2v-BERT 2.0 is a multilingual model. w2v-BERT has not been released to the research community, so we cannot evaluate its performance.
LaBSE model. Figure 1 illustrates the training process of SAMU-XLSR.
SONAR is a new multilingual and -modal text embedding space trained in encoder-decoder architecture for 200 languages, which substantially outperforms existing approaches like LASER3 (Heffernan et al., 2022) or LaBSE in multilingual similarity search. Authors apply a teacher-student approach to extend this embedding space to the speech modality and currently cover 36 languages.
Mining is performed in data from publicly available repositories of web data (tens of billions of sentences) and speech (4 million hours). In total, to train the model, they align more than 443,000 hours of speech with texts and create about 29,000 hours of speech-to-speech alignments. Figure 2 illustrates the training process of SONAR.
Overall, we experimented with 14 different speech encoders. Table 2 shows all the details of SSL speech encoder models. In addition to the aforementioned SSL models, we also evaluated using the recently released Whisper models from OpenAI (Radford et al., 2023). Unlike self-supervised speech models, Whisper is
a multilingual ASR model trained using a large amount of labelled audio transcription data (680k hours). Using Whisper was motivated by the recent achievements reported by Wang et al. (2023). The detailed properties of Whisper models used in this study are presented in Table 3.

heading: 3 Experiments and Results

heading: 3.1 Training details
text: For comparison purposes, we set the same parameters for all the models using the different speech encoders.

heading: 3.1.1 ASR
text: In addition to the speech encoder model, we incorporate an extra layer with 1024 neurons and LeakyReLU as the activation function, followed by a fully-connected layer and a final 40-dimensional softmax layer, each dimension corresponding to a character. The weights of these two additional layers were randomly initialized, while the weights of the speech encoder part for SSL models of the neural architecture were initialized using the pretrained weights. The fine-tuning is done with the TARIC training set using a character-level CTC loss function. We optimize the loss with an Adam optimizer of learning rate = 0.0001 for both speech encoder, and Adadelta with learning rate = 1.0 for the linear layer.

heading: 3.1.2 SLU
text: We formulate the end-to-end SLU task as a character level prediction where slots are delimited by tagspecific special characters, as in Yadav et al. (2020); Ghannay et al. (2018); Mdhaffar et al. (2022). We also added the speech act token to the reference annotation as the first token of each sequence of words. This way, the end-to-end model learns to both classify the utterances in terms of speech act, and recognize slot/value pairs present in the speech segment. As input, the neural network receives a WAV audio file (PCM, 16 bits, 16kHz, signed integer), and the output is a transcription enriched with semantic labels and speech acts tags. After processing through the softmax layer (which has the size of 1064), the outputs are generated by a simple greedy decoder. We optimize the loss with an Adam optimizer of learning rate = 0.0001 for
440 characters that cover the alphabet of the TARIC dataset, 62 characters for slots, one character for closing slots and three characters for speech acts
both speech encoder, and Adadelta with learning rate = 1.0 for the linear layer.

heading: 3.1.3 Training details for ASR and SLU
text: We employed a batch size of 4 samples, distributed across 4 NVIDIA V100 32GB GPU cards. For MMS-1B, we used 4 NVIDIA A100 80GB GPU cards. We utilized two optimizers: Adadelta for updating the additional layers weights and Adam for fine-tuning the self-supervised learning (SSL) model. The initial learning rate for Adadelta was 1.0, while for Adam it was set to 0.0001. Our models were implemented using the SpeechBrain toolkit (Ravanelli et al., 2021). The speech encoders are obtained through the fairseq (Ott et al., 2019) framework for SAMU-XLSR 5, SONAR and data2vec 2.0 and through HuggingFace for the remaining of models. More details about our models and the configuration files will be publicly available as a part of the Speechbrain toolkit.

heading: 3.2 Results
text: In this section, we report, analyze and discuss the performance of our models across various dimensions. All the results, including both ASR and SLU evaluations, are presented in Table 4. ASR models are evaluated with WER and SLU models are evaluated using WER (after removing the semantic tags and the speech act tag from the system output), COER, SAER and CVER. The first part of the table shows results by using SSL models, and the second part shows results by using Whisper models. The first part is divided into three sub-parts according to the type of SSL models: (i) the first sub-part, colored blue for monolingual models, (ii) the second sub-part, colored pink for cross-lingual models, and (iii) the third sub-part, colored yellow for the cross-lingual model after a teacher-student multi-modal semantic training. The second part of this table is dedicated to the results obtained when using Whisper small and medium models.
Overall, except for the SLU SAER score on the dev set, the best ASR and SLU results are obtained when our models are trained using w2v-BERT 2.0 model. Below, an analysis of the obtained results according to the type of the used SSL models.
Monolingual models: When we compare the performances of models trained using monolingual SSL (blue section of the results table), we observe that WavLM is by far the best-performing model
5SAMU-XLSR is not yet publicly available; it has been kindly shared by the authors of Khurana et al. (2022)
for the ASR task with 32.28% and 26.7% WER for dev and test sets respectively. This is confirmed as well when evaluating WER of the SLU model output (column WER (SLU) in table 4). Regarding the SLU task, while we did not observe any emerged trend, we observed a better performance of data2vec 2.0 in terms of COER evaluation.
Cross-lingual models without teacher-student multi-modal semantic training: Results indicate that w2v-BERT 2.0 yields the best performance in both ASR and SLU tasks (expect for the SAER).
All cross-lingual models: When it comes to cross-lingual models, with (colored in pink) or without teacher-student multi-modal semantic training (colored in yellow), setting apart the w2vBERT 2.0 model, SAMU-XLSR clearly outperforms VP-100k and MMS based models.
Mono-lingual vs. cross-lingual models: If we compare performances between monolingual and cross-lingual SSL models, cross-lingual models achieve better results for the SLU task. Indeed, data2vec 2.0 shows competitive COER scores while its WER is higher compared to other models. This leads to the conclusion that while data2vec 2.0 demonstrates the ability to identify semantic
concepts, it falls short in providing accurate transcriptions.
Cross-lingual models with teacher-student refinement: As stated earlier, the SLU performances of VP-100K, XLSR-128, and MMS exhibit a consistent trend, with XLSR-128 showing the highest performance among them. Combining this multilingual SSL (XLS-R-128) and teacher-student multimodality training in one model (SAMU-XLSR) provides better results for both ASR and SLU. Shifting the focus to SONAR models, as expected, SONAR-ARB demonstrates superior performance compared to both SONAR-ENG and SONAR-FRA for both ASR and SLU tasks. SONAR models are based on the English version of w2v-BERT. Unfortunately, the w2v-BERT model used to initialize the speech encoders of SONAR is not available to assess its contribution.
Whisper models: Results show that Whispermedium outperform results obtained using Whisper-small.
Whisper models vs. all SSL models: If we compare performances between Whisper models (second part of the table) and all SSL models (first part of the table), almost SSL models achieve better
results for the slot filling SLU task. To transcribe audio files, WER obtained by the ASR or the SLU show competitive results. For example, ASR results for the dev set (32.5%) obtained by Whisper medium outperform almost all SSL models except for w2v-BERT 2.0, SAMU-XLSR and wavLM. This leads to the conclusion that while Whisper models demonstrates the ability to transcribe audio files, it falls short in providing accurate semantic extraction compared to the use of SSL speech encoders.

heading: 4 Discussion
text: We carried out some error analyses to quantify some SLU systems performance across different conditions. We focus on 6 SSL models: LeBenchmark, XLS-R-128, data2vec 2.0, SAMU-XLSR, w2v-BERT 2.0 and SONAR-ARB.

heading: 4.1 Acoustic complexity
text: First, we used the WER of the transcripts produced by the model giving the best ASR system (w2vBERT 2.0) to quantify the general complexity of the utterance. We defined three groups of segments with low (WER <= 20), medium (20 < WER <= 50) and high (WER > 50) complexity. Figure 3 shows that, as expected, the SLU performances are overall better for spoken utterances belonging to the low complexity group. The x-axis represents different levels of WER, and the y-axis represents COER. However, when we compare the COER of the SLU systems by group, we observe that w2vBERT performs better for segments with low WER. For segments with medium WER, we observe that all the systems have a comparable behavior except for LeBenchmark: the system shows a higher COER. For utterances hard to transcribe, SAMUXLSR is the best one followed by w2v-BERT 2.0.

heading: 4.2 Semantic complexity
text: In the semantic complexity analysis, we used the number of semantic tags per utterance in the reference as a proxy of the semantic complexity. Figure 4 shows the results obtained by the evaluated models. The x-axis represents the number of semantic tags in test utterances, and the y-axis represents COER. Across the board, utterances with two to six concepts seem to be the easiest. While data2vec 2.0 and SONAR-ARB perform worse when there are fewer semantic concepts to extract. They perform best when there are more than six.

heading: 5 Conclusion
text: Our study investigates the usage of various SSL speech encoders for a Spoken Language Understanding task in challenging circumstances characterized by a scarcity of both SLU and ASR training data and the low-resource characteristics of the targeted Tunisian Arabic dialect. Our findings emphasize the efficacy of SSL pre-trained speech encoders in such conditions, with notable success observed when employing the w2v-BERT 2.0 model, with 600 millions parameters trained on 4.5M hours of speech from 143 languages. Additionally, we highlight the noteworthy performance of data2vec 2.0, pre-trained on English monolingual data, particularly excelling in handling semantically complex utterances. These outcomes collectively provide valuable information for advancing SLU methodologies, especially in resource-limited linguistic contexts. Last, SAMU-XLSR provides very competitive results thanks to semantic enrichment made by the teacher student approach and in future work we plan to train a SAMU-w2vBERT 2.0 model to take benefit of the joint SAMU and w2v-BERT 2.0 capabilities.

heading: 6 Acknowledgements
text: This work was performed using HPC resources from GENCI-IDRIS (grant AD011012108R1) and received funding from the EU H2020 SELMA (grant No 957017), ANR TRADEF project (ANR-22-ASGC-0003) and ESPERANTO research and innovation programme under the Marie Skłodowska-Curie (grant No 101007666). We would like to especially thank our colleagues Sameer Khurana and Antoine Laurent for sharing their SAMU-XSLR model with us.

heading: A Appendix: URLs of all models used in this study
text: For reproducibility of all results obtained in this paper, we put in the table 5 the url for each model.

heading: Future Work
text: Last, SAMU-XLSR provides very competitive results thanks to semantic enrichment made by the teacher student approach and in future work we plan to train a SAMU-w2vBERT 2. 0 model to take benefit of the joint SAMU and w2v-BERT 2. 0 capabilities