# Bidirectional Representations for Low-Resource Spoken Language Understanding
## 5 Analysis  
### Low-Resource Scenario  
To explore low-resource scenarios, we split Grabo and Patience per speaker. For each speaker, we randomly select a fixed number of examples per class for the training set of the SLU model. By gradually increasing the size of the training set, we are able to measure the learning curve. This operation is performed three times per speaker, with different splits each time. We compute the micro-averaged F1 score for each experiment and report the average F1 score with its standard deviation in Figure 2.  
We compare our representations (MLM in Figure 2) with three types of features: NLP features generated by encoding the gold transcriptions with bert-base-dutch-cased [33], Pipeline features resulting from the encoding of ASR transcripts predicted by ESPnet's hybrid ASR model [12] trained on CGN with bert-base-dutch-cased and CLM features corresponding to the output of the penultimate layer of the ASR model. To provide a fair comparison, neither the NLP, ASR nor MLM model encountered training examples from the SLU dataset. In other words, we do not fine tune any of the representation models on the downstream datasets at this stage. Both the ASR and MLM models are trained on CGN, and the NLP model is trained on a collection of five text corpora for a total of about 2.4B tokens. Considering the amount of training data, it is no surprise to see that the gold transcript encoded with the NLP model performs very well in all data regimes. As expected, the ASR transcripts show the weakest performance. Converting features into discrete symbols forces the model to make decisions, resulting in potential errors from which the model \begin{table}
\begin{tabular}{l l c c c c c} \hline \hline  & & \multicolumn{2}{c}{**FSC (Ch.)**} & \multicolumn{2}{c}{**Smartlights**} & \multicolumn{2}{c}{**Smartlights (Ch.)**} \\  & & **Spk.** & **Utt.** & **Random** & **Spk.** & **Utt.** \\ \hline \multirow{3}{*}{Frozen} & MLM & **95.2** & **86.3** & **84.0** & **79.5** & **69.0** \\  & E2E SLU [25] & 90.9 & 73.4 & 83.2 & 73.2 & 67.4 \\  & ST-BERT [30] & - & - & 81.2 & - & - \\ \hline \multirow{3}{*}{Fine tuning} & MLM & **98.8** & **88.0** & 85.8 & 81.9 & 74.6 \\  & E2E SLU [25] & 92.3 & 78.3 & **88.0** & **82.6** & **78.5** \\ \cline{1-1}  & ST-BERT [30] & - & - & 84.7 & - & - \\ \cline{1-1}  & ESPnet-SLU [11] & 97.5 & 78.5 & - & - & - \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Test accuracies on the Fluent Speech Commands (challenge splits) and Smartlights datasets with random and challenge splits. In the first stage, the pretrained model is frozen, and the SLU layers are trained. In the second stage, the pretrained model is partially (or totally in [11]) fine-tuned. Figure 2: Comparison of different models trained with increasing train set sizes on (**a**) Grabo and (**b**) Patience. Each curve represents the F1 score on the test set as a function of the size of the training set (utterance count). The colored areas represent the 68% confidence interval. We use the F1 score for comparison with the NMF baseline [32]. cannot recover. In contrast, the CLM and MLM models use the hidden representations produced by the model, thus avoiding this problem. Nonetheless, we see that the CLM features perform worse than our model. The masked language modeling objective allows the MLM model to look at both right and left contexts, whereas the CLM model only has access to previous predictions. Because each predicted unit is conditioned on the whole sentence, the model is able to derive better representations than by only looking at the past. We also compare our results with the state of the art on both datasets [32]. This model combines ASR pretraining and an NMF decoder for intent recognition [32; 34]. The NMF decoder uses a bag-of-words approach with multihot intent representation, thus ignoring the order of the words in the sentence. Although this model was presented for dysarthric speech, results are available for both the Grabo and Patience datasets [32].