# Bidirectional Representations for Low-Resource Spoken Language Understanding
## 6 Conclusions  
Despite the significant advancements in speech processing, there remains a crucial gap in the field concerning the development of speech representation models that capture semantics akin to the capabilities exhibited by language models. In this paper, we introduce a novel bidirectional representation model pretrained on an ASR task, which demonstrates remarkable effectiveness in transferring learned features to the domain of intent recognition, all without necessitating additional fine tuning. Our pretraining strategy incorporates CTC, together with an MLM objective, yielding speech features endowed with bidirectional contextual awareness. Throughout our research, we examined the representations in various layers of the model, particularly with regard to their performance when applied to previously unseen datasets. Our findings show the versatility of these representations in training an effective SLU model. Notably, our representations either match or surpass the performance of state-of-the-art models on the SLU task, especially prior to the fine-tuning phase. We have also introduced a novel approach leveraging class attention mechanisms to summarize sequences of representations. This approach not only proves highly efficient, owing to its transformer-like architecture and minimal parameter requirements, but also offers insights into the model's decision-making process, shedding light on the input patterns that significantly influence predictions. Furthermore, our combination of class attention and ASR pretraining exhibits substantial gains in terms of data efficiency, as evidenced in a low-resource scenario in which we intentionally constrained the number of training examples available for the SLU task. In an era of increasingly massive and data-hungry models, our research underscores the importance of pursuing resource-friendly and efficient solutions. Figure 4: Class attention weight visualization of the two layers of the downstream class attention model. Although the model receives sequences of embeddings, we label the graph with the corresponding tokens for demonstration purposes. The two layers focus on different positions to predict the intent and arguments. However, it is important to acknowledge a limitation of our presented model, namely its challenges when confronted with SLU datasets that differentiate between terms seldom encountered or entirely absent in the pretraining data. In such cases in which two words are largely interchangeable except for a few instances in the pretraining dataset, our model's representations may become overly similar, hindering the ability to discriminate between these distinct classes without fine tuning. Addressing this limitation will be a focal point of our future research endeavors. Conceptualization, Q.M., H.V.h. and M.-F.M. ; methodology, Q.M. ; software, Q.M. ; validation, Q.M., H.V.h. and M.-F.M. ; formal analysis, Q.M. ; investigation, Q.M. ; resources, H.V.h. ; data curation, Q.M. ; writing--original draft preparation, Q.M. ; writing--review and editing, Q.M. ; visualization, Q.M. ; supervision, H.V.h. and M.-F.M. ; project administration, H.V.h. and M.-F.M. ; funding acquisition, H.V.h. and M.-F.M. All authors have read and agreed to the published version of the manuscript. This research received funding from the Flemish Government under the \"Onderzoeksprogramma Artificie Intelligentie (AI) Vlaanderen\" program. Not applicable.