Abstract: Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

Introduction: Spoken language understanding (SLU) systems infer the meaning or intent of a spoken utterance BIBREF0 . This is crucial for voice user interfaces, in which the speaker's utterance needs to be converted into an action or query. For example, for a voice-controlled coffee machine, an utterance like “make me a large coffee with two milks and a sugar, please” might have an intent representation like {drink: "coffee", size: "large", additions: [{type: "milk", count: 2}, {type: "sugar", count: 1}]}. The conventional SLU pipeline is composed of two modules: an automatic speech recognition (ASR) module that maps the speech to a text transcript, and a natural language understanding (NLU) module that maps the text transcript to the speaker's intent BIBREF1 , BIBREF2 , BIBREF3 . An alternative approach that is beginning to gain popularity is end-to-end SLU BIBREF4 , BIBREF5 , BIBREF6 , BIBREF7 , BIBREF8 . In end-to-end SLU, a single trainable model maps the speech audio directly to the speaker's intent without explicitly producing a text transcript (Fig. FIGREF4 ). Unlike the conventional SLU pipeline, end-to-end SLU: End-to-end models have been made possible by deep learning, which automatically learns hierarchical representations of the input signal BIBREF9 , BIBREF10 , BIBREF11 , BIBREF12 , BIBREF13 . Speech is natural to represent in a hierarchical way: waveform INLINEFORM0 phonemes INLINEFORM1 morphemes INLINEFORM2 words INLINEFORM3 concepts INLINEFORM4 meaning. However, because speech signals are high-dimensional and highly variable even for a single speaker, training deep models and learning these hierarchical representations without a large amount of training data is difficult. The computer vision BIBREF14 , BIBREF15 , natural language processing BIBREF16 , BIBREF17 , BIBREF18 , BIBREF19 , BIBREF20 , and ASR BIBREF21 , BIBREF22 communities have attacked the problem of limited supervised training data with great success by pre-training deep models on related tasks for which there is more training data. Following their lead, we propose an efficient ASR-based pre-training methodology in this paper and show that it may be used to improve the performance of end-to-end SLU models, especially when the amount of training data is very small. Our contributions are as follows:

Question: How was the dataset collected?
Answer: data was collected using crowdsourcing where speakers were recorded saying random ordered phrases for each intent twice