Question: Have the majority of NLP algorithms been trained and evaluated on text or the output of an ASR system?
Evidence: Spoken language understanding is a challenging and difficult task BIBREF3. All problems present in natural language understanding are significantly exacerbated by several factors related to the characteristics of spoken language.
Firstly, each ASR engine introduces a mixture of systematic and stochastic errors which are intrinsic to the procedure of transcription of spoken audio. The quality of transcription, as measured by the popular word error rate (WER), attains the level of 5%-15% WER for high quality ASR systems for English BIBREF4, BIBREF5, BIBREF6, BIBREF7. The WER highly depends on the evaluation data difficulty and the speed to accuracy ratio. Importantly, errors in the transcription appear stochastically, both in audio segments which carry important semantic information, as well as in inessential parts of the conversation.
Another challenge stems from the fact that the structure of conversation changes dramatically when a human assumes an agency in the other party. When humans are aware that the other party is a machine (as is the case in dialogue chatbot interfaces), they tend to speak in short, well-structured turns following the subject-verb-object (SVO) sentence structure with a minimal use of relative clauses BIBREF8. This structure is virtually nonexistent in spontaneous speech, where speakers allow for a non-linear flow of conversation. This flow is further obscured by constant interruptions from backchannel or cross-talk utterances, repetitions, non-verbal signaling, phatic expressions, linguistic and non-linguistic fillers, restarts, and ungrammatical constructions.
A substantial, yet often neglected difficulty, stems from the fact that most SLU tasks are applied to transcript segments representing a single turn in a typical scripted conversation. However, turn-taking in unscripted human-human conversations is far more haphazard and spontaneous than in scripted dialogues or human-bot conversations. As a result, a single logical turn may span multiple ASR segments and be interwoven with micro-turns of the other party or the contrary - be part of a larger segment containing many logical turns.
ASR transcripts lack punctuation, normalization, true-casing of words, and proper segmentation into phrases as these features are not present in the conversational input BIBREF9. These are difficult to correct as the majority of NLP algorithms have been trained and evaluated on text and not on the output of an ASR system. Thus, a simple application of vanilla NLP tools to ASR transcripts seldom produces actionable and useful results.
Finally, speech based interfaces are defined by a set of dimensions, such as domain and vocabulary (retail, finance, entertainment), language (English, Spanish), application (voice search, personal assistant, information retrieval), and environment (mobile, car, home, distant speech recognition). These dimensions make it very challenging to provide a low cost domain adaptation.
Last but not least, production ASR systems impose strict constraints on the additional computation that can be performed. Since we operate in a near real-time environment, this precludes the use of computationally expensive language models which could compensate for some of the ASR errors.
ASR transcripts lack punctuation, normalization, true-casing of words, and proper segmentation into phrases as these features are not present in the conversational input (Zelasko et al. ˙ , 2018). These are difficult to correct as the majority of NLP algorithms have been trained and evaluated on text and not on the output of an ASR system. Thus, a simple application of vanilla NLP tools to ASR transcripts seldom produces actionable and useful results.
Answer: On text.