Planning for a Corpus of Continuous Ratings of Spoken Dialog Quality

While many aspects of speech processing, including speech recognition and speech synthesis, have seen enormous advances over the past few years, advances in dialog have been more modest. This difference is largely attributable to the lack of resources that can support machine learning of dialog models and dialog phenomena. The research community accordingly needs a corpus of spoken dialogs with quality annotations every 100 milliseconds or so. We envisage a large and diverse collection: on the order of fifty hours of data, representing hundreds of speakers and many genres, with every instant labeled for interaction quality by one or more human judges. To make it maximally useful, its design will be a community effort. This technical report is an edited version of a proposal to the National Science Foundation, submitted to the CISE Community Research Infrastructure Program in February 2019. I thank David DeVault, Milica Gasic, Kallirroi Georgila, Svetlana Stoyanchev, Tatsuya Kawahara, Olac Fuentes, and David Novick for helpful discussions. 1 Motivation: Enabling Faster Progress in Spoken Dialog Systems Research and Development Dialog is a uniquely powerful form of interaction. Text-based interactions have their place, but if we want to get to know someone, negotiate plans, make lasting decisions, get considered advice, resolve a workplace issue, or have fun together, we usually seek a realtime spoken dialog, face to face or by phone. Sometimes it would be helpful if computers could interact with us in the same way, for example as a coaches, tutors, or workplace assistants, but this is currently beyond the state of the art, and spoken language systems today are mostly confined to a few usage niches. Siri, for example, was initially designed to avoid dialog if at all possible, to instead get the job done with one response to one input, and this is still a common strategy. Indeed, at SLT 2018 (the IEEE Spoken Language Technology Workshop), one presenter, the lead developer of a well-known personal assistant system, revealed that for his system the average dialog length is 1.1 turns. While impressive in some ways, this is also testament to the difficulty of supporting true dialogs today. In the research arena, researchers have shown how we can do better, producing prototype systems with amazing responsiveness. For example, Gratch produced a system capable of active listening better than most people, DeVault produced a system capable of in-game collaboration as fast as the average human, Litman demonstrated a system that could pick up on subtle indications of a student’s cognitive state and respond in ways that increased learning gains, Acosta produced a system that prosodically tailored its utterances to show empathy and thereby establish rapport, and Yu demonstrated how a robot through dialog could shape the user’s attention [1, 2, 3, 4, 5]. These illustrate that there are many ways in which the functionality and usability of dialog systems could be greatly improved. However such capabilities have not been taken up in commercial systems. One major reason is that their development involved custom corpora, careful policy design, and intense engineering and tuning. These do not scale. In contrast, the astounding recent advances in other areas of speech and language technology — speech recognition, speech synthesis, machine translation, speaker identification, emotion recognition, and many others — have been enabled by the application of deep learning models to large corpora. To enable rapid advances also in spoken dialog, the research community needs large corpora of data with suitable annotations to support deep learning approaches. In this project we propose to lay the groundwork for developing such corpora. 2 Research Landscape: How Spoken Dialog Systems are Built and Trained Most task-oriented dialog systems are today designed by hand. This is because creating effective dialog today, even for something as familiar as automated banking, is as much an art as a science, both in the creation phase and in the refinement/tuning phase. Developers tend to be conservative and follow heuristics, such as avoid overlap at all costs, and avoid prosodic variation in favor of conveying everything explicitly with words. Such heuristics originated in the early days of dialog systems development, when the component technologies were far inferior to where they are today. They are ultimately rooted in informal quality judgments made by developers: they apply their intuitions about what users like and don’t like, and may augment these with simple call-log statistics, for example to find reasons for glaring quality issues, such as those leading to dialog breakdown or call abandonment [6, 7]. In the research arena, by contrast, the current mainstream eschews questions of design in favor of developing ways to tune dialog systems from data. The general strategy is to devise cost functions and then create models that can be trained to optimize them. There are three main families of approaches. 1. For developing chat systems, standard practice is to train a system to match observed behavior, for example in choosing word sequences based on the degree of match to sequences seen in response to similar inputs in training data, such as chat corpora or movie screenplays. This approach is the core of neural conversational models, end-to-end dialog models and related approaches, which over the past three years have, from a few seminal papers [8, 9, 10, 11] grown to a tidal wave of work, with many hundreds of publications so far. While to date applied, as far as we know, only to text and to the content aspects of spoken dialog, it is only a matter of time before we see work generalizing these models to handle spoken interaction. At the same time, this family of approaches suffers from crippling problems with the metrics [12] (such as the commonly-used Bleu), which relate to simplistic concepts of match to observed behavior. In particular, human-human dialog interaction quality is not uniformly high, and there are many speakers and many actions which we do not want our dialog systems to mimic. (There are of course partial workarounds for this issue, including recording dialogs with an exemplary speaker, such as a champion-level customer service representative, and obtaining multiple response tracks, and then training the system to produce consensus behavior [13].) 2. For developing task-oriented dialog systems, there are more principled ways to optimize