Robust acoustic and semantic modeling in a telephone-based spoken dialog system

In this thesis, we investigate various robustness issues of speech recognition and spoken language understanding models in a telephone-ba sed spoken dialog system. We also show the feasibility of building a robus t, multi-domain telephone-based spoken dialog system framework that can be seamlessly used for multiple application domains in different languages wh ile using domaindependent resources. In automatic speech recognition, some level of robustness c an be achieved by using domain-specific recognition resources. In additio n, the existence of considerable acoustic similarity within speakers of the same gender, accent, and age-group suggests that the use of user-group depe ndent acoustic models can give improved recognition performance. In this t e is, we group users based on gender and accent to exploit the shared vocal c har cteristics of speakers in the same group. We show that a tremendous perform ance boost can be obtained by efficiently tailoring gender-dependent a coustic models trained on native US-English speech data to the particular v ocal characteristics of German-accented English speakers. We also demons trate the effectiveness of cross-language accent adaptation where native German enrollment data is used to adapt native US-English acoustic models to th e German accent. To use group-dependent acoustic models, one has to dis cern the group (i.e., gender and/or accent) of a speaker from a spoken utter anc . Most current approaches to accent recognition use accented speech d ata to train an accent recognizer. In this thesis, we demonstrate a high-pe rformance accent recognizer that can be trained on a merger of native speech da ta of two or more languages. We also build a gender recognizer using ceps tral features and effectively use it to load the acoustic model correspond ing to the recognized gender. In the domain of spoken language understanding, we introduc e a new approach to hierarchical semantic modeling that enriches a re cognized utterance with semantic information at various levels of detail. The model is essentially built by grouping semantically and hierarchical ly related low-level concepts into higher level structures using prior domain kn owledge and training examples. The proposed model possesses a number of featu res; namely, it offers a remarkable ambiguity resolution ability, high p redictive power and produces a structured, semantically rich information that is convenient for dialog management. Moreover, it is robust in that it success fully deals with utterances containing unseen observations, and a significa nt percentage of out-of-vocabulary words can be correctly labeled using the surrounding context. Besides, the model allows us to safely ignore semantic lly irrelevant speech recognition errors. The model is also suited to prope rly handle noisy input containing false starts, filled pauses, hesitations, etc. More importantly, the model can be readily trained on completely unlabeled dat a with relatively less human supervision. The required additional human effo rt to design the proposed hierarchical model is much less than the laborious and error-prone semantic annotation of a training data set or hand-crafting a semantic grammar as no particular linguistic expertise is required. Furt hermore, the resulting hierarchical model outperforms the flat-concept mo del and has been successfully used in our demonstration system. We demonstrate our approaches on two corpora in two applicat ion domains; namely, airline travel planning in English (©2001 Communic ator Evaluation) and train information inquiries in German (©ERBA). After ca refully building the required recognition resources for each applicatio n, we evaluate the performance of the models in real-time use and the usability of the system as a whole with actual test users. In general, the results obtai ned are promising while rooms for improvement have been identified. Most of the presented approaches in this thesis have been pub lished in appropriate international media.