Geometric and landmark-based approaches to speech representation and recognition
Authors: Partha Niyogi
Venue:  (2008)

Anyone who has used an automatic speech recognition (ASR) system, either on a customer support line or on their own personal computer, knows firsthand there is vast room for improvement. While many state-of-the-art commercial systems perform very well in near-ideal environments, system robustness remains far below human levels. The prevailing hidden Markov model (HMM) based paradigm will undoubtedly see gains in future decades as increased computing capacity admits more complex acoustic models that encompass a range of acoustic environments. However, this engineering-driven approach to the speech recognition problem has yet to adequately exploit the current scientific understanding in speech-related fields. In order to address the shortcomings of current approaches, several research groups, armed with vastly expanded computing power and a more sophisticated statistical toolbox, have returned to the knowledge-based, pre-HMM school of thought popular from the 60s to the mid-80s. Underpinning this shift is the belief that significant technological progress will rely on a fundamental understanding of both how humans produce and perceive speech, as well as the nature of the speech signal itself. With this revived scientific paradigm as motivation, this document focuses on two broad, knowledge-based directions in speech representation and recognition. The first research track examines the implications of adopting a geometric perspective on the space of speech sounds. The basic acoustic physics governing the vocal apparatus imply that vowel sounds are restricted to a low-dimensional subspace (formally, they lie on or near a curved submanifold) of the high-dimensional input space. In addition to providing mathematical justification for the perceptual magnet effect and Stevens' quantal theory of speech, this fact suggests the suitability of manifold learning algorithms for the speech domain. With this motivation, two graph Laplacian-based manifold learning algorithms are presented. The first is an approach to intrinsic (i.e., in the basis of the manifold) Fourier analysis for compact spectrographic representations; the second is a semi-supervised learning algorithm that leverages underlying manifold structure to improve phonetic classification performance with minimal labelled training examples. The second research track addresses the lacking scientific motivation for the frame-based acoustic models employed by nearly all modern speech recognizers. By associating each frame with a corresponding phonological state, the prevailing &ldquo;beads-on-a-string&rdquo; paradigm does not conform to current linguistic, neurobiological, and psychological theories that suggest the human perceptual mechanism integrates information on multiple context-dependent time scales and that the speech signal may be more appropriately modelled as an asynchronous collection of events in time. In attempt to reconcile the computational strategy, a novel recognition framework is presented that (i) transforms the speech signal into a collection of acoustic event landmarks, and (ii) decodes the underlying linguistic content according to a probabilistic model of the temporal dynamics of such events. This framework represents a fundamental shift from frame-based to point process-based speech modelling. A variety of point process models are explored, leading to competitive broad class recognition, phonetic recognition, and keyword spotting performance.