In the past eight years our speech recognition technology has moved from 1,000 word to a practically unlimited vocabulary. Simple Markovian language models, invented in the '70s, are clearly suboptimal for this task, yet are surprisingly hard to beat. We are working on models that consider information beyond the immediate past and attempt to exploit the deeper structure of language. However data from that domain is not always available in large enough quantities. One of the biggest challenges of language modeling is in rapid adaptation to new domains, new styles, and new topics.
Spontaneous speech is very different from read, rehearsed or otherwise structured speech.
Spontaneous speech is highly agrammatical, rich with disfluencies, and has a distinct word usage profile. Conventional parsing techniques, already at the breaking point with 'proper' language, are next to useless when faced with these variations. We are working on methods to exploit the deep structure of language without assuming grammaticality or fluency. One of our goals as a research university is to make technology available and accessible, especially to industry and to other academic institutions. As of June 1997, the toolkit has been in use by over 40 academic and industrial research groups in 14 countries.