A very large corpus of concurrent Mandarin and English textual and spoken news stories is available for conducting such research. Given a query in either language, we will then investigate systems that retrieve relevant documents in both languages for the user. Several schemes for recognizing the audio, indexing the text, and for estimating translation models to match queries in one language with documents in another language will be investigated in the summer. Applications of this research include audio and video browsing, spoken document retrieval, automated routing of information, and automatically alerting the user when special events occur. A dramatic example is the so-called McGurk effect, where a spoken sound ga is superimposed on the video of a person uttering ba.
Most people perceive the speaker as uttering the sound da.
We will strive to achieve automatic lip-reading by computers, i.e., to make computers recognize human speech even better than is now possible from the audio input alone, by using the video of the speaker's face. When people speak casually in daily life, they are not consistent in their pronunciation. In listening to such casual speech, it is quite common to find many different pronunciations of individual words. There are many reasons for this. In casual speech, one phoneme can shift to another.