Consideration of Lombard effect for speechreading

0 We propose a method for integrating audio and visual information to enhance speech recognition in adverse environments. We train the audio hidden Markov model and the visual hidden Markov model separately, and then use a Viterbi algorithm to decode both channels in parallel. The decoding process is asynchronous between the two channels to capture the asynchronous nature of audio and visual speech. We test the proposed method using speech corrupted by various types of noises and speech with the Lombard effect. INTRODUCTION The performance of existing automatic speech recognition (ASR) systems typically degrades significantly when there is background noise or if the speaking style varies. One type of speaking style variations, referred to as the Lombard effect, arises when the speaker is in a noisy envinmmmt [I]. Speech recognition under the Lombard effect has been studied in [2,3]. One solution is to use visual information to enhance speech recognition, comnonly known as lipreading or speechreading [4]. hi this paper, we present our effort in speechreading based on joint audio-visual hidden Markov models @MM) with asynchronous Viterbi decoding. We use 13-dimensional Me1 frequency cepstral coefficients (MFCC) as the audio features. Others robust features such as relativespectra perceptual linear prediction (RASTA-PLP) have been proposed [5] but they often assunie stationary noises and therefore are not suitable for non-stationaq noises. For visual features, we perform lip tracking [6] to extract the mouth width, the heights of the upper and lower lips. TheE are several ways to combine the audio and visual information. In early integration, basically feature integration, the audio feature vector and the visual feature vector are cascaded to form a joint feature vector, with certain weighting to train the joint HMMs and to be used in Viterbi decoding. Late integration, basically classifier integration, takes the weighted sum of the scores fiom the audio classifer and the visual classifier as the final score for recognition. MIDDLE INTEGRATION WITH VlTERBI DECODING In late integration, each of audio and visual decoding finds the optimal path in its own channel. Since speech production is bimodal, we can do better by searching the optinial path in the joint audio-visual state space. To achieve this, we explore middle integration. During training, we train the audio HMM and the visual HMM 0-7803-7025-2/0 1/$10.00 02001 IEEE. 613 separately. During recognition, we extend the Viterbi algorithm to decode the two cliannels in parallel. Synchronous Decoding A simple way to decode the audio and visual chamels in parallel is to force the aigdio and visual sequences to jump from state to state synchronously. The observation probability & C O , ) and transition probability qJ of the joint HMM can bl: expressed as: p, (0, ) = (p" J(o,))" . (p", (o,))'-" 1 5 I 5 N qJ = ( C X " J J ) ' " . ( C X " ~ J ) ' " 1 l l l N 1 < j < N ahere pJ(0,) and ano are the observation probability and transition probability of the audio HMM, p " , ( ~ , ) and a', are the observation probability and transition probability of the visual HMM, andiv is the weighting factor between 0 and 1. One stample of this approach can been seen in [7]. Asynchronous Decoding The audio and visual channels may not be synchronous in terms of state transitions. That is, using the Viterbi algorithm, the result of the audio channel state transition may be different ftom that of the visual channel state transition. To examine the synchrony of audio and visual sequences, we use Viterbi decoding to find the optimal paths of the observed audio sequence and visual sequence scparately and obtain Table 1 that shows the joint probability of audio states and vi sua1 states. Table 1 : Joint probability of audio and visual states If the audio and visual sequences were perfectly synchronous, we would expect all the off-diagonal elements in Table 1 to be zero. In fact, most of the off-diagonal elements are non-zero, which indicates that the audio and visual sequences are often asynchronous. Therefore, we propose to use two-stream Viterbi decoding to search for the optimal path in the Cartesian product of the joint audio-visual state space. This is equivalent to decoding in a virtual 25-state HMM, as shown in Figure 1. This HMM is OvirtualO because its transition and observation probabilities are obtained by simply combining parameters fiom the separately trained audio and visual HMMs, rather than estimate them directly from the training data.