Successful audio-visual speech recognition should combine audio and visual speech so the accuracy is never lower than highest of either modality. However, there are other ways to improve accuracy in noise - such as building noise matched models that have be trained under noisy conditions or by using noise masking techniques such as spectral subtraction. We have investigated the use of all of these using the results of the MSA visual speech recogniser on the AVLetters database. For late integration we must determining the value of the weight alpha applied to the audio and visual vectors. This is the best possible result obtained by searching the full range of values for alpha.
Noise compensation Compare spectral subtraction and matched models results.
Using matched models gives better performance than audio-visual using uncompensated audio. Noise compensated audio-visual Visual information is added to the compensated audio results. Visual information is still useful even when noise compensation is used. Summary These results show that using visual speech information is beneficial even if one uses acoustic noise compensation techniques. The main problems with MSA are that it is not scale invariant and cannot track the mouth.