Patent Document ID: 5586215
Application ID: 07889619

Base Claim:
1. A speech recognition system for recognizing utterances belonging to a pre-established set of allowable candidate utterances using acoustic speech signals and selected concomitant dynamic visual facial feature motion between selected facial features associated with acoustic speech generation, comprising: a. an acoustic feature extraction apparatus for converting signals representative of acoustic speech into a corresponding acoustic feature vector set of signals; b. a dynamic visual feature extraction apparatus for converting signals representative of the selected concomitant dynamic visual facial feature motion associated with acoustic speech generation into a corresponding visual feature vector set of signals; and c. a time delay neural network classifying apparatus for generating a conditional probability distribution of the allowable candidate speech utterances by accepting and operating on a set of current and time delayed dynamic acoustic feature and visual feature vector sets respectively supplied by the acoustic and visual feature extraction apparatus.

---

Claim 10:
10. The speech recognition system of claim 1 wherein the time delay neural network classifying apparatus comprises: a. a tapped delay line input layer for accepting a sequence of acoustic and visual time-varying feature vectors and for providing a multiplicity of sequential acoustic and visual feature vectors in parallel at the delay line output taps: b. a hidden layer of neural cells coupled to the output taps of the tapped delay line for enhancement of time dependent features; c. a classification layer of neural cells coupled to the output of the hidden layer neural cells for generating a set of output time varying signals each representative of the probability of its corresponding utterance being present; and d. an averaging layer apparatus, connected to the output of the classification layer, for generating a set of time averaged varying outputs, one for each allowable utterance type, representative of a combined conditional probability that the associated utterance was spoken.