Patent ID: 8442820

Claim:
A combined lip reading and voice recognition multimodal interface system, comprising: a voice recognition module, executed by an audio signal processor, recognizing an instruction through performing voice recognition; and a lip reading module, executed by a video signal processor, performing lip reading recognition and providing an image, wherein the lip reading module comprising: a lip detector that detects lip features using the input image from a lip video image input unit; a lip model generator that generates a shape model and an appearance model using an active appearance model (AAM) lip model; a lip tracker that tracks lip feature points obtained as a result of the AAM fitting after lip detection using the shape model generated by the lip model generator and a Lucas-Kanade (LK) algorithm; a speech segment detector that inputs frame data of a predetermined period into a neural net recognizer to determine whether the segment is a speech segment or a silence segment based on a series of lip model parameters obtained as the result of lip tracking on consecutive input images; a system mode determiner that determines whether the system is in a learning mode in which the label of lip feature data is known or in a recognition mode in which the label thereof is not known; a lip reading recognition learning unit that learns a k-nearest neighbor (K-NN) learner using feature data and an input label if the system is in the learning mode; an instruction recognition unit that finds a learning pattern most similar to the feature data through the learned K-NN recognizer and outputs a result instruction as a feature value if the system is in the recognition mode; and a lip feature database that stores patterns for each instruction that are learned offline or online.