Patent ID: 8103502

Claim:
A method of recognizing an utterance comprising: recognizing a first mode in a plurality of modes at a multimodal meaning recognition system comprising a processor; outputting a recognition result for the first mode in the plurality of modes; generating a recognition model based on the recognition result outputted for the first mode in the plurality of modes, the generating a recognition model comprising: receiving a gesture recognition lattice and a first finite-state transducer; relating a first portion of the utterance comprising the first mode to a second portion of the utterance comprising a second mode with the first finite-state transducer; and generating a second finite-state transducer, wherein the second finite-state transducer comprises a gesture and speech recognition model finite-state transducer based on the gesture recognition lattice and the first finite-state transducer; and recognizing the second mode in the plurality of modes using the recognition model for the first mode in the plurality of modes, wherein the recognizing a first mode in a plurality of modes comprises recognizing a gesture mode and the recognizing the second mode in the plurality of modes comprises recognizing a speech mode.