Patent ID: 7538711

Claim:
An integrated remote control device receiving multi-modal inputs, the device comprising: an input module that receives inputs of at least two modalities from a user; a first modality recognition module that recognizes and interprets an input of a first modality received from the input module; a second modality recognition module that recognizes and interprets an input of a second modality received from the input module; a command database (DB) that stores commands and correlation information between controlled devices, wherein the correlation information includes grouping information of controlled devices on a home appliances network according to the manner in which the user operates the devices; a control module that interprets a user command by synthesizing information about the inputs interpreted by the first and second modality recognition modules, and extracts a control command from the command DB, the control command being for remotely controlling a plurality of devices and corresponding to the interpreted user command; an output module that outputs the control command extracted by the control module; and a state monitoring module that checks the state of the controlled device, monitors the state of the controlled device upon execution of the control command, and reflects any changes in the state of the controlled device in a control history DB, wherein, when the inputs interpreted by the first and second modality recognition modules are insufficient to interpret a particular control command due to error, the control module also interprets the particular control command using the state information from the control history and using the correlation information that indicates a relationship of the controlled device to other controlled devices.