Patent Document ID: 20170220854
Application ID: 15010351
Patent Status: 0

Claim One:
1. A multimodal action classification system, comprising: a video camera; at least one sensor device that comprises one or more of the following: a microphone, a motion sensor, a positional sensor, or a temperature sensor; a processing device; a data store of classes and actions; and a computer-readable memory device containing programming instructions that are configured to, when executed, cause the processing device to: receive, from the video camera, a sequence of digital images, wherein each of the digital images is associated with a time stamp, receive, from the at least one sensor device, other sensed data, wherein each item of other sensed data is associated with a time stamp, extract a plurality of video features from the sequence of digital images so that each extracted image feature is associated with a time period, extract a plurality of other features from the other sensed data so that each extracted other feature is associated with the time period, fuse a group of the extracted video features having time stamps corresponding to the time period and a group of the extracted other features having time stamps corresponding to the time period to create a fused feature representation, analyze the fused feature representation to identify a class that applies to both the extracted video features and the extracted other features, access the data store of classes and actions to identify an action that is associated with the class, and save the identified action to a computer readable memory device.