Patent Document ID: 9805255
Application ID: 15010351
Patent Status: 1

Claim One:
1. A multimodal action classification system, comprising: a video camera; at least one sensor device that comprises one or more of the following: a microphone, a motion sensor, a positional sensor, or a temperature sensor; a processor; a data store of classes and actions; and a computer-readable memory device containing programming instructions that are configured to, when executed, cause the processor to: receive, from the video camera, a sequence of digital images, wherein each of the digital images is associated with a time stamp, receive, from the at least one sensor device, other sensed data, wherein each item of other sensed data is associated with a time stamp, extract a plurality of video features from the sequence of digital images so that each extracted image feature is associated with a time period, extract a plurality of other features from the other sensed data so that each extracted other feature is associated with the time period, use a multi-layer long-short-term memory network to fuse a group of the extracted video features having time stamps corresponding to the time period and a group of the extracted other features having time stamps corresponding to the time period to create a fused feature representation, wherein the multi-layer long-short-term memory network comprises: a first set of layers, wherein each layer in the first set is configured to independently operate on a single data modality, and wherein the first set of layers is configured to generate a set of per-modality features; and a second set of layers configured to receive the per-modality features from the first set of layers and generate the fused feature representation as a set of fused features across data modalities, analyze the fused feature representation to identify a class that applies to both the extracted video features and the extracted other features, access the data store of classes and actions to identify an action that is associated with the class, and save the identified action to a computer readable memory device.