Patent ID: 11960576
Assignee: INCEPTION INSTITUTE OF ARTIFICIAL INTELLIGENCE LIMITED
Field: Computer technology (Electrical engineering)
Classification: CPC G | IPC G

Claim 0:
1. A method for identifying an activity occurring in media content captured in different possible lighting conditions including low light conditions, comprising:
receiving the media content comprising video content and audio content;
extracting video features from at least a portion of the video content using a video model;
extracting audio features from at least a portion of the audio content using an audio model;
generating a darkness-aware feature by applying the video features to a darkness-aware evaluation model, the darkness-aware feature providing an indication of the ability to extract discriminative information from the video content;
modulating the video features and the audio features using the darkness-aware feature to dynamically adjust an amount of noise filtered out according to illumination of the media content; and
predicting an activity occurring in the media content using the modulated video and audio features by:
generating an audio prediction by applying the modulated audio features to an audio classifier;
generating a video prediction by applying the modulated video features to a video classifier; and
combining the audio prediction and the video prediction to predict the activity occurring in the media content,

wherein the darkness-aware feature is generated according to:

fd=Φ′d(Fv),

where:
fd is the darkness-aware feature;
Φ′d denotes the darkness-aware evaluation model obtained by removing a fully connected layer from Φd;
Φd comprises a 3D convolutional network followed by the fully connected layer, Φd trained as a binary classification problem with the fully connected layer outputting a single score D∈ indicating whether visual content can provide discriminative features;
Fv is the extracted video features.