Emotion recognition and notification system

The present invention generally relates to an emotion recognition and notification system comprises an input unit includes a camera for capturing a facial image of a user's face; a pre-processing unit to remove noise and enhance the details of the captured facial image; a face detection unit to detect a face from the captured facial image; a features extraction unit to extract a set of image features from detected face from the captured facial image; a central processing unit equipped with a two channel facial expression recognition network based on CNN and LSTM to recognize an emotion of a user upon comparing the set of image features from a pre-stored image features; and a control unit equipped with a communication module to alert a registered user about a recognized emotion.

FIELD OF THE INVENTION

The present disclosure relates to computer vision processing technologies, and in particular, to an emotion recognition and notification system using computational techniques.

BACKGROUND OF THE INVENTION

Emotional recognition has drawn a lot of interest from academics both domestically and internationally as a significant research topic in the field of computer vision. Both expression recognition based on static photos and expression recognition based on video sequences are now used in the recognition of facial expressions. To extract static expression characteristics from a single facial expression image is the main goal of facial expression analysis based on static photos. The static and dynamic elements of the expression sequence are combined to analyse the emotional state of the persons in the video. These two techniques now have a high recognition rate and generalisation capacity for facial expression recognition. Therefore, to conduct face emotional identification, the current patent employs the single-frame static picture of the facial expression data set and the information aspects of the expression sequence.

The manual extraction of facial characteristics is a stage in the standard facial expression identification approach, but the features are only extracted once, the calculations are difficult, and the model's applicability is restricted. Recent years have seen a lot of interest in the scientific area of deep learning. Instead of the conventional method of manually selecting features, it combines multiple abstract data processing layers to create a computational model, allowing the machine to learn the features of data samples on its own while effectively avoiding the drawbacks of the conventional method of manually selecting features. Additionally, the deep learning approach uses a lot more data to learn the features than the conventional way of manually picking them, which might result in a fuller description of the feature information in the data. In summary, deep learning outperforms the conventional approach in terms of both accuracy and recognition time.

In the view of the forgoing discussion, it is clearly portrayed that there is a need to have an emotion recognition and notification system.

SUMMARY OF THE INVENTION

The present disclosure seeks to provide a facial features and expression-based emotion recognition and notification system using two channel facial expression recognition network.

In an embodiment, an emotion recognition and notification system is disclosed. The system includes an input unit includes a camera for capturing a facial image of a user's face. The system further includes a pre-processing unit to remove noise and enhance the details of the captured facial image. The system further includes a face detection unit to detect a face from the captured facial image. The system further includes a features extraction unit to extract a set of image features from detected face from the captured facial image. The system further includes a central processing unit equipped with a two channel facial expression recognition network based on CNN (Convolutional Neural Network) and LSTM (Long and Short Term Memory Network) to recognize an emotion of a user upon comparing the set of image features from a pre-stored image features, wherein the central processing unit executes a process for acquiring the recognition value corresponding to an expression of the face on the basis of differences between relative positions of pre-stored image features extracted from the face using an image previously received from the input unit at a timing when the input unit inputs an image, and relative positions derived in advance as references for the pre-stored image features extracted from one or more of the faces. The system further includes a control unit equipped with a communication module to alert a registered user about a recognized emotion.

In another embodiment, the set of image features are selected from a local feature including a pattern or distinct features including a point, edge, or small image patch, wherein the pre-stored image features are selected from a group of local feature including a pattern or distinct features including a point, edge, or small image patch configured with a group of expression classifications.

In another embodiment, a pre-stored image features is obtained upon pre-treating a plurality of facial expression sequence in a dataset, which includes tracking a face area of each of the plurality of images in the plurality of facial expression sequence through face detection tracking and cutting out a human face in the image, thereby generating the pretreated facial expression sequence; and processing the plurality of facial expression sequence into two facial expression sequences.

In another embodiment, the two-channel facial expression recognition network comprises a first channel and a second channel facial expression, wherein a last frame of the facial expression sequence with a first resolution is feeded into the first channel, and the facial expression sequence with the second resolution is feeded into the second channel, wherein either of the first resolution includes low resolution and the second resolution includes high resolution.

In another embodiment, the system comprises a training unit to train the two-channel facial expression recognition network based on CNN and LSTM using facial expression sequences with the two resolutions in the training set and the verification set into the first channel and the second channel.

In another embodiment, the emotions are selected a group of bored, excited, frantic, relaxed, surprise, fear, happiness, anger, neutral, sadness, disgust and the like.

In another embodiment, the set of image features detected from the user's face extracted from the captured image includes one or more of the skin texture appearance changes comprises one or more of the expression wrinkles, furrows, bulges, or illumination variations or a blushing, or multiple directional illumination conditions and multiple pose variations.

In another embodiment, the alert about a recognized emotion notifies the registered user including family members and hospital to help or assist the person of the recognized emotion.

An object of the present disclosure is to detect emotion of a person using its facial features and expression.

Another object of the present disclosure is to develop a two-channel facial expression recognition network for most appropriate facial emotion recognition.

Yet another object of the present invention is to deliver an expeditious and cost-effective emotion recognition and notification system to help and assist the person in need.

To further clarify advantages and features of the present disclosure, a more particular description of the invention will be rendered by reference to specific embodiments thereof, which is illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail with the accompanying drawings.

DETAILED DESCRIPTION

It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be restrictive thereof.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. The system, methods, and examples provided herein are illustrative only and not intended to be limiting.

Referring toFIG.1, a block diagram of an emotion recognition and notification system is illustrated in accordance with an embodiment of the present disclosure. The system100includes an input unit104includes a camera102for capturing a facial image of a user's face, wherein a mobile camera may be selected for capturing a facial image, wherein image can be fed uploading an image on a user interface to recognize image of any person.

In an embodiment, a pre-processing unit106is connected to the input unit104to remove noise and enhance the details of the captured facial image.

In an embodiment, a face detection unit108is connected to the pre-processing unit106to detect a face from the captured facial image.

In an embodiment, a features extraction unit110is connected to the face detection unit108to extract a set of image features from detected face from the captured facial image.

In an embodiment, a central processing unit112is equipped with a two channel facial expression recognition network based on CNN (Convolutional Neural Network) and LSTM (Long and Short Term Memory Network) to recognize an emotion of a user upon comparing the set of image features from a pre-stored image features, wherein the central processing unit112executes a process for acquiring the recognition value corresponding to an expression of the face on the basis of differences between relative positions of pre-stored image features extracted from the face using an image previously received from the input unit104at a timing when the input unit104inputs an image, and relative positions derived in advance as references for the pre-stored image features extracted from one or more of the faces.

In an embodiment, a control unit114is equipped with a communication module116and connected to the central processing unit112to alert a registered user about a recognized emotion. The alert is received on a registered user computing device through the communication module116.

In another embodiment, the set of image features are selected from a local feature including a pattern or distinct features including a point, edge, or small image patch, wherein the pre-stored image features are selected from a group of local feature including a pattern or distinct features including a point, edge, or small image patch configured with a group of expression classifications.

In another embodiment, a pre-stored image features is obtained upon pre-treating a plurality of facial expression sequence in a dataset, which includes tracking a face area of each of the plurality of images in the plurality of facial expression sequence through face detection tracking and cutting out a human face in the image, thereby generating the pretreated facial expression sequence; and processing the plurality of facial expression sequence into two facial expression sequences. The dataset is stored in a cloud server118.

In another embodiment, the two-channel facial expression recognition network comprises a first channel and a second channel facial expression, wherein a last frame of the facial expression sequence with a first resolution is feeded into the first channel, and the facial expression sequence with the second resolution is feeded into the second channel, wherein either of the first resolution includes low resolution and the second resolution includes high resolution.

In another embodiment, a training unit is connected to the central processing unit112to train the two-channel facial expression recognition network based on CNN and LSTM using facial expression sequences with the two resolutions in the training set and the verification set into the first channel and the second channel.

In another embodiment, the emotions are selected a group of bored, excited, frantic, relaxed, surprise, fear, happiness, anger, neutral, sadness, disgust and the like.

In another embodiment, the set of image features detected from the user's face extracted from the captured image includes one or more of the skin texture appearance changes comprises one or more of the expression wrinkles, furrows, bulges, or illumination variations or a blushing, or multiple directional illumination conditions and multiple pose variations.

In another embodiment, the alert about a recognized emotion notifies the registered user including family members and hospital to help or assist the person of the recognized emotion.