A Multimodal Approach to Performing Emotion Recognition
By: Nithanth Ram
Tags: Audio Processing, Image Processing, Emotion Recognition, Multimodal Learning, Random Forest Classifiers

Introduction

Recently, work in the field of Human-Computer Interaction (HCI) has been bridging the abstract gap between humans and computing systems more and more. Significant advances in hardware and software systems have made the existence of specific applications possible. With the advent of powerful new technologies like advances in deep learning and flexible, integratable sensors, it’s more possible to create robust HCI applications. Emotional recognition is an HCI domain that carries many potential benefits upon exploration. With areas like mental health, productivity, and communication being prioritized today as aspects of human behavior, an emotion recognition system suddenly becomes very pertinent.

Multimodal sensing is a machine learning technique that allows for the expansion of sensor-driven systems. It combines or “fuses” sensors in order to leverage multiple streams of data to extract more rich, contextual sentiments regarding an event. The goal of this project is to create a multimodal emotion recognition system utilizing audio and image processing as well as early and late fusion. By creating a multimodal emotion classifier, I hypothesize that a more powerful emotion recognition system can be built and utilized for a host of practical applications.

Prior Work

A plethora of prior work has been conducted in this area. By observing the work of others, I gained insight into design choices for my system as well as ways to distinguish my project. Tzikaris et. al (2017) specifically used convolutional neural networks and LSTM networks in order to perform feature extraction, outlier filtering, and both audio and image recognition to build an emotion recognition system. By looking at their work, I was able to see how powerful a multimodal system of both audio and image processing could be. However, a deep learning solution seems computationally expensive as well as consistently used across different studies. I’ll look to deviate from a deep learning solution to see how other classification algorithms perform. Dzedzickis et. al (2020) conveyed an exhaustive overview of sensors and methods that apply to human emotion recognition, teaching me that there exist multiple internal channels of information for emotional recognition i.e. electrocardiography (heart) and electroencephalography (brain). Finally, Venkataraman et. al (2019) explore the critical nature of MFCCs, pitch, and energy as critical features for audio recognition. I’ll look to explore these features in my system.

Dataset: RAVDESS

For this project, the Ryerson Audio-Visual Database of Emotional Speech and Song (RAVDESS) was used for training and testing. The data samples are small 3–4 second sentences of two variations: “Kids are talking by the door” and “Dogs are sitting by the door”. The dataset consists of 24 actors (male and female), two modalities (audio, video), two vocal channels (speech, song) and eight emotions: neutral, calm, happy, sad, angry, fearful, disgust, surprised. Instead of using the entire dataset, I took a subset of 10 actors and only used the speech vocal channel to decrease the load of the dataset.

RAVDESS Training Data Analysis

Looking at the subset of data used for training, both the gender distribution and emotion distribution are basically uniform (with the exception of the ‘neutral’ samples).

Feature Extraction

Audio

Feature extraction for the audio portion was performed by segmenting and framing audio signals using Librosa. Specifically, frames of length 0.1 seconds with 50% overlap were extracted as samples. Regarding the actual features calculated, from the Mel spectrogram, the first 20 Mel-Frequency Cepstrum Coefficients (MFCC) were extracted along with the mean and standard deviation of the zero crossing rate (ZCR), signal root mean squared value (RMS) and spectral rolloff. These features were thought to be intuitive to use for this task. The MFCCs (aside from being a widely used feature for most audio recognition problems) are able to explicitly detail the nuanced structure of the envelope of the time power spectrum. A human auditory response is well approximated by the MFCCs; specifically, unique phonemes are accurately distinguished. This makes the MFCCs a solid choice for a audio feature.

Librosa audio signals and Mel-frequency spectrogram (On the right is a male actor and on the left is a female actress. Both are speaking the same sentence with an angry emotion).

The ZCR, RMS, and spectral rolloff of the signal are also good features because they deal with providing context to the articulation and intensity of audio signals: two attributes that are sure to distinguish emotions from each other. The chroma vector seemed intuitive to use initially, as it makes up pitch profiles with audio signals, but it degraded performance during testing. Could this be because of pitch-gender mapping issues?

Image

Image Feature Extraction

Regarding image feature extraction, the video modality of each data sample was passed into a frame extractor (using OpenCV). The front and tail-end of the sequence of frames were discarded to allow the actors time to visually emote the specific emotion. Initially, these frames were thresholded for de-noising, but this technique got rid of the facial features that would be crucial for image-based emotion classification. Instead, images were gray-scaled and edged, using a canny-edge algorithm from Scikit-Image. They were also reshaped to a smaller scale factor to get a smaller amount of feature pixels.

Individual Models

Before constructing the multimodal classifier, individual models needed to be trained and analyzed to see how each modality independently solves this emotion recognition task. Three models were trained for each modality, with a train-test-split of 0.8 and 0.2. 10-fold cross validation was performed on these models to prevent any overfitting. The three classifiers used were the Random Forest classifier, the Naive Bayes classifier, and the Multilayer Perceptron classifier (all courtesy of Scikit-Learn). A deep learning approach could have been utilized. The power of CNNs, with regards to image and audio classification as well as multimodal channel layers, makes them a very logical choice. However, consistently in the prior work that was examined, deep learning approaches were taken. To embark on a different route, I utilized these less computationally intensive classifiers to see what the effect would be on emotion recognition. From the prior work that was researched, no studies really capitalize on these algorithms for multimodal emotion recognition. They instead opt to use the deep learning techniques instead. The three models for each modality had their hyper-parameters tuned with GridSearchCV.

Individual Model Performance

As seen above, the Random Forest classifier performed the best for both modalities, with accuracy and F1-scores for audio and image of 0.71, 0.68 and 0.62, 0.55 respectively. Here, we can also see the audio models outperform the image models for all three of the classifiers. The audio Random Forest classifier’s 0.71 accuracy score and 0.68 F1-score doesn’t really represent the greatest performance of a model, but it is a good baseline to work from. Now, maybe the sensor fusion techniques can improve the model performances.

Sensor Fusion

Early Fusion

Early Fusion Diagram

One option of augmenting our individual models into a multimodal system is to use early fusion. Early fusion entails assembling one large feature matrix, where the data from both modalities is combined to represent one comprehensive data sample. In this case, the image and audio features were matched up to each other, with the frame of the video pairing with whichever signal segment it corresponds to with regards to time. There are two issues I experienced with this early fusion technique. First, the feature matrix becomes incredibly dense and increases the training load and time. Second, and more importantly, with early fusion, the data across different modalities needs to be synchronized. This was difficult to do, since the audio was sampled at 22050 Hz and the video was sampled at 30 FPS (Hz). The mismatch of sampling rates causes data alignment issues, and this was especially evident in the performance of the early fusion model. The same classification models were trained on this new, augmented feature matrix (with 10-fold CV), and resulted in a sharp degradation in model performance.

Early Fusion Model Performance

MLP Early Fusion Confusion Matrix

Here we can see how the optimal model, the Random Forest classifier, has decreased to an accuracy score of 0.51 and an F1-score of 0.48. The early fusion models are riddled with inaccurate predictions as demonstrated especially with the very poor performance of the MLP classifier. Looking at its confusion matrix, we can see how the predictions are wildly inaccurate. It seems like early fusion is not the answer for building the multimodal emotion recognition system.

Late Fusion

Late fusion is a different multimodal technique that utilizes the work of individual models in tandem to produce a singular output. The base input models can either pass their outputs to another layer containing a third model, or they can utilize some sort of democratic system to make the prediction together themselves.

Late Fusion Diagram

The latter is the approach taken to building a late fusion model. The Random Forest classifiers from the individual training phase were selected (due to their optimal performance in both modalities), and they were jointly attached in a late fusion model. The late fusion model works as such. Entire video and audio signals from the same sample are passed to the respective input models. These models segment/frame each signal and keep a running frequency count of their predictions for each analyzed frame. The common emotion frequencies across each model are summed in the output layer, and the emotion with the highest total is the predicted label that is outputted. Since the audio Random Forest model performed better than the image one, the audio model’s predictions each have a weighted coefficient of 1.15 (the ratio of accuracy scores from audio model to video model). The equation below shows how the predictions over one data sample were aggregated to vote and decide on one final output prediction.

Late Fusion Model Prediction Equation

Random Forest Late Fusion Confusion Matrix

Late Fusion Random Forest Model Performance

Regarding the model’s performance, we see the late fusion Random Forest model improve upon the baseline accuracy and F1-score. The late fusion Random Forest model achieves 0.79 accuracy with a 0.76 F1-score, indicating improvement due to the late fusion technique. Looking at the confusion matrix, we see how it is much cleaner, with predictions far more accurate this time around. Each class of emotions also seems to have relatively high (or at least improved) precision, recall, and F1-score.

Late Fusion Random Forest Model Class Metrics

Out of all the models built, trained, and tested, the late fusion Random Forest classifier is the optimal one, showing how a multimodal approach can improve upon existing recognition models if the right technique is chosen. In this case, it happened to be late fusion.

In-the-wild Model Performance

The late fusion Random Forest model works well with the RAVDESS dataset, but how well does it generalize to real-time collected samples? I wanted to test out this model by collecting in-the-wild samples, but due to COVID-19, I was restricted to operating within my household. With four members in my household, including myself, I collected three samples of data for each emotion to observe how well newly collected data is processed by the model. The model accuracy degrades down to 0.66 accuracy and 0.56 F1-score, meaning that more diverse data needs to be used for training. Generalizability is one of the top priorities of recognition systems, and in this case, more varied samples need to be fed to the model to maintain/improve the accuracy score of 0.79. Another concept I wanted to observe was the cross-cultural effects of language on emotion recognition. Throughout the multimodal development process, I was concentrated specifically on the computing component of the HCI process, but I neglected the human aspect of these systems. Specifically, I wanted to see if speaking a different language had a noticeable effect on the model’s ability to classify emotions. For the emotions “happy”, “sad”, “angry”, and “neutral”, five samples were taken across the same five participants. This time however, they spoke the sentence samples in Tamil (the language spoken in my house). The model’s accuracy and F1 scores decreased to 0.65 and 0.55 respectively.

In-the-wild Model Performance of Late Fusion Random Forest Model

The reason for this degradation may be attributed to cross-cultural variances in communication. More specifically, different languages may have different vocal signals, pitch inflections, facial expressions, etc. when they are spoken and emoted. As a result, the model may not recognize these differences because it was trained purely in English. Not enough samples were collected to make a definitive statement, but analyzing the impact of different languages on this late fusion Random Forest model definitely piqued my interest. This observation just serves to demonstrate how the human aspect of HCI applications should be just as heavily considered and analyzed as the computational side.

Key Takeaways

Personally, I learned many different lessons throughout the course of this project. I got to experience the rigorous nature of feature selection and extraction. I learned that there is no umbrella solution that perfectly applies to feature extraction. A strong set of features for one model may not work for another, justifying the no free lunch theorem that is found widely throughout statistical inference and machine learning theory. However, I did get to observe the power of MFCCs and recognize how they perform well in a variety of audio-based recognition systems. I also got to experience the burden of early fusion firsthand. Working with dense feature vectors and difficult data synchronization issues really demonstrated to me the fundamental issues of an early fusion approach. I find it hard to see early fusion being utilized with n-modal systems that grow beyond the order of two sensors. Even though early fusion was not successful in this instance, I still got to observe the power of leveraging a multimodal approach for systems. I got to see how the late fusion Random Forest model was able to better aggregate its predictions by using two modalities of data. It’s intuitive that data from a different perspective can only add to, not damage, the likely classification of an event. With the case of the late fusion technique in this project, it’s critical that the proper method is selected otherwise multimodal approaches can severely degrade the accuracy of a model. Finally, I was able to observe the cross-cultural impact of language on the model’s performance. Different languages and cultures have a different style of communication that may not be recognized by a system trained on a completely English corpus of data. These varied linguistic data samples need to be included in the training phase so that the model can extend across different cultures.

Conclusion

The late fusion Random Forest model was determined to be the optimal model, with an accuracy and F1-score of 0.79 and 0.76 respectively. This shows how a multimodal approach can to capitalize off of fusion techniques and improve a single-modality model. The household-collected data elicits an accuracy and F1-score of 0.68 and 0.62 respectively, demonstrating that extension of the input data and more rigorous training needs to be performed to make the model more generalizable. The use of a different language, Tamil, resulted in the late fusion Random Forest model to decline in accuracy to 0.65, showing how cultural and linguistic differences in human communication and emotion exist. This thread should be explored further in the future to see if a culture-comprehensive emotion recognition system can be developed.

Future Work and Applications

Future work on this project involves utilizing a deep learning approach to see if the accuracy of the system can be improved. Convolutional neural networks seem to be the gold standard currently when it comes to image and audio recognition. They also have the capability to create multimodal systems inherently with the ability to have multiple input channels in their architecture. Specifically, existing architectures such as VGG16 and VGG19 can be explored to see if model performance increases. Another extension of this project I would like to explore is making the model more generalizable to not just different languages but different disabilities. I would like to observe a model that can recognize emotions but for people with disabilities such as down syndrome, facial palsy, speech impediments, etc. Often times, the disabled population is overlooked in the training phase of systems, introducing an inherent bias in the core models of these systems. It is critical to train these systems to extend to people with disabilities, since their inclusion deserves the same degree of accuracy as the training data permits. Finally, I would like to actually build out this system and deploy it in reality. Conceptually, I have a rudimentary system mapped out, and it would be interesting to see how the late fusion Random Forest model performs from an application perspective.

Conceptual Diagram of Emotion Recognition System

Future applications of this work are also vast. An emotion recognition system can be used for virtual counseling and emotional monitoring. Given the recent rise in mental health awareness and the potential decline in mental health due to this pandemic, a virtual emotion analyzing and monitoring system would incredibly useful in tracking the mental health of users. An emotion recognition system could also be used to track early child development, in order to let users know what changes to make in order to foster the healthy upbringing of a child. Finally, the visually-impaired could also utilize an emotion recognition system to better communicate with others. A system consisting of a camera and headphones could let those that are visually-impaired better know the emotional state of the individual they are interacting with. Overall, I enjoyed working on this project a lot. I hope to continue this work in the future, pursuing some of the aforementioned extensions.

References

https://zenodo.org/record/1188976#.X73KC6pKg1J (RAVDESS Dataset) Morency, Baltrusaitis, L. P. M. T. B. (2016, January 1). Tutorial on Multimodal Machine Learning [Slides]. Carnegie Mellon University. https://www.cs.cmu.edu/~morency/MMML-Tutorial-ACL2017.pdf Kannan Venkataramanan, & Haresh Rengaraj Rajamohan. (2019). Emotion Recognition from Speech. P. Tzirakis, G. Trigeorgis, M. A. Nicolaou, B. W. Schuller and S. Zafeiriou, “End-to-End Multimodal Emotion Recognition Using Deep Neural Networks,” in IEEE Journal of Selected Topics in Signal Processing, vol. 11, no. 8, pp. 1301–1309, Dec. 2017, doi: 10.1109/JSTSP.2017.2764438. Dzedzickis, A., Kaklauskas, A., & Bucinskas, V. (2020). Human Emotion Recognition: Review of Sensors and Methods. Sensors (Basel, Switzerland), 20(3), 592. https://doi.org/10.3390/s20030592

Packages used: Librosa, OpenCV, Scikit-Learn, Scikit-Image, Matplotlib, Numpy, Pandas, Seaborn