Integration of Driver Behavior into Emotion Recognition Systems: A Preliminary
Study on Steering Wheel & Vehicle Acceleration
Sina Shafaei, Tahir Hacizade and Alois Knoll
Technical University of Munich
Department of Informatics

sina.shafaei@tum.de†First International Workshop on Advanced Machine Vision for

Real-life and Industrially Relevant Applications
Perth, Dec. 3nd 2018

Outline#
Introduction

Objectives

Experiments

Results

Conclusion & Future
Works

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications2 Introduction
Status QuoI Current status of the emotion recognition systems in cars is mostly focused on facial-based approaches
I Modeling behavior of the driver in cabin has great impact on developing intelligent and autonomous driving
I According to 7-38-55 rule, 93% of human communication is performed through nonverbal means, which consists of facial
expressions, body language and voice tone.

I Recent studies have shown a high level of correlation between driving behavior and emotional status

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications3 Introduction
Main objectives of this work:

I How emotions affect the behavior of the driver?
I How to map the driving behavior, to the current emotional status?
I What are the benefits of multimodality in emotion recognition systems?

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

4

# Objectives
Proposed system in this work:

I Facial approach is based on HOG (Histogram of Oriented Gradients) descriptors and SVM
I Acceleration/Deceleration and Steering Wheel usage are considered as the driving behavior-related modalitiesI Decission-level fusion is used for combining the modalities (arousal-valence measure)
Facial Expressions

FacialModality
Fusion ofModalities

Steering Wheel
Data Input
StreamEmotion Classiﬁcation
Behavior
Modality

Acceleration

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

5

# Objectives - Facial Approach
I Facial landmark detector was used to detect ROI,I After detecting the face, HOG descriptors were used byapplying a fixed size sliding window over an image pyramidbuild upon them,

I Model was build on a liner kernel SVM with decision functionof One-Vs.-Rest,

I For training the model k-fold cross validation was used with k
set to 10,

I CK+ and JAFFE databases were used for training,featureVector ← init list
2: SVMClassifier ← load model
3: while newFrame is exist do
4:frame ← FetchVideoStream()
5:
grayFrame ← GrayscaleImage(frame)6:
if faceTracker (grayFrame).Score < threshold then7:
face ← detectFace(grayFrame)
8:
else
9:
face ← faceTracker (grayFrame).Position
10:
end if11:
ROIarray ← FetchROI (face)12:
for each ROI in ROIarray do
13:hog ← HOGDescriptor (ROI )14:
featureVector ← featureVector + hog
15:
end for16:
result ← SVMClassifier (featureVector )17: end while
1:

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

6

 Experiments
Testbed:I A real “SMART” car simulator, 15 participants, each driving 3
scenarios , 36 minutes on average for all of the scenarios,

I Their attitude was different toward scenarios due to theprehistory condition narrated to them and predefined situations
on the road,

I Participants were asked right after each ride about theiractual emotional status using a questionnaire,

I Facial expressions are recorded through a camera alongwith the respective signals of steering wheel and acceleration
through the virtual test drive software of the simulator
Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

7

 Results
I Frequency distribution of vehicle acceleration in 4 groups of emotional status
High Arousal(Awake)

Angry

Category II

Suprised

Stressed
Confused

Category IHappyLow Valence(Negative Emotions)High Valence(Positive Emotions)
NeutralSad

Category III

Relaxedm

Tired

Calm

Category IV

Low Arousal(Tired)

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

8 Results
I Angry/stressed/confused drivers tend to accelerate/decelerate fasterHigh Arousal(Awake)Angry

Category II

Suprised

Stressed
Confused

Category I

Happy

Low Valence(Negative Emotions)High Valence(Positive Emotions)
NeutralSad

Category III

Relaxed

m

Tired

Calm

Category IV

Low Arousal(Tired)

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

9 Results
I 50 samples of collected data are considered,I Decision tree of combining the 3 different module using 50samples,

I A considerable impact of the vehicle acceleration fromcategory II,

I All 18 samples of category II are grouped using only one
condition from vehicle acceleration,

I Only samples from category I and IV are left un-grouped,

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

10 Results
Conditions formulated by steering wheel (SW) rotation and happy, neutral and sadness features from the facial expressions
module, are considered together to form the feature vector,

I After analyzing a single decision tree, we use the samefeature vectors from 50 samples to train a random forestclassifier

ModuleVASW

Facial Expression

Vector index
123456789

Parameter Name
Acceleration
Steering Rotation
NeutralAnger
DisgustFearHappySadness
Surprise

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

Value0 or 10 to ∞0 to 10 to 1
0 to 1
0 to 10 to 1
0 to 1
0 to 111 Results
I The 77.27% of accuracy is obtained using multimodalAuthors

Facial MethodAccuracy

J.F.Cohn and T.Kanade et al.

Active Appearance Models

83%

H. Alshamsi et al.

BRIEF Feature Extractor

89%

W.Swinkels et al.

Ensemble of Regression Trees

89.7%

I This condition is prone to errors and false predictions since

Sébastien Ouellet

Convolutional Network

94.4%

in real-life situations the 2-minutes range could be easily

R.A.Khan et al.

HOG-based

95%

M. F. Donia et al.

HOG-based95%

Our Method

HOG on ROI regions

93%

emotion recognition system on data samples with 2 minutes oflength,

falsified by situations like staying behind a red light,

I To mitigate this issue, we consider the decision taking step at
the end of each ride by summarizing the emotion predictions
performed for only sub-samples and choosing the mostfrequently felt emotion

MethodAccuracy

Precision

F1 Score

RecallFacial-based Module

54.54%

54.75%

50.45%

49.86%

SW-based Module

37.5%

10.3%

13.6%

25%

VA-based Module68.18%

35.51%

37.76%

41.37%

Fusion of All Three Modules

77.27%

73.39%

73.59%

75.89%

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

12 Results
Comparison of different unimodal and multimodal emotion recognition systems based on accuracy and different number ofemotional classes

SystemType

MethodClasses

Accuracy[1]

UnimodalElectrodermal Activity (EDA)

370%

[2]

UnimodalFacial Emotion Recognition.

670.2%[3]

UnimodalSpeech Emotion Recognition

388.1%[4]

UnimodalSpeech Emotion Recognition.

280%[5]Multimodal

EDA and Skin Temperature

492.42%[6]

Multimodal

Speech & Facial Emotion Recognition

757%

[7]

Multimodal

Acoustic & Facial Emotion Recognition

390.7%

Our System

Multimodal

Facial and Vehicle Parameters494.4%

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

13

 Conclusion & Future WorkI Most of the studies on emotion recognition are based on unimodal approaches where only audio or visual is examined,I Adding behavior related modalities increases the accuracy of the predictions and robustness of the systems,
I We have only investigated the impacts of integrating two modalities of acceleration and steering wheel usage but too many areleft for the future,

I The proposed system was capable of classifying the emotions into 4 main categories with the final accuracy of 94.4%,
I We are going to extend our system by integration of other behavior-related modalities and will study the shared models among
the drivers according to their emotional states

Thank You.
Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications14 References
The State-Of-The-Art Unimodal/Multimodal Emotion Recognition Methods
J. S. K. Ooi, S. A. Ahmad, Y. Z. Chong, S. H. M. Ali, G. Ai, and H. Wagatsuma, “Driver emotion recognition framework based on electrodermal activity
measurements during simulated driving conditions,” in Biomedical Engineering and Sciences (IECBES), 2016 IEEE EMBS Conference on. IEEE,
2016, pp. 365–369.
R. Theagarajan, B. Bhanu, A. Cruz, B. Le, and A. Tambo, “Novel representation for driver emotion recognition in motor vehicle videos,” in ImageProcessing (ICIP), 2017 IEEE International Conference on. IEEE, 2017, pp. 810–814.
A. Tawari and M. Trivedi, “Speech based emotion classification framework for driver assistance system,” in Intelligent Vehicles Symposium (IV), 2010
IEEE. IEEE, 2010, pp. 174–178.
C. M. Jones and I.-M. Jonsson, “Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses,” in Proceedings ofthe 17th Australia conference on Computer-Human Interaction: Citizens Online: Considerations for Today and the Future. Computer-HumanInteraction Special Interest Group (CHISIG) of Australia, 2005, pp. 1–10.
M. Ali, F. Al Machot, A. H. Mosa, and K. Kyamakya, “Cnn based subject-independent driver emotion recognition system involving physiological signalsfor adas,” in Advanced Microsystems for Automotive Applications 2016. Springer, 2016, pp. 125–138.
D. Datcu and L. Rothkrantz, “Multimodal recognition of emotions in car environments,” DCI&I 2009, 2009.
S. Hoch, F. Althoff, G. McGlaun, and G. Rigoll, “Bimodal fusion of emotional data in an automotive environment,” in Acoustics, Speech, and Signal
Processing, 2005. Proceedings.(ICASSP’05). IEEE International Conference on, vol. 2. IEEE, 2005, pp. ii–1085.

Sina Shafaei (TUM) | First International Workshop on Advanced Machine Vision for Real-life and Industrially Relevant Applications

15