# Detecting and Influencing Driver Emotions Using Psycho-Physiological Sensors and Ambient Light

Mariam Hassib, Michael Braun, Bastian Pfleging, Florian Alt

► To cite this version:

Mariam Hassib, Michael Braun, Bastian Pfleging, Florian Alt. Detecting and Influencing Driver Emotions Using Psycho-Physiological Sensors and Ambient Light. 17th IFIP Conference on Human-Computer Interaction (INTERACT), Sep 2019, Paphos, Cyprus. pp.721-742, 10.1007/978-3-030-29381-9_43. hal-02544538

HAL Id: hal-02544538

https://inria.hal.science/hal-02544538

Submitted on 16 Apr 2020

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# Detecting and Influencing Driver Emotions using Psycho-physiological Sensors and Ambient Light

Mariam Hassib¹, Michael Braun¹,², Bastian Pfleging¹,³, and Florian Alt¹,⁴

¹ LMU Munich, Munich, Germany

² BMW Group Research, New Technologies, Innovations, Munich, Germany

³ Universiteit Eindhoven, Eindhoven, Netherlands

⁴ Bundeswehr University, Munich, Germany

mariam.hassib@ifi.lmu.de, michael.bf.braun@bmw.de, b.pfleging@tue.nl,

florian.alt@unibw.de

**Abstract.** Driving is a sensitive task that is strongly affected by the driver's emotions. Negative emotions, such as anger, can evidently lead to more driving errors. In this work, we introduce a concept of detecting and influencing driver emotions using psycho-physiological sensing for emotion classification and ambient light for feedback. We detect arousal and valence of emotional responses from wearable bio-electric sensors, namely brain-computer interfaces and heart rate sensors. We evaluated our concept in a static driving simulator with a fully equipped car with 12 participants. Before the rides, we elicit negative emotions and evaluate driving performance and physiological data while driving under stressful conditions. We use three ambient lighting conditions (no light, blue, orange). Using a subject-dependent random forests classifier with 40 features collected from physiological data we achieve an average accuracy of 78.9% for classifying valence and 68.7% for arousal. Driving performance was enhanced in conditions where ambient lighting was introduced. Both blue and orange light helped drivers to improve lane keeping. We discuss insights from our study and provide design recommendations for designing emotion sensing and feedback systems in the car.

**Keywords:** Affective Computing, Automotive UI, EEG, Ambient Light

## 1 Introduction

Driving is a sensitive task, deeply embedded in our everyday lives. While modern cars are designed to reduce the driver's physical effort through assistive systems and features, the demand on focus and cognitive abilities is still high. Even as we move towards the era of (semi-) automated driving, we expect that drivers will still need to maneuver in various situations and take over control. Hence, it is important to understand and react to the driver's state [40, 56].

The driver's state does not only comprise cognitive abilities or how sleepy or focused they are, but also includes their emotional state. Prior research shows that emotions have a strong impact on driving performance and capabilities,
and negative emotions while driving (e.g., sadness, anger) can lead to undesired consequences and driving errors [15]. Extreme positive emotions like overexcitement, where the driver's arousal (i.e., activation) state is very high, can also have negative effects on driving [65, 27, 16, 61]. Hence, monitoring and reacting to driver emotion is an important rising area of automotive HCI research.

With wearable sensors and sensing capabilities embedded in modern cars we are a step closer to realizing the vision of having a ubiquitous sensing environment inside the car. Using sensors, researchers can detect driver drowsiness through camera-based methods and physiological sensing [28, 62], driver stress through GPS traces [59], or the driver's cognitive load and interruptibility using physiological sensors [31, 56]. While the importance of maintaining balanced emotional states while driving has been recognized, there is little work on closing the loop by not only sensing emotions, but also providing feedback [25, 41, 65].

We introduce the concept of a full sensing and feedback loop in automotive contexts using wearable physiological sensors and ambient light. We look into the use of light-weight psycho-physiological sensors as an implicit emotion detection method: Consumer-level bio-electric signals such as electroencephalography/electromyography (EEG/EMG) and heart rate (HR) sensors to detect emotional arousal and valence. These sensors have proven their ability to detect emotional and cognitive states with acceptable accuracies [4, 17, 23, 56]. On the feedback side, we explore the use of ambient light as an emotional feedback modality. Light was shown to have an effect on moods and emotions, e.g., by influencing the circadian system [10]. Ambient lighting in the car has been explored as a means of providing a more comfortable interior, through warning signals of upcoming traffic or to calm down the driver [35]. Combining input and output modalities we aim to assess the complete concept.

We investigate the effects of easy and stressful driving scenarios under elicited negative emotions on driver performance. In an experiment ($N = 12$), we explore two different ambient light colors (blue and orange) and their effects on the driving performance, physiological data, and self-reported emotional state. Results show that ambient lighting feedback can positively impact driving performance and lead to more focus or relaxed states. We envision a future where the car becomes an emotional feedback companion for the driver which attempts to support them by reacting to their emotional state.

**Contribution Statement** This paper makes the following contributions: First, we introduce our concept and vision of the car as an emotional sensor and feedback companion. We then present an evaluation of the concept in a static driving simulator with a real car, to investigate the influence of (a) negative emotions during easy and stressful rides and (b) ambient lighting on driving performance, physiological data, and self-reported emotional state. Third, we provide recommendations for designers of emotional feedback systems in cars.
## 2 Background & Related Work

When considering emotions in the car, we see three related research aspects, namely: the effect of emotions in driving scenarios, the detection of emotions using psycho-physiological sensors, and finally, in-car responses to regulate and influence driver emotions. We therefore divide prior work that influenced our research into these three main groups.

### 2.1 Emotion in Driving Scenarios

The emotional state of drivers has a strong impact on their driving performance [15, 20, 21, 29]. Prior work identified emotional states which influence driving and relate to driving safety [9, 27]. These include aggressiveness, happiness, anger, fatigue, stress, sadness, confusion, urgency, and boredom.

When driving a car, the driver's tasks are typically divided into three classes [9]: (1) *Primary driving tasks* include all necessary tasks in order to keep the vehicle on track such as steering, lane selection, accelerating, braking, and stabilizing, (2) *Secondary tasks* comprise activities to improve driving performance or safety (e.g., blinking, or activating wipers and headlights), and (3) *Tertiary tasks* consist of all other tasks that are performed while driving including changing temperature, adjusting radio settings, interacting with a cellphone or talking to other passengers. The aforementioned emotions differently impact the driver's tasks: Primary tasks are strongly related to safe driving and are usually compromised by negative emotions. Secondary and tertiary tasks affect the driver's comfort more than ensuring safe driving [9]. However, these factors often lead to a change in emotion or a shift in attention that endangers safe driving.

According to Russell's model of affect [52], emotions can be defined on two axes, valence and arousal: Valence refers to whether the emotion is more positive or negative, and arousal refers to the amount of activation in the emotion [52]. Using this model, research found positive emotions (i.e., a more positive valence) to result in a better driving performance and happy drivers to produce fewer accidents [20, 21, 29]. However, extremely positive emotions (having a very high level of arousal / activation) can also negatively affect safe driving [3, 21]. Yerkes and Dodson [63] found in an experiment that the best human performance values are measured with a medium level of arousal (activation), keeping in mind that the optimal level depends on task difficulty. Coughlin et al. [12] applied this model to the automotive domain.

Looking at negative emotions, prior work determined that aggressiveness and anger (i.e., low valence, high arousal) as well as sadness (i.e., low valence, low arousal) all negatively impact driving behavior and are shown to increase the risk of causing an accident [61, 13]. Sadness usually is accompanied by resignation and passiveness, resulting in longer reaction times not just in critical situations, but also by reducing the driver's attention [13]. The low arousal state may also result in fatigue or sleepiness, which is a very dangerous precondition since it negatively affects all abilities that are necessary for safe driving [28, 62].
As for all other tasks that require cerebral capacity, stress is very likely to occur while driving. The primary driving task itself is often a stressful task. Moreover, drivers often experience a higher workload due to additional tasks beyond driving: additional factors or tasks such as following a car, making faster progress (changing lanes during rush-hour traffic), receiving phone calls, the need to arrive on time, or communicating with passengers, increase the mental workload [56]. High mental workload comes with high arousal, which reduces driver performance [39, 46, 56, 59].

## 2.2 Driver Emotion Detection

The steady development of accurate emotion recognition techniques allows its application in different contexts, including driving. Eyben et al. [15] state four major modalities for emotion recognition in automotive contexts: audio (i.e. speech), video, driving style, and physiological measurements. However, not every measurement technique is suitable to detect every emotion. Prior work investigated the use of *audio* recording to detect anger and nervousness by employing speech features such as volume and pitch [14, 15, 41]. A disadvantage of speech in the car is the necessity for drivers to constantly speak or express themselves in an audible way. Emotion recognition from *driving style* was explored by different researchers to detect states of stress, high cognitive workload, interruptibility, and drowsiness [31, 59, 62]. High arousal states were found to result in more actions, such as frequent lane changes or having a large longitudinal variance, whereas low arousal states usually result in less active driving. Riener et al. [48] recognized nervousness from posture and motion in the seat. Their hypothesis was that nervous drivers move more than relaxed ones.

The emotional and cognitive states of humans is reflected through physiological signals which can be detected using, for example, body-worn sensors providing fine-grained feedback. Implicit emotion recognition while driving using *psycho-physiological sensors* was investigated by several researchers [47, 55]. For example, heart rate gives an indication of the driver's state of arousal [30, 56]. Lower heart rates indicate a more relaxed state, whereas higher heart rates occur during high driver activation. Respiration rate is also connected to arousal states, slower and shallower breathing indicates a relaxed state whereas alerted or active states result faster breathing and indicate emotional excitement [15]. Skin conductance levels (SCL) are associated with measures of emotion, arousal, and attention [25, 56]. EEG signals measured from the top of the scalp give information about the cognitive and emotional state of the user [8, 23, 49].

Katsis et al. [30] used EMG, HR, respiration, and SCL to classify stress, euphoria, and disappointment in car-racing drivers. De Waard et al. [60] conducted a field study to investigate the effect of driving on different types of roads on the heart rate variability (HRV) and consequently the mental demand of drivers. Solovey et al. used machine learning classifiers with features from HR, SCL, and driving performance to detect the driver's mental workload [56]. Healey et al. [24] classified stress levels using HR, EMG, SCL, and respiration during driving on highways and urban roads. Jahn et al. [26] conducted a large-scale
study and concluded that heart rate changes reflect emotional strain. Collet et al. [11] collected heart rate and skin resistance data during driving on a closed track and concluded that both physiological measures increased when performing additional tasks such as phone conversations. EEG sensing was used to detect drowsiness while driving [8] and to detect cognitive states in simulated virtual reality driving [34]. Schneegass et al. [54] presented a real-world driving study in which ECG, SCL, and skin temperature data was collected while participants drive in differing road environments. They found that SCL varied significantly across road types [54].

## 2.3 In-Car Responses to Regulate & Influence Emotions

While the larger body of automotive affective computing research is concerned with reliably detecting emotions, reflecting and regulating emotions once detected remains a challenge. Research introduced multiple mitigation strategies to either increase the driver's awareness of their emotional state [25] or introduced design suggestions to help shifting the driver's state to a more desirable one [65]. Zhu et al. [65] and Fakhrhosseini et al. [16] investigated the use of music to relieve anger situations while driving. Braun et al. explored the viability of ambient light, visual feedback, voice interaction, and an empathic voice assistant as strategies to regulate sadness and anger while driving [6]. Nass et al. investigated mirroring voice with driver emotions and found that when drivers' emotions matched the car's voice emotion, they had fewer accidents, focused more on the road and spoke more to the car [41]. Harris and Nass researched behavioral and attitudinal effects of cognitively re-framing frustrating events using voice prompts [22]. They found that voice prompts telling drivers that the actions of others on the road were unintentional reduced driver frustration and negative emotions [22]. Roberts et al. [50] studied the differences between warning users through visual and auditory alerts in real-time or post-hoc. They found drivers to be more receptive to post-hoc critic [50]. Hernandez et al. envision a concept of a reflective dashboard, making drivers aware of their stress levels measured through skin conductance sensors by showing red or green light. They showed that people slow down upon red light [25].

## 2.4 Summary

Related work shows that negative emotions impact driving performance. Researchers investigated the use of physiological sensors to gain insight into driver emotions, and, more recently, started to explore different design opportunities to reflect, relieve, or mitigate negative emotions. In our work, we present a concept which combines emotion detection and reflection in the car. We investigate the feasibility of using lightweight EEG and heart rate sensors to detect negative valence while driving, and the effects of using dashboard ambient lighting to reflect and influence emotions.
# 3 Concept and Vision

We envision the car as a companion which senses, reflects, and communicates feedback to the driver in a subtle and seamless manner. Our concept uses psycho-physiological sensors for continuously detecting the driver's emotional state without jeopardizing drivers' attention by asking repeatedly for subjective feedback (e.g., by using questionnaires). To provide emotional feedback to the driver we use ambient lighting on the dashboard through LEDs to provide subtle, yet perceivable feedback. The intention is that this light shifts the driver's emotions towards a desirable state through emotional awareness and regulation. Below we discuss both input and output modalities used in our concept.

## 3.1 Emotion Detection: Psycho-physiological Sensing

Researchers explored different psycho-physiological correlates that enable emotion recognition [4]. Signals captured from the human body reveal a plethora of information about users' current emotional, physical and cognitive states. In our concept we rely on EEG/EMG and heart rate sensing wearables. The proliferation of consumer-level wearable sensors into the market in suitable form factors allowed researchers to further explore their use in HCI [23].

In our concept, we use both consumer-level EEG and heart rate sensors for emotion detection. Whereas heart rate has been successful in detecting arousal rates [17], EEG has been successful in detecting emotional valence [4, 34]. Physiological sensors in general allow for collecting fine-grained unbiased emotional information, without adding further workload on users which is critical when driving a car. In addition, compared to camera-based techniques, using physiological sensors is not sensitive to light conditions or occlusions. On the other hand, physiological sensing, is person-dependent and prone to be influenced by muscle and movement artifacts [57].

## 3.2 Emotion Feedback: In-Car Ambient Light

For the output modality, we chose ambient lighting as a subtle way to visualize feedback in the car. Using different lighting techniques in the car is not a new concept in itself. Many modern cars include ambient lighting to provide a feedback about different states (e.g. doors open, car locked), or as reading lights (for example, BMW Moodlight<fnref n="5" />). Outside the car, ambient lighting is also used in other road environments such as tunnels<fnref n="6" />. This familiarity makes it a useful and suitable modality to augment the car's interior with further information that can easily be perceived by the driver.

Prior work investigated using ambient lighting in the car for signaling, for increasing awareness [36], enhancing night vision [51], or signaling upcoming road

<footnote n="5">5 https://legacy.bmw.com/com/en/newvehicles/x/x6/2014/showroom/design/ambiente-light.html, accessed February 2018</footnote>

<footnote n="6">6 http://www.thornlighting.com/download/TunnelINT.pdf, accessed September 2018</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** A driver in our simulator study to evaluate our concept, wearing the EEG and heart rate sensors during the blue (left) and orange (right) ambient lighting conditions. The sensors were used to detect the driver's emotions while the ambient light was used to influence driving behavior. Both light colors improved driving performance compared to a baseline ride due to their warning (orange) and calming (blue) effects.</figcaption></figure>

conditions [33]. Löcken et al. present a survey on in-car ambient lighting [35]. However, ambient lighting in the car has rarely been used to reflect and influence the driver's emotional state.

In our concept we chose two ambient lighting colors, a cool color (blue) and a warm color (orange): Blue ambient lighting is related to vitality, energy, and power. Additionally, it is perceived as a calming and pleasant color but barely arousing emotions [37]. Red and orange are associated with a higher arousal level [35]. To differentiate the warm color stimulus from a warning signal (e.g. such as traffic lights), we chose orange instead of red to increase arousal. To evaluate our concept, we conducted a simulator study that integrates different emotion evoking rides and uses psycho-physiological sensors for emotion detection and ambient light conditions for regulation and reflection.

# 4 Simulator Study: Emotional Driving

To evaluate our concept, we conducted a driving simulator study equipped with a real car. In the study we tested the effect of driving performance under negative elicited emotions during easy and stressful rides, and different ambient lighting conditions. Our main goals were: (1) to analyze psycho-physiological responses during actual driving context and the feasibility to classify emotions in this setup using light-weight wearable sensors; (2) to analyze the effect of negative emotions while driving easy and hard rides; and (3) to investigate the effect of ambient lighting on driving performance and emotional arousal and valence.

## 4.1 Apparatus

**Emotion Elicitation** In this study we focused on driving starting in a negative emotional state. As we have presented in the related work section, negative emotions such as sadness have a negative effect on driving performance.

To ensure that drivers were in a negative state before the start of the driving tasks, we used the DEAP database [32] which consists of 120 excerpts of music
videos from different music genres that are rated according to valence and arousal on the SAM scale [5]. This database was already used and evaluated with medical grade EEG data collection and promising results were found: In a lab study, Koelstra et al. extracted 40 videos from the database which showed the strength of elicited emotions [32]. For our study, we chose four videos from the dataset that were ranked lowest. These videos (#23, #24, #28, and #30) were all rated in the low arousal and low valence quadrant [32].

**Driving Simulator and Ride Description** Our static driving simulator consisted of a fully equipped stationary car (BMW i3), a projector, and speakers. The projector showed the driving scenario on a 5 m×3 m wall. We used four drives in our study: one easy baseline drive where the driver had a car-following task on an almost empty highway, and three stressful car-following drives where the driver was on a busy highway and faced several annoying driving maneuvers from other drivers. Each drive was six minutes long.

**Baseline drive:** The simulation was modeled according to SAE J2944 standard criteria [19]. The driver follows another vehicle in the center of the lane, with constant speed and headway, without lane changes, on a straight highway.

**Stressful drives:** This concept was adapted from Schmidt et al. [53] who designed a number of traffic scenarios to induce negative emotional states. The rides contain multiple lane changes and various stressful events, such as a close encounter with trucks or a construction site with narrowed lanes. Participants were also instructed to follow a designated vehicle in the center of the lane and keep a constant and safe distance.

**Data Collection** During the study we collected physiological data, driving performance, and emotional ratings. To collect and record EEG/EMG signals, we used a Muse brain-sensing headband<sup>7</sup>. This headband uses four electrodes placed on the frontal and parietal lobes according to the 10-20 positioning system, namely: AF7, AF8, TP9, and TP10. The device provides access to raw EEG and relative EEG frequency bands, blinks, and jaw clenches. The data is sent to a computer via Bluetooth. To measure participants' heart rate, we used a Polar H7 chest strap sensor<sup>8</sup>. The sensor sends HR information via Bluetooth low energy at a rate of 1 Hz. All data streams and task triggers were combined in an experimenter interface, where consistent timestamps were assigned.

To collect ground truth data about driver emotions in a driving context, we used the automotive self-assessment method (ASAM) [7]. Using a 9-point SAM would have been quite intrusive during the rides. In this case, users would need to choose a SAM rating from radio buttons during driving. On the other hand, asking users to verbally indicate their emotional ratings whilst driving can lead to biased results due to the experimenter being there to collect the answers.

<footnote n="7">⁷ https://www.choosemuse.com/</footnote>

<footnote n="8">⁸ https://www.polar.com/us-en/products/accessories/H7_heart_rateSensor</footnote>
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Three images showing the simulator study setup: (A) The projected driving scenario during an overtake. The distance to the followed vehicle is shown in yellow (B) The dashboard of the car showing the ambient lighting LEDs around the wheel and along the passenger side. On the right, the tablet is shown depicting the continuous ASAM scale. (C) The driving simulator showing the stationary car and the projected driving scenario.</figcaption></figure>

Hence, we fitted a tablet to the right of the driver with two continuous scales which can easily be reached and clicked by the driver with the right hand. Figure 2 (B) shows the interior of the car, depicting the tablet, the scales, and a smiley face in the middle. The top scale, arousal, is reflected in the eyes of the smiley face in the middle which goes from a sleepy face to an awake face. The bottom scale depicts the valence and it adjusts the mouth of the smiley going from negative to positive. The scales are from 1–100. We adjusted the sensitivity of the scales so that the driver can click anywhere over or under the top or bottom of the scale and it would adjust accordingly. The tablet was always within arm’s reach. Finally, we collected driving data through the driving simulator. This included speed and acceleration, distance to followed car, lane variations, and crashes.

**Dashboard Ambient Light** We used Philips Hue<fnref n="9" /> LED light stripes with 1,600 lumen to create ambient light insight the car. Connected over the Philips Hue bridge, we selected the colors of the light strips with the corresponding mobile app. We used a 2 m strip of the Hue LEDs which were fixed around the dashboard as shown in Figure 1 and Figure 2 (B). As explained in the concept section, we evaluated the effect of two colors, blue and orange.

<footnote n="9">⁹ https://www.meethue.com/, last access: 2018-09-19</footnote>
<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Study procedure block diagram showing each step with durations. The baseline relaxation phase and easy drive were always fixed in the beginning. The order of the color conditions during the stressful drives was counter balanced between participants. In the end a debriefing session and semi-structured interview were conducted.</figcaption></figure>

## 4.2 Study Design

We used a repeated measures design with two independent variables, namely, driving scenario (4 levels) and light color condition with three levels (no light, blue light, orange light). As explained previously, we had four main drives – one baselines drive and three stressful drives. The duration of all drives was six minutes. During the baseline drive, no ambient light was triggered. One stressful drive was in the *no light* condition, where no light was triggered, one was in the *blue light* condition, and one in the *orange light* condition.

Figure 3 illustrates a block diagram of the procedure of the whole study with durations. The light was triggered in fixed intervals of one minutes and lasting for 30 seconds each time. ASAM ratings were triggered at 1.5 minute intervals constituting four ASAM ratings per drive. The order of the rides was counter-balanced to reduce learning effects. Figure 4 depicts the process of triggering light and ASAM experience sampling questions during the stressful drives, with (a) showing the light conditions and (b) the no light condition.

## 4.3 Participants and Procedure

Twelve participants took part in our study (4 females, 21–61 years, M = 31, SD = 11.4). Participants were mostly engineers or students, all had driving licenses.

After our participants arrived at the lab we explained that the purpose of the study was to collect physiological data while driving in different scenarios and showed them the sensors. We introduced how we collect the subjective ASAM feedback on the mounted tablet during the ride and explained that the participant's input will be triggered several times during each ride with a short beep sound. Participants did not know a priori about the use of the installed ambient light. Before the study, the participants signed a consent form.
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** The procedure of one run from the stressful drives which included the color condition (no light, blue, orange). (a) depicts timings for the blue and orange light conditions, and (b) depicts timings for the no light condition. The timing of the ASAM triggers was exactly the same as for the blue/orange/no light as the figure shows.</figcaption></figure>

We first asked the participants to put on the sensors and ensured good con-
tact. Next, participants adjusted the car seat and started a short test drive to
get used to the car and simulation. The scenario used for this ride was an empty
highway. A test ASAM question was then triggered on the tablet with a short
beep and participants were requested to answered it while driving. When partic-
ipants stated to be comfortable with driving, we terminated the test drive and
started the study.

The first part of the study included a one minute relaxation task to collect baseline EEG and HR measurements. Afterwards, participants watched the first music video on the projection wall while they were seated in the car and received an ASAM prompt at the end of the video clip. The first ride was then the baseline ride for six minutes. We reminded the participants that they should keep a distance between 50 to 70 meters to the car lead vehicle. After the end of this ride, the participants continued with the three other video-ride combinations with the different color conditions. The order of the videos and the ambient light conditions were randomized. After the study we conducted a short semi-structured interview to gather feedback about their perceptions of the rides and the ambient lighting conditions. Participants were asked whether the emotion elicitation worked, if and how they perceived the different lighting modes, and whether they think any of these stimuli influenced their driving performance or stress levels. The duration of the study was around 1 hour.

5 Emotional Driving Study Results

In the following we discuss the results from our study, including the analysis of
the subjective in-car experience sampling emotion ratings, the classification of
physiological data, and finally the driving performance analysis.

**5.1 Emotional Ratings**

We collected 480 ratings from the twelve participants, 240 for each arousal and valence. Four ratings per drive and one rating per music video making up 20 rat-
ings for each arousal and valence from each participant. We calculated the mean and standard deviations of the arousal and valence scores from the continuous 1–100 ASAM ratings. Our results show that, first, the music videos were indeed successful in putting participants in a negative valence before each ride, with a mean rating of 48.5 (SD = 23.0) for arousal and 40.25 (SD = 18.04) for valence. Participants rated the easy baseline rides with a mean of 57.3 (SD = 18.34) for arousal and 52.5 (SD = 16.5) for valence. They rated stressful drives with no ambient lighting almost the same on the arousal scale (M = 57.9, SD = 20.16) but lower on the valence scale (M = 48.2, SD = 15.24), indicating that they were in a more negative mood during the stressful rides.

Looking at the ambient lighting conditions, we found that participants rated both arousal and valence higher than for the no ambient lighting condition for both the orange and the blue lights. The mean arousal for blue light was 61.5 (SD = 18.34), and the mean valence was rated 53.4 (SD = 17.38). For the orange ambient lighting condition the mean arousal was 61.04 (SD = 16.5), and the mean valence was rated 52.04 (SD = 16.8).

Since the scales for arousal and valence are nonparametric, we used nonpara-
metric tests to test for significance (Friedman and Wilcoxon tests). Wilcoxon
sign-rank test for pairwise comparisons yielded no significant results except for
valence between videos and the blue light condition (p=0.003), and valence of
light and no-light condition (p=0.02). The results overall show an increase in
valence in the ambient lighting conditions compared to the no light condition
under the same stressful driving scenario.

## 5.2 EEG and HR Classification

For the analysis of the heart rate we used the data collected via the Polar chest strap. The data from three participants was removed due to hardware issues. We averaged the heart rate from the last minute for each drive per person to get insights into the overall change in heart rate depending on the drive type [53]. The mean baseline heart rate was 67.4 bpm (SD = 8.4). For the easy drives, the mean heart rate was 69.6 bpm (SD = 7.4). The stressful drives all increase the heart rate means from the baseline and easy drives with the stressful drive in the *no light* condition having the highest average of 71.8 bpm (SD = 7.7). The stressful drive under the *blue light* condition had a mean of 70.2 bpm (SD = 6.6) and finally the stressful drive with *orange light* achieving a mean of 71.4 bpm (SD = 5.4).

Although the data from only nine participants was considered in the analysis, we see that heart rates increased for the stressful drives compared to the baseline and easy drives. Additionally, the blue light condition achieved lower heart rates than both the orange and the no light conditions.

For drives in the ambient lighting conditions, we analyzed the 30 second seg-
ments which had blue or orange light compared to the 30 second segments before
or after. A Wilcoxon sign-rank test found significant effects on the heart rate be-
tween the 30 seconds before the orange segment and the 30 seconds during the
orange segment (Z = -1.955, p = 0.05). Whereas we did not find significant
differences for the blue segments and the segments before them, we found signif-
icant differences when comparing the blue segments to the segments after them
(Z = -2.037, *p* = 0.038). This shows that the blue and orange ambient lighting
had indeed an effect on heart rate. Overall, heart rate decreased in the stressful
rides with ambient lighting compared to the no light stressful ride.

For the analysis of the EEG data, we first extracted the EEG frequency band
powers provided by the Muse headband, which were common average referenced
and band-passed between 0.1 Hz and 30 Hz and notch-filtered at 50 Hz. We first
epoched the EEG data into 2.5-second windows. We calculated the 2.5-second
mean of the spectral powers for each electrode and frequency resulting in 20
features. We calculated 20 more features from asymmetry differences and asym-
metry ratios that were successful in prior work [64]. The asymmetry differences
for each frequency band on each electrode pair (TP and AF) were calculated
as follows: AsymD<sub>f</sub> = f<sub>Right</sub> - f<sub>Left</sub> where AsymD represents the asymmetry
difference and f are the left (AF7, TP9) and right side (AF8, TP10) mean spec-
tral powers. Calculating all asymmetry values for all frequency bands produces
another 10 features. We calculated the asymmetry ratios of the frequency bands
according to the formula AsymR<sub>f</sub> = f<sub>Right</sub>/f<sub>Left</sub>, where AsymR is the ratio
between two frequency bands and f are the left (AF7, TP9) and right side (AF8,
TP10) mean spectral powers resulting in 10 more features (40 features in total).

We labelled the data according to the aggregated ASAM scores collected from
the digitized ASAM ratings presented on the tablet to obtain a score between
1 (low arousal/valence) to 4 (high arousal/valence). We chose a random forest
classifier and classified the data using Weka<fnref n="10" />. This particular classification al-
gorithm was chosen due to its success application in other EEG classification
tasks [23, 64]. We performed a person-dependent classification with a 10-fold
cross validation.

The results are promising for classifying 4-class arousal and valence ratings.
For arousal, all four classes were represented through our participants' ASAM
ratings. F1 scores have an average of 68.7% over all four classes. For the valence
classification, F1 scores have an overall average of 78.9% for all four classes,
albeit the absence of two of the classes (classes 1 and 4 ) completely from three
participants and the representation of only one class for one participant (P10).

5.3 Driving Performance Analysis

We calculated mean headway variability as well as standard deviation of lane position (SDLP) for each tested concept and ride. Headway variability is influenced by the behavior of preceding traffic, like lane changes, and provides a value of how well a driver is following the car in front [1]. We observed a mean headway variability of 52.98 *m* (*SD*=5.63 *m*) for the baseline ride and a significantly higher value of 73.00 *m* (*SD*=10.56 *m*) for the stressful ride without lights (*F* = 14.65, *p* < 0.001). Orange and blue lights during the ride did not lead to significant differences to either baseline or no-light condition with

<footnote n="10">¹⁰ http://www.cs.waikato.ac.nz/ml/weka/</footnote>
<figure><img src="image_6.png" /><figcaption>**Fig. 5.** Results from the driving performance analysis. Top: The overall SDLP during the orange and blue light conditions showing the variations between light on and light off segments. Bottom: The mean and SD of headway variability (left) and SDLP (right) for each of the rides. The two right most bars show lower values during the segments with the orange or blue lights on.</figcaption></figure>

67.49 *m* (*SD*=11.69 *m*) and 61.00 *m* (*SD*=8.14 *m*), respectively. If we look at the subsections of each ride where light was displayed, we can, however, see significant differences to all rides (Figure 5, left). Orange light led to a headway variability of 32.82 *m* (*SD*=19.79 *m*) and blue light to 42.74 *m* (*SD*=17.59 *m*). This is a substantial decrease in headway variability when lights are displayed.

The standard deviation of lane position (SDLP) is a measure of lateral movement during the ride which is considered a core metric for assessing driving performance in simulations and provides high test-retest reliability [42, 58]. We report insignificant differences between the four rides with SDLPs from 0.47 *m* to 0.49 *m* as shown in Figure 5 (middle). Here again, the segments of the ride where light was shown improved the driving performance significantly (*F* = 19.38, *p* < 0.001). When orange light was displayed, a SDLP of 0.28 *m* (*SD* = 0.04 *m*) was measured and blue light performed comparably with 0.28 *m* (*SD* = 0.07 *m*).

At first glance, we suspected the data was influenced by sequence effects as the lights were always shown during the ride and not at the very start. We could, however, verify the effect by visualizing the ride progress and associated SDLP values. Figure 5 (right) shows the values for sequences with and without light
compared to the polynomial trend of the stressful ride without lights. We can
clearly see here that SDLP is lower when the lights are turned on and higher if
they are off.

5.4 Qualitative Feedback

We collected feedback through semi-structured interviews after the study. All participants stated that the drives were quite stressful, due to all the overtaking and catching up, and following the car. This indicates that the rides were successful in putting participants in a challenging situation.

When we asked participants how they perceived the different ambient lighting conditions, we got varying opinions. Several participants stated that they surely perceived the lights but did not think it had any relation or effect on their driving performance or mood (P1, P2, P4, P5). Two participants stated that they felt the lights were alerting them to be more focused on the road and avoid getting bored, distracted, or sleepy, regardless of the color of the light (P3, P7). One participant stated that the effect of the driving scenario on him is greater than the effect of any ambient lighting regardless of the color (P11). Two participants indicated that the orange light made them more alarmed, since it uses the same color metaphor as alerts (P9, P4). One participant stated that the orange color made him more 'critical' of his driving, thinking back at what he did wrong and what he can do better in the following phase (P9). Two participants stated that the blue light made them feel more relaxed, comfortable yet focused. However they were not sure if that really had an effect on their driving (P8, P10).

Most participants perceived blue light as relaxing and providing a nice feel to the interior of the car, whereas orange was perceived as an alarming, undesirable light, except for short periods of time to make users focus more on the road.

5.5 Limitations and Lessons Learned

We explored the feasibility of using psycho-physiological sensors and ambient lighting in a real vehicle. For this, we utilized light-weight wearable sensors for emotion recognition. We acknowledge that this setup could have introduced more artefacts in the measured physiological data than a controlled context. We used a machine learning approach with signal filtering algorithms to pre-process the data aiming to reduce artefacts. However, more complex signal processing approaches for more rigorous artefact filtering would be required in a scenario, e.g., with a moving car), to compensate for movement artefacts.

For three participants, heart rate was not recorded properly. Hence, we decide to exclude this feature from classification. We acknowledge that using features from heart rate information such as heart rate variability (HRV) could further enhance the classifier model [56].

We used videos to elicit emotions at the beginning of each ride to have consistent emotional baselines across all participants. We only elicited negative emotions on the low arousal and low valence level as a starting point before the beginning of each drive. In a real scenario the emotional states of the user may be
more diverse, for example, highly excited or very angry. To keep our study con-
sistent and confined in timing, we deliberately focused on certain combinations
of arousal and valence. Future work could look at more combinations.

Finally, eliciting emotions for studies is a challenging task. Future work could
look at using other methods for doing so.

# 6 Discussion & Design Recommendations

We discuss our findings and provide recommendations to designers of emotional
feedback in the car. We provide insights regarding implicit emotion sensing and
privacy, the use of ambient lighting as emotional awareness or influencing modal-
ity. Finally, we suggest how the findings from our studies can be used in ubiqui-
tous road environments and for semi-autonomous driving scenarios.

## 6.1 Emotion Sensing: Privacy Considerations

The use of physiological sensing to detect emotions has been subject to recent
research. It is no longer confined to laboratory settings and experiments but
slowly finds it way into day-to-day life contexts. This creates the need for several
privacy considerations. Emotions, naturally, are very private [44]. People have
the freedom to hide their emotions by not talking about them or keeping a
neutral facial expression purposefully.

However, overriding or faking emotions that are collected through physio-
logical sensing is quite difficult [2, 38, 44]. Does this mean that future affective
systems diminish the choice of self-expression and desired state of self presenta-
tion (cf. Goffman's work on self representation [18])?

In our first investigation of the concept, we did not consider the car a social
setting shared with other people. Albeit that, we got feedback from our semi-
structured interviews that tapped into this area. One participant even mentioned
that he was feeling watched, although he knew that no one is currently looking at
his sensed data and neither is it shared with anyone. Multiple other participants
stated that they felt as if the car is warning them about themselves or criticizing
their driving (mostly in the orange light condition). Note, that in our study, the
drivers were the only people in the simulator and no other drivers or passengers
were in the car. This means, the emotional feedback was limited to the driver.
This suggests that, counter-intuitively, situations were the user is driving alone
should be subject to investigation, looking into how emotional states can be
presented in a privacy-preserving manner [45]. In addition, this is also relevant
in situations were other passengers are present.

We encourage designers of emotional feedback systems to alter the feedback
depending on the context. For example, when using ambient lighting, designers
can limit the location of the feedback light to the front of the driver only when
multiple passengers are in the car. This however, may affect how the light affects
the driving performance. Future work should further investigate scenarios with
passengers, considering in particular their relationship to the driver.
## 6.2 Ambient Lighting: Awareness or Influence

Our drivers did not know a priori what the ambient lighting meant. Qualitative feedback showed that multiple participants thought that the light was triggered in reaction to either their sensed physiological data or their subjective emotional feedback. Multiple participants stated that the orange light, owing to its closeness to red, indicated that something was wrong, and raised their *awareness*. They stated that they definitely focused and drove better afterwards. This was also reflected in the driving performance analysis where the lowest variability in headway and in lane positions was achieved during period of orange light. In contrast, participants stated that the blue light was there to *influence* their emotional state and driving performance making them more relaxed.

Through our study we cannot determine if one type of feedback, *awareness or influence*, worked better. While our participants drove better under the *orange* condition, which multiple participants felt was an *awareness* cue, several participants stated that they did not find the orange light very comfortable. On the other hand, the blue drives were also successful in reducing driving errors, and also in reducing the heart rate. This shows that it indeed had a calming effect on the drivers. This is in line with findings from prior work. For example, Nass et al.'s work on mirroring in-car voice to current emotions [41] which proved to work better using a contrasting tone to the current emotion.

Designing emotional feedback, be it ambient lighting or a different form, can fall into either category. While we only evaluated the use of two colors during emotional driving scenarios, it was clear that there is indeed an effect based on the choices of colors. Future work should investigate the mental models associated with the different forms of feedback, or variations in one form (e.g. colors in ambient lighting scenarios) as well as personally customized color choices. Designers of emotional feedback systems should ensure that users have the correct mental model of the system.

## 6.3 Ambient Lighting in the Wild

Through our studies, participants repeatedly mentioned their familiarity with ambient lighting as a modality, from its recent integration in home and car environments. We see this as an opportunity for providing and influencing emotional states on the road. Several participants mentioned that night lights on the streets and in particular in tunnels can use this concept. A possible idea would be to use blue lighting in tunnels, e.g., to calm drivers down, especially those not comfortable with driving in narrow and dark places.

Another suggestion is to use car-to-car communication systems to trigger lighting in or outside of the car, depending on the traffic state. For example, when there is traffic congestion or an accident, the predicted emotional state of the drivers arising from these traffic situations could be considered. Extending this concept to other types of vehicles, such as buses and trains, by equipping the vehicle with LED lights can not only influence the driver, but also other passengers whose wellbeing influences driving performance through decreasing
distractions [43]. In the aviation industry, ambient light similarly supports the
flight experience and helps to arrive relaxed and with less jetlag<fnref n="11" />.

7 Conclusion and Future Work

In this work we explored the concept of using physiological sensing, namely EEG and HR, as emotion sensing during driving scenarios, and ambient lighting as emotional feedback. In a simulator study with a real car we investigated (1) the feasibility of classifying emotions based on physiological data collected in context, and (2) the effect of different ambient lighting conditions on the emotional state and driving performance during stressful driving scenarios. Our findings show that it is possible to use light-weight sensors to classify emotional arousal and valence in a driving context with an acceptable accuracy. We also found that using ambient lighting in the car enhances driving performance. Participants found that blue light relaxed them and that orange light made them more critical of their performance.

Future work could explore the design of different ambient lighting colors and locations. We intend to explore scenarios with multiple passengers in the car. In addition, we are interested in exploring the use of physiological sensors and ambient lighting in a real road driving scenario. Also, embedding more sensing technologies (e.g., measuring the skin conductance level, SCL) may allow higher classification accuracies and more fine grained information to be achieved.

References

1. AAM Driver Focus-Telematics Working Group: Statement of principles, criteria and verification procedures on driver interactions with advanced in-vehicle information and communication systems. Washington, DC: AAM (2006)

2. Arapakis, I., Konstas, I., Jose, J.M.: Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In: Proc. MM ’09. pp. 461–470. ACM, New York, NY, USA (2009). https://doi.org/10.1145/1631272.1631336

3. Arnett, J.J.: Developmental sources of crash risk in young drivers. Injury Prevention 8(suppl 2), ii17–ii23 (2002). https://doi.org/10.1136/ip.8.suppl.2.ii17

4. Bos, D.O.: Eeg-based emotion recognition the influence of visual and auditory stimuli (2007)

5. Bradley, M.M., Lang, P.J.: Measuring emotion: the self-assessment manikin and the semantic differential. Journal of behavior therapy and experimental psychiatry 25(1), 49–59 (1994)

6. Braun, M., Schubert, J., Pfleging, B., Alt, F.: Improving driver emotions with affective strategies. Multimodal Technologies and Interaction (2019)

7. Braun, M., Serres, K.: Asam: an emotion sampling method for the automotive industry. In: Adj. Proc. AutoUI ’17. pp. 230–232. ACM (2017)

8. Brown, T., Johnson, R., Milavetz, G.: Identifying periods of drowsy driving using eeg. Annals of advances in automotive medicine 57, 99 (2013)

<footnote n="11">¹¹ http://www.a350xwb.com/cabin/, last access: 2018-09-20</footnote>
9. Bubb, H.: Fahrerassistenz - primär ein beitrag zum komfort oder für die sicherheit? In: Der Fahrer im 21. Jahrhundert. pp. 25–44. VDI-Berichte 1768, VDI-Verlag, Düsseldorf, Germany (2003)

10. Canazei, M., Weiss, E.: The influence of light on mood and emotion, vol. 1, pp. 297–306 (10 2013)

11. Collet, C., Clarion, A., Morel, M., Chapon, A., Petit, C.: Physiological and behavioural changes associated to the management of secondary tasks while driving. Applied Ergonomics 40(6), 1041 – 1046 (2009). https://doi.org/10.1016/j.apergo.2009.01.007, psychophysiology in Ergonomics

12. Coughlin, J.F., Reimer, B., Mehler, B.: Monitoring, managing, and motivating driver safety and well-being. IEEE Pervasive Computing 10(3), 14–21 (7 2011). https://doi.org/10.1109/MPRV.2011.54

13. Dula, C.S., Geller, E.: Risky, aggressive, or emotional driving: Addressing the need for consistent communication in research. Journal of Safety Research 34(5), 559 – 566 (2003). https://doi.org/10.1016/j.jsr.2003.03.004

14. Eyben, F., Wöllmer, M., Graves, A., Schuller, B., Douglas-Cowie, E., Cowie, R.: On-line emotion recognition in a 3-d activation-valence-time continuum using acoustic and linguistic cues. Journal on Multimodal User Interfaces 3(1), 7–19 (Mar 2010). https://doi.org/10.1007/s12193-009-0032-6

15. Eyben, F., Wöllmer, M., Poitschke, T., Schuller, B., Blaschke, C., Färber, B., Nguyen-Thien, N.: Emotion on the road—necessity, acceptance, and feasibility of affective computing in the car. Advances in human-computer interaction 2010 (2010). https://doi.org/10.1155/2010/263593

16. Fakhrhoseini, S.M., Landry, S., Tan, Y.Y., Bhattarai, S., Jeon, M.: If you're angry, turn the music on: Music can mitigate anger effects on driving performance. In: Proc. AutoUI '14. pp. 18:1–18:7. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2667317.2667410

17. Gable, T.M., Kun, A.L., Walker, B.N., Winton, R.J.: Comparing heart rate and pupil size as objective measures of workload in the driving context: Initial look. In: Adj. Proc. AutoeUI '15. pp. 20–25. ACM, New York, NY, USA (2015). https://doi.org/10.1145/2809730.2809745

18. Goffman, E.: The presentation of self. Life as theater: A dramaturgical sourcebook (2006)

19. Green, P.: Standard definitions for driving measures and statistics: Overview and status of recommended practice j2944. In: Proc. AutoUI '13. pp. 184–191. ACM, New York, NY, USA (2013). https://doi.org/10.1145/2516540.2516542

20. Grimm, M., Kroschel, K., Harris, H., Nass, C., Schuller, B., Rigoll, G., Moosmayr, T.: On the Necessity and Feasibility of Detecting a Driver's Emotional State While Driving, pp. 126–138. Springer, Heidelberg (2007). https://doi.org/10.1007/978-3-540-74889-2_12

21. Groeger, J.A.: Understanding driving: Applying cognitive psychology to a complex everyday task. Psychology Press (2000)

22. Harris, H., Nass, C.: Emotion regulation for frustrating driving contexts. In: Proc. CHI '11. ACM, NY, USA (2011). https://doi.org/10.1145/1978942.1979050

23. Hassib, M., Pfeiffer, M., Schneegass, S., Rohs, M., Alt, F.: Emotion actuator: Embodied emotional feedback through eeg and ems. In: Proc. CHI '17. pp. 6133–6146. ACM, New York, NY, USA (2017). https://doi.org/10.1145/3025453.3025953

24. Healey, J.A., Picard, R.W.: Detecting stress during real-world driving tasks using physiological sensors. IEEE Transact. ITS 6(2), 156–166 (June 2005). https://doi.org/10.1109/TITS.2005.848368
25. Hernandez, J., McDuff, D., Benavides, X., Amores, J., Maes, P., Picard, R.: Autoemotive: Bringing empathy to the driving experience to manage stress. In: Proc. DIS Companion '14. pp. 53–56. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2598784.2602780

26. Jahn, G., Oehme, A., Krems, J.F., Gelau, C.: Peripheral detection as a workload measure in driving: Effects of traffic complexity and route guidance system use in a driving study. Trans. F 8(3) (2005). https://doi.org/10.1016/j.trf.2005.04.009

27. Jeon, M., Walker, B.N.: What to detect? Proc. Human Factors and Ergonomics Society Annual Meeting 55(1), 1889–1893 (2011). https://doi.org/10.1177/1071181311551393

28. Ji, Q., Yang, X.: Real-time eye, gaze, and face pose tracking for monitoring driver vigilance. Real-Time Imaging 8(5), 357 – 377 (2002). https://doi.org/10.1006/rtim.2002.0279

29. Jones, C.M., Jonsson, I.M.: Automatic recognition of affective cues in the speech of car drivers to allow appropriate responses. In: Proc. OZCHI '05. pp. 1–10. CHISIG, Narrabundah, Australia (2005). https://doi.org/1108368.1108397

30. Katsis, C., Ntouvas, N., Bafas, C., Fotiadis, D.: Assessment of muscle fatigue during driving using surface emg. In: Proc. IASTED BioEng '04'. vol. 262 (2004)

31. Kim, S., Chun, J., Dey, A.K.: Sensors know when to interrupt you in the car: Detecting driver interruptibility through monitoring of peripheral interactions. In: Proc. CHI '15. pp. 487–496. ACM, New York, NY, USA (2015). https://doi.org/10.1145/2702123.2702409

32. Koelstra, S., Muhl, C., Soleymani, M., Lee, J., Yazdani, A., Ebrahimi, T., Pun, T., Nijholt, A., Patras, I.: Deap: A database for emotion analysis ;using physiological signals. IEEE Transact. Affective Computing 3(1), 18–31 (Jan 2012). https://doi.org/10.1109/T-AFFC.2011.15

33. Laquai, F., Chowanetz, F., Rigoll, G.: A large-scale led array to support anticipatory driving. In: Proc. SMC '11. pp. 2087–2092 (Oct 2011). https://doi.org/10.1109/ICSMC.2011.6083980

34. Lin, C.T., Lin, K.L., Ko, L.W., Liang, S.F., Kuo, B.C., Chung, I.F.: Nonparametric single-trial eeg feature extraction and classification of driver's cognitive responses. EURASIP Journal on Advances in Signal Processing 2008(1), 849040 (2008)

35. Löcken, A., Heuten, W., Boll, S.: Enlightening drivers: A survey on in-vehicle light displays. In: Proc. AutoUI '16. pp. 97–104. ACM, New York, NY, USA (2016). https://doi.org/10.1145/3003715.3005416

36. Löcken, A., Müller, H., Heuten, W., Boll, S.: "should i stay or should i go?": Different designs to support drivers’ decision making. In: Proc. NordiCHI ’14. pp. 1031–1034. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2639189.2670268

37. Loehmann, S., Landau, M., Koerber, M., Butz, A.: Heartbeat: Experience the pulse of an electric vehicle. In: Proc. AutoUI '14. pp. 19:1–19:10. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2667317.2667331

38. Lopatovska, I., Arapakis, I.: Theories, methods and current research on emotions in library and information science, information retrieval and human-computer interaction. Information Processing & Management 47(4), 575 – 592 (2011). https://doi.org/10.1016/j.ipm.2010.09.001

39. Matthews, G., Dorn, L., Hoyes, T.W., Davies, D.R., Glendon, A.I., Taylor, R.G.: Driver stress and performance on a driving simulator. Human Factors 40(1), 136–149 (1998). https://doi.org/10.1518/001872098779480569, pMID: 9579108
40. Miller, D., Sun, A., Johns, M., Ive, H., Sirkin, D., Aich, S., Ju, W.: Distraction becomes engagement in automated driving. Proc. HFE Annual Meeting 59(1), 1676–1680 (2015). https://doi.org/10.1177/1541931215591362

41. Nass, C., Jonsson, I.M., Harris, H., Reaves, B., Endo, J., Brave, S., Takayama, L.: Improving automotive safety by pairing driver emotion and car voice emotion. In: CHI '05 EA. pp. 1973–1976. ACM, New York, NY, USA (2005). https://doi.org/10.1145/1056808.1057070

42. National Highway Traffic Safety Administration: Visual-manual nhtsa driver distraction guidelines for in-vehicle electronic devices. Washington, DC: National Highway Traffic Safety Administration (NHTSA), Department of Transportation (DOT) (2012), https://www.nhtsa.gov/staticfiles/rulemaking/pdf/Distraction\_NPFG-02162012.pdf

43. Neyens, D.M., Boyle, L.N.: The influence of driver distraction on the severity of injuries sustained by teenage drivers and their passengers. Accident Analysis & Prevention 40(1), 254–259 (2008)

44. Picard, R.W.: Affective computing: challenges. International Journal of Human-Computer Studies 59(1), 55 – 64 (2003). https://doi.org/10.1016/S1071-5819(03)00052-1, applications of Affective Computing in Human-Computer Interaction

45. Picard, R.W., Healey, J.: Affective wearables. In: Wearable Computers, 1997. Digest of Papers., First International Symposium on. pp. 90–97. IEEE (1997)

46. Rakauskas, M.E., Gugerty, L.J., Ward, N.J.: Effects of naturalistic cell phone conversations on driving performance. Journal of Safety Research 35(4), 453 – 464 (2004). https://doi.org/10.1016/j.jsr.2004.06.003

47. Rebolledo-Mendez, G., Reyes, A., Paszkowicz, S., Domingo, M.C., Skrypchuk, L.: Developing a body sensor network to detect emotions during driving. IEEE Transact. ITS 15(4), 1850–1854 (Aug 2014). https://doi.org/10.1109/TITS.2014.2335151

48. Riener, A., Ferscha, A.: Driver activity recognition from sitting postures. In: Mensch & Computer Workshopband. pp. 55–62 (2007)

49. Riener, A., Jeon, M., Alvarez, I., Frison, A.K.: Driver in the loop: Best practices in automotive sensing and feedback mechanisms. In: Automotive User Interfaces, pp. 295–323. Springer (2017)

50. Roberts, S.C., Ghazizadeh, M., Lee, J.D.: Warn me now or inform me later: Drivers’ acceptance of real-time and post-drive distraction mitigation systems. International Journal of Human-Computer Studies 70(12), 967 – 979 (2012). https://doi.org/10.1016/j.ijhcs.2012.08.002

51. Rösler, D., Krems, J.F., Mahlke, S., Thüring, M., Seifert, K.: Evaluation of night vision enhancement systems: Driver needs and acceptance. Fahrerass.-syst. (2006)

52. Russell, J.: A circumplex model of affect. Journal of personality and social psychology 39(6), 1161–1178 (1980)

53. Schmidt, E., Decke, R., Rasshofer, R.H.: Correlation between subjective driver state measures and psychophysiological and vehicular data in simulated driving. IEEE IV '16 pp. 1380–1385 (2016). https://doi.org/10.1109/IVS.2016.7535570

54. Schneegass, S., Pfleging, B., Broy, N., Heinrich, F., Schmidt, A.: A data set of real world driving to assess driver workload. In: Proc. AutoUI '13. pp. 150–157. ACM, New York, NY, USA (2013). https://doi.org/10.1145/2516540.2516561

55. Singh, R.R., Conjeti, S., Banerjee, R.: A comparative evaluation of neural network classifiers for stress level analysis of automotive drivers using physiological signals. Biomedical Signal Processing and Control 8(6), 740 – 754 (2013). https://doi.org/10.1016/j.bspc.2013.06.014
56. Solovey, E.T., Zec, M., Garcia Perez, E.A., Reimer, B., Mehler, B.: Classifying driver workload using physiological and driving performance data: Two field studies. In: Proc. CHI '14. pp. 4057–4066. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2556288.2557068

57. Tan, D., Nijholt, A.: Brain-computer interfaces and human-computer interaction. In: Brain-Computer Interfaces, pp. 3–19. Springer (2010)

58. Verster, J.C., Roth, T.: Standard operation procedures for conducting the on-the-road driving test, and measurement of the standard deviation of lateral position (sdlp). International Journal of General Medicine p. 359 (2011). https://doi.org/10.2147/ijgm.s19639

59. Vhaduri, S., Ali, A., Sharmin, M., Hovsepian, K., Kumar, S.: Estimating drivers' stress from gps traces. In: Proc. AutoUI '14. pp. 20:1–20:8. ACM, New York, NY, USA (2014). https://doi.org/10.1145/2667317.2667335

60. Waard, D.D., Jesserun, M., Steyvers, F.J.J.M., Regatt, P.T., Brookhuis, K.A.: Effect of road layout and road environment on driving performance, drivers’ physiology and road appreciation. Ergonomics 38(7), 1395–1407 (1995). https://doi.org/10.1080/00140139508925197, pMID: 7635129

61. Wells-Parker, E., Ceminsky, J., Hallberg, V., Snow, R.W., Dunaway, G., Guiling, S., Williams, M., Anderson, B.: An exploratory study of the relationship between road rage and crash experience in a representative sample of us drivers. AAP 34(3) (2002). https://doi.org/10.1016/S0001-4575(01)00021-5

62. Yang, J.H., Jeong, H.B.: Validity analysis of vehicle and physiological data for detecting driver drowsiness, distraction, and workload. In: Proc. SMC '15. pp. 1238–1243 (Oct 2015). https://doi.org/10.1109/SMC.2015.221

63. Yerkes, R.M., Dodson, J.D.: The relation of strength of stimulus to rapidity of habit-formation. Journal of Comparative Neurology and Psychology 18(5), 459–482 (1908). https://doi.org/10.1002/cne.920180503

64. Zheng, W.L., Zhu, J.Y., Lu, B.L.: Identifying stable patterns over time for emotion recognition from eeg. IEEE Transact. Affective Computing PP(99), 1–1 (2017). https://doi.org/10.1109/TAFFC.2017.2712143

65. Zhu, Y., Wang, Y., Li, G., Guo, X.: Recognizing and releasing drivers’ negative emotions by using music: Evidence from driver anger. In: Adj Proc. AutoUI '16. pp. 173–178. ACM, NY, USA (2016). https://doi.org/10.1145/3004323.3004344