ThermalDrive - Towards Situation Awareness over Thermal
Feedback in Automated Driving Scenarios
Xiaru Meng

Jiawen Han

mengxiaru@kmd.keio.ac.jp
Keio University
Yokohama, Japan

George Chernyshov

Keio University
Yokohama, Japanhanjiawen@keio.jp

Kirill Ragozin

Keio University
Yokohama, Japan

Kai Kunze

Keio University
Yokohama, Japan

Keio University
Yokohama, Japan
kai@kmd.keio.ac.jp

Ther
mal
Dr
i
ve

(
1)

(
3)

(
2)

(
4)

Figure 1: (1) Concept of ThermalDrive (heat/cold feedback on the face). (2) Experimental Setup: User with the ThermalDrive
VR headset in a simulated driving environment. (3) The ThermalDrive Headset Prototype: A VR headset with 6 peltier elements (showing hot and cold actuation on a thermal camera. (4) The virtual environment overlaid with the heat information,
obstacles and other cars are encoded over cold/hot feedback accordingly.

ABSTRACTKEYWORDS

We present ThermalDrive, a thermal interface that provided situational awareness information using thermal feedback on theface of the driver. A prototype is built to simulate the autonomous
driving and the thermal interface in virtual reality. We conduct
an experiment to investigate the impact of displaying system situation awareness information via the thermal feedback in a VR
driving simulation (16 participants). The initial results indicate thatthe thermal interface might be a suitable feedback mechanism toconvey some information in autonomous driving. In particular,
cold thermal feedback was effective in terms of notability and user
preference.

thermal Interface, human-vehicle interaction, highly automateddriving, trust

CCS CONCEPTS• Human-centered computing → Haptic devices.
Permission to make digital or hard copies of part or all of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for third-party components of this work must be honored.
For all other uses, contact the owner/author(s).
IUI ’22 Companion, March 22–25, 2022, Helsinki, Finland© 2022 Copyright held by the owner/author(s).
ACM ISBN 978-1-4503-9145-0/22/03.
https://doi.org/10.1145/3490100.3516453

101

ACM Reference Format:
Xiaru Meng, Jiawen Han, George Chernyshov, Kirill Ragozin, and Kai Kunze.
2022. ThermalDrive - Towards Situation Awareness over Thermal Feedbackin Automated Driving Scenarios. In 27th International Conference on Intelligent User Interfaces (IUI ’22 Companion), March 22–25, 2022, Helsinki, Finland.
ACM, New York, NY, USA, 4 pages. https://doi.org/10.1145/3490100.3516453

1

INTRODUCTION

As steering a vehicle is a highly demanding task, there are a lotof research efforts suggesting and evaluating different feedback
methods [4, 15, 20]. We are also witnessing the transformationof the automotive industry from building cars towards building
partially self driving autonomous systems [1, 19]. This change requires developing new interaction paradigms and driver/passenger
interfaces [5, 6].
In driving scenarios where visual and auditory interactions are
the main focus, haptic feedback has been introduced by some researchers to further improve human-vehicle interaction [4, 12, 15].
Kohei Sonoda and Takahiro Wada built a tactile display that contributes to driver trust by providing spatial information related

IUI ’22 Companion, March 22–25, 2022, Helsinki, Finland

Xiaru Meng, et al.

Scene A

System1

Scene B

System2

Figure 2: Schematic diagram of the driving route of the autonomous vehicle in scene A and scene B.
to traffic objects [16]. Vibrotactile feedback in the context of haptic feedback has been much studied in autonomous driving situations. Researchers have provided vibration feedback in various
vehicles through wearable devices, driver seats, and steering wheels[8, 10, 17]. Multiple vibration patterns have been designed andtested to respond to a variety of specific messages [3, 18]. There isalso a lot of thermal feedback research in hci[2, 11], yet usually notapplied to the driving context with a few exceptions[4, 13].
In our research, we are particularly interested communicating
traffic situations in (semi-)autonomous driving, including transition
periods between driver and auto pilot. To this end we present
an initial study evaluating a novel feedback mechanism, thermal
feedback on the face, to present traffic situations.
The contributions of this paper are as follows: (1) we present theconcept of ThermalDrive, encoding the traffic situation (obstacles,cars) in thermal feedback to the face, raising the driver’s awarenessin regard to status of the autonomous vehicle. (2) We present an
initial prototype and a VR simulation, using a headset equippedwith 6 peltier elements. (3) We show initial insights from a user test
(n= 14), evaluating the prototype, perceived trust etc.
Overall, we believe facial thermal feedback is an interesting
modality for semi-autonomous driving and similar application scenarios.

2

APPROACH AND CONCEPT

There is a multitude of autonomous driving feedback mechanisms [14]. We believe that haptic, especially thermal, modalities
are underused and not so well explored yet. Using haptics, weavoid overstimulating the limited visual and auditory perception
channels.
For in car interactions, thermal feedback on the face is not explored so far. To this end we present the ThermalDrive concept,
an in-vehicle thermal interface improving the driver’s situationalawareness of the system’s selected actions. The interface provides
spatial information related to traffic situation through thermal stimulation. Although our first prototype is integrated in a headset,
we envision the future interface to be touch-less (see the concept
picture in Fig. 1 ).
In driving scenarios, an autonomous driving system operates
with a process that detects the traffic environment, makes decisions, and executes actions, such as controlling the steering wheel,
acceleration or brake pedals at the appropriate time. Without a an

102

appropriate interface, it is not easy for human drivers to know thenext action of the autonomous driving system before it executesa specific action. For the driver, the lack of information on the future behavior of the automated system makes it difficult to trust
the system and leads to anxiety [7]. We believe our concept canincrease situational awareness for the driver and in the long run
can positively impact trust of the driver in the system.

3

INITIAL USER-TESTSIn this study a virtual driving simulator was used, where thermal
elements were attached to a VR headset to simulate the non-solid
thermal interface, providing participants with thermal feedback indifferent pattern designs. The ethics board of Keio Media Design
approved the study design and hardware setup.
Apparatus: Peltier elements (attached to a heatsink) were
mounted into the removable foam face interface of the Oculus
Quest HMD to set up the thermal interface, as shown in Figure 1.
A total of 6 elements is assembled, each with a size of 15x15mm
and rated at 2A 3.7V max. 6 elements are split into 3 channels thatare controlled with an Allegro A3909 H-bridge chip. The feedback
module is controlled by an esp32 module, acting as a Wi-fi Access
Point (AP) with Oculus connecting to this AP directly and communicating with the thermal device over Wi-Fi. Computer graphics
in VR driving simulations are generated using software (Unity 3D;
Unity Technologies).
In the experimental setup, driving scenarios in which an autonomous vehicle passes over a mountain road with fallen rocks inthe way were considered. The driving automation level was defined
as level 4. There are two simulated automated driving systems
set up in Unity. The self-driving car has different decisions andperformance in different scenes, as shown in Figure 2.
The participants experienced the two automated driving systems
under three conditions. The three conditions are: no thermal feedback, with hot thermal feedback, and with cold thermal feedback.
In hot and cold feedback conditions, the thermal pattern provided by the thermal interface map the location of the obstacle. For
example, if an obstacle appears on the right side of the car, the thermal feedback on the right side of the interface will be activated. thethermal element is activated when it is possible for the participant
to notice the rock. Thermal feedback lasts 3 seconds at a time anddisappears before the autonomous vehicle takes steering action. At

ThermalDrive

IUI ’22 Companion, March 22–25, 2022, Helsinki, Finland

a room temperature of 23.5 degrees Celsius, the thermal element
is usually 27 degrees Celsius, reaching a maximum of 28 degrees
Celsius after heating and a minimum of 26.7 degrees Celsius after
cooling.
Participants. So far, a total of 16 participants (8 male, 8 female,age 22 to 31) have taken part in the experiment. Two of the experimental runs were invalid (software and recording trouble), leaving
us with n =14 (8 male).
Procedure. Participants read and signed the consent form anddata protection policy. Participants are asked to fill in their basic
information (age, gender, driving experience). They were informed
about the experimental setup.
Then the participants began the simulated autonomous driving
experience. The order of the scenes was counterbalanced. Each
scene lasted 30 seconds. When participants experience the virtual
driving scenario under hot thermal feedback or cold thermal feedback conditions, each time the self-driving car approaches a rockin the way, the thermal elements at the corresponding location on
the VR headset will be passively activated and the participant’s
face will feel the thermal stimulus. The participant was equippedwith only the VR headset and could interact with the virtual environment while sitting in the seat. No other means of interaction
were provided.Every time participants completed an experience
with a virtual scene, they filled out a twelve-item questionnairedeveloped by Jian et al. to evaluate the automated system in thisscene on a Likert scale of 1 to 7 [9]. Participants were required togo through a total of six different virtual driving experiences andcomplete six questionnaires. After all virtual autonomous driving
experiences were completed, participants were asked to share theiroverall opinions and comments on the entire experience.
80

ACKNOWLEDGMENTS

70

This work is partially supported by JSPS Kakenhi Kiban B Grant
No. 18H03278.

60
50
Scence A

40

Scence B

30

REFERENCES

20
10
0
No feedback

Hot feedback

Cold feedback

Figure 3: Trust values in the automated driving system under 3 experimental conditions: no thermal feedback, hotthermal feedback , and cold thermal feedback.

4

trust is higher for the system with cold thermal feedback. Seven
participants mentioned that hot feedback had a warning effect onthem.
A first statistical analysis of the questionnaire suggests that participants trust the autonomous driving system with cold thermal
feedback more than the autonomous driving system without thermal feedback. It can be said that under the experimental conditions,cold thermal feedback did enhance people’s trust in the autonomous
driving system. Repeated measures factorial analyses of variance(ANOVAs), with thermal feedback (non-thermal feedback, hot thermal feedback and cold thermal feedback) and system type (system1and system2) as the independent variables were used to determine
the statistical significance of the independent variables on the dependent variable (participants’ perceived trust in automated driving
systems as given from the questionnaire [9]). The effect of thermal
feedback on the perceived trust was statistically significant, F(1.612,
20.953) = 10.222, p=0.01. The difference in trust under the influenceof non-thermal feedback and cold thermal feedback is statisticallysignificant (P=0.006) with a mean difference of 10.036 (95percent
confidence interval: 2.846 - 17.225), and the difference in trust under
the influence of hot thermal feedback and cold thermal feedback is
statistically significant (P = 0.033) with a mean difference of 4.857(95percent confidence interval: 0.346 - 9.368). The comparison ofparticipants’ perceived trust values in the automated driving system under the influence of different independent variables in theexperimental conditions is shown in Figure 3. The hot thermal feedback and cold thermal feedback provide the same information, yetwere perceived quite differently from the users. This work shows
an initial exploration of thermal feedback to the face for situationalawareness. We presented an initial prototype and showed resultsfrom a first user simulation.

INITIAL RESULTS, DISCUSSION AND
CONCLUSION

All participants reported that the thermal feedback was perceivable and effective in capturing their attention. Participants could
understand the information conveyed by the thermal feedback andthe correspondence between the thermal feedback pattern and theposition of the rocks. 11 of the 14 participants mentioned theirpreference for cold feedback. Interestingly, one participant mentioned in the interview that he thought hot feedback was better, but
the results of the questionnaire analysis showed that his perceived

103[1] Jackie Ayoub, Feng Zhou, Shan Bao, and X. Jessie Yang. 2019. From Manual
Driving to Automated Driving: A Review of 10 Years of AutoUI. In Proceedings ofthe 11th International Conference on Automotive User Interfaces and Interactive
Vehicular Applications (Utrecht, Netherlands) (AutomotiveUI ’19). Association
for Computing Machinery, New York, NY, USA, 70–90. https://doi.org/10.1145/
3342197.3344529[2] Jas Brooks, Steven Nagels, and Pedro Lopes. 2020. Trigeminal-Based Temperature
Illusions. Association for Computing Machinery, New York, NY, USA, 1–12.
https://doi-org.kras1.lib.keio.ac.jp/10.1145/3313831.3376806[3] Marine Capallera, Peïo Barbé-Labarthe, Leonardo Angelini, Omar Abou Khaled,and Elena Mugellini. 2019. Convey Situation Awareness in Conditionally Automated Driving with a Haptic Seat. In Proceedings of the 11th International
Conference on Automotive User Interfaces and Interactive Vehicular Applications:
Adjunct Proceedings (Utrecht, Netherlands) (AutomotiveUI ’19). Association for
Computing Machinery, New York, NY, USA, 161–165. https://doi.org/10.1145/
3349263.3351309[4] Patrizia Di Campli San Vito, Gözel Shakeri, Stephen Brewster, Frank Pollick,
Edward Brown, Lee Skrypchuk, and Alexandros Mouzakitis. 2019. Haptic Navigation Cues on the Steering Wheel. Association for Computing Machinery, New York,NY, USA, 1–11. https://doi-org.kras1.lib.keio.ac.jp/10.1145/3290605.3300440[5] Anna-Katharina Frison, Philipp Wintersberger, Tianjia Liu, and Andreas Riener.
2019. Why Do You like to Drive Automated? A Context-Dependent Analysisof Highly Automated Driving to Elaborate Requirements for Intelligent User
Interfaces. In Proceedings of the 24th International Conference on Intelligent User
Interfaces (Marina del Ray, California) (IUI ’19). Association for Computing Machinery, New York, NY, USA, 528–537. https://doi.org/10.1145/3301275.3302331

IUI ’22 Companion, March 22–25, 2022, Helsinki, Finland

Xiaru Meng, et al.

[6] Renate Häuslschmid, Max von Bülow, Bastian Pfleging, and Andreas Butz. 2017.
SupportingTrust in Autonomous Driving. In Proceedings of the 22nd International
Conference on Intelligent User Interfaces (Limassol, Cyprus) (IUI ’17). Association
for Computing Machinery, New York, NY, USA, 319–329. https://doi.org/10.
1145/3025171.3025198[7] John D Lee and Katrina A See. 2004. Trust in automation: Designing for appropriate reliance. Human factors 46, 1 (2004), 50–80.
[8] Zhuoluo Ma, Yue Liu, Dejiang Ye, and Lu Zhao. 2019. Vibrotactile Wristband for
Warning and Guiding in Automated Vehicles. In Extended Abstracts of the 2019
CHI Conference on Human Factors in Computing Systems (Glasgow, Scotland Uk)(CHI EA ’19). Association for Computing Machinery, New York, NY, USA, 1–6.
https://doi.org/10.1145/3290607.3312819[9] Maria Madsen and Shirley Gregor. 2000. Measuring human-computer trust.
In 11th australasian conference on information systems, Vol. 53. Emerald Group
Publishing Limited, Brisbane, Australia, 6–8.
[10] Sunyoung Oh and Jeha Ryu. 2013. Preliminary evaluation of multi-vibration
haptic feedback on a steering wheel spinner. In 2013 13th International Conference
on Control, Automation and Systems (ICCAS 2013). IEEE, Gwangju, Korea, 1079–
1082.
[11] Roshan Lalintha Peiris, Wei Peng, Zikun Chen, Liwei Chan, and Kouta Minamizawa. 2017. ThermoVR: Exploring Integrated Thermal Haptic Feedback
with Head Mounted Displays. In Proceedings of the 2017 CHI Conference on
Human Factors in Computing Systems (Denver, Colorado, USA) (CHI ’17). Association for Computing Machinery, New York, NY, USA, 5452–5456. https://doi.org/10.1145/3025453.3025824[12] Matthew J. Pitts, Gary E. Burnett, Mark A. Williams, and Tom Wellings. 2010.
Does Haptic Feedback Change the Way We View Touchscreens in Cars?. In
International Conference on Multimodal Interfaces and the Workshop on Machine
Learning for Multimodal Interaction (Beijing, China) (ICMI-MLMI ’10). Association
for Computing Machinery, New York, NY, USA, Article 38, 4 pages. https://doi.org/10.1145/1891903.1891952[13] Patrizia Di Campli San Vito, Stephen Brewster, Frank Pollick, Stuart White, Lee
Skrypchuk, and Alexandros Mouzakitis. 2019. Thermal Feedback for Simulated

104

Lane Change Scenarios. International Journal of Mobile Human Computer Interaction (IJMHCI) 11, 2 (2019), 39–57.
[14] Tobias Schneider, Sabiha Ghellal, Steve Love, and Ansgar R.S. Gerlicher. 2021.
Increasing the User Experience in Autonomous Driving through Different Feedback
Modalities. Association for Computing Machinery, New York, NY, USA, 7–10.
https://doi.org/10.1145/3397481.3450687[15] Gözel Shakeri, John H. Williamson, and Stephen Brewster. 2018. May the Force
Be with You: Ultrasound Haptic Feedback for Mid-Air Gesture Interaction in Cars.
In Proceedings of the 10th International Conference on Automotive User Interfaces
and Interactive Vehicular Applications (Toronto, ON, Canada) (AutomotiveUI ’18).
Association for Computing Machinery, New York, NY, USA, 1–10. https://doi.
org/10.1145/3239060.3239081[16] Kohei Sonoda and Takahiro Wada. 2017. Displaying system situation awarenessincreases driver trust in automated driving. IEEE Transactions on Intelligent
Vehicles 2, 3 (2017), 185–193.
[17] Ariel Telpaz, Brian Rhindress, Ido Zelman, and Omer Tsimhoni. 2015. Haptic
Seat for Automated Driving: Preparing the Driver to Take Control Effectively. In
Proceedings of the 7th International Conference on Automotive User Interfaces andInteractive Vehicular Applications (Nottingham, United Kingdom) (AutomotiveUI’15). Association for Computing Machinery, New York, NY, USA, 23–30. https://doi.org/10.1145/2799250.2799267[18] Jingyan Wan and Changxu Wu. 2018. The effects of vibration patterns of takeover request and non-driving tasks on taking-over control of automated vehicles.
International Journal of Human–Computer Interaction 34, 11 (2018), 987–998.
[19] Daniel Watzenig and Martin Horn. 2016. Automated driving: safer and more
efficient future driving. Springer, Cham, Switzerland.
[20] Philipp Wintersberger, Dmitrijs Dmitrenko, Clemens Schartmüller, AnnaKatharina Frison, Emanuela Maggioni, Marianna Obrist, and Andreas Riener.
2019. S(C)ENTINEL: Monitoring Automated Vehicles with Olfactory Reliability
Displays. In Proceedings of the 24th International Conference on Intelligent User
Interfaces (Marina del Ray, California) (IUI ’19). Association for Computing Machinery, New York, NY, USA, 538–546. https://doi.org/10.1145/3301275.3302332