## 8 Thermal IR Imaging-Based Affective Computing in HRI
used a mobile social robot called N-MARIA (New-Mobile Autonomous Robot for Interaction with Autistics), which was built in UFES/Brazil to assist children during social relationship rehabilitation [86]. In the interaction with the child, which lasted two minutes, the child was encouraged to make communication and tactile interaction with the robot. The robot was equipped with low-cost hardware (thermal and visible camera) used to give information about the emotional state of the child, which was related to five emotions (i.e., surprise, fear, disgust, happiness, and sadness). Results showed that the system was able to recognize those emotions, achieving 85.75% accuracy. Such accuracy was comparable with gold standard techniques in emotion recognition, such as facial expression analysis and speech tone analysis [102; 103; 104; 105; 106]. [PERSON] et al. used the social robot \"Mio Amico\" robot, produced by ()Liscianigiochi, in which the mobile thermal system FLIR ONE (which includes a thermal and a visible camera) was installed on the head of the robot [87]. The study aimed to endow the robot with the capability of real-time assessment of the interlocutor's state of engagement (positive, neutral, and negative emotional engagement). During the interaction between the robot and the child, the robot could either tell a fairy tale or sing a song. At the end of the fairy tale or the song, the robot asked the child if he/she liked the fairy tale/song and if he wanted to listen to another one. Based on the child's answer, the robot could choose the next action. The engagement state of the infant was classified by analyzing the child's thermal modulation using a low computational processing pipeline. Figure 5 summarizes in a-c the processing pipeline for the interlocutor's state of engagement identification and (d) the child-robot interaction. The accuracy reached was 70%. Although this study presented a lower level of accuracy compared to [PERSON] et al. [86], it is worth mentioning that the estimation of the level of engagement can be considered a hard task, since it represents a complex emotion (combination of the basic ones), and it has been poorly investigated in the literature. Besides, the study reported in [PERSON] et al. [87] represents the first and unique study in which the robot could actually change its activities based on the child affective state, opening the way to a bidirectional interaction. Thermal IR Imaging-Based Affective Computing in Intelligent Systems Such as Driver-Assistance Systems or Autonomous Vehicles  
Besides humanoid robots, systems such as driver-assistance systems or autonomous vehicles can also be classified as \"robots\" due to their intelligent behaviors. Indeed, driving functions are becoming increasingly automated; consequently, motorists could run the risk of being cognitively removed from the driving process. Thermal IR imaging was demonstrated to be a valid technique for assessing variation in cognitive load [52; 107; 108]. Most of all, it could enable sensing the real-time state of motorists non-invasively i.e., without disrupting driving-related tasks and, unlike RGB cameras, independently from external light conditions [109]. Studies of thermal IR imaging in driving contexts stated that a rise in mental workload leads to an increase in the difference between nose and forehead temperature [108; 110]. Moreover, thermal IR imaging was demonstrated to be a valuable indicator of the driver arousal level, from alertness to drowsiness [64; 111]. A recent study employed it to detect human thermal discomfort in order to develop a fully automated climate control system in the vehicles [112]. Even in the view of automated vehicles, in complex situations, they will need human  
Figure 5: **(A–C)** Processing pipeline. **(A)** The first phase relied on the visible image to detect the child’s face and locate specific facial landmarks. **(B)** The corresponding landmarks on the thermal image were then obtained thanks to a previous optical calibration procedure.