# A Multimodal Perception Framework for Users Emotional State Assessment in Social Robotics
## 3 Experiments
### Testing the IDTm Software  
This experiment is focused on the capability of the IDTm to keep track of the users while monitoring their psycho-physiological state. With this purpose, two are the main parameters to be investigated: the _Pixels Standard Deviation_ of the image captured by the camera and the _Subjects Position_, in terms of relative distance between the subjects and the vision sensor.  
To stress the acquisition system, we selected the following scenario: two persons, a male and a female, distinguished by two different QR codes applied on their clothes, are at a similar initial distance from the robot. The male will approach the robot and place his fingers on the TMP contacts for the physio data extraction, which will last the minimum time necessary to let the TMP acquire reliable IBI and EDA signals (60 s). During the gathering of the physiological data he will occlude the vision sensor, however, the system must be aware of who is touching the patches. Moreover, when the subject moves away from the robot, the perceptual framework has to be able to recognise again and differentiate the two subjects with their previous IDs.  
In Figure 10 we report the results of this experiment. The upper chart shows the trend of the pixels standard deviation of a cropped central portion of the image captured by the Kinect (green line). The minimum threshold at 50 has been highlighted with a straight dotted purple line. In the lower chart, it is possible to see the relative distances between each of the two subjects and the vision sensor. They are a 26 years old female having a QR code correspondent to ID 1 (red line in the line chart) and a 27 years old male, whose ID is 2 (blue line).  
Some events are labelled in both charts. For example, at the beginning, and for the first 15 s, the robot has the possibility to see and analyse the social scene in which is involved, the fact that the camera is not occluded is given by the high standard deviation of the p  
Figure 10: IDTm (ID Tracking Module) Validation Test; (**a**) Pixels Standard Deviation of the elaborated image; (**b**) position of the subjects in terms distance (m).  
variable image captured by the vision system. In the scene, both subjects are detected: subject 1, whose QR code is recognised from the beginning, stands at 1.6 m from the robot; subject 2 is recognised at 5.3 s and stands at 1.5 m from the robot. After a few seconds, subject 2 approaches the FACE robot and his detected distance decreases. At \(t=11\) s subject 2 is so close to the robot that covers the view of subject 1, who is lost by the vision system (\(d=0\)). After only one second, even subject 2, who is approaching, exceeds the minimum distance needed by the SA to detect and recognise subjects, and gets lost by the vision system. At \(t=15\) s, subject 2 has reached the electronic patches of the TMP occluding the camera enough to let the pixel standard deviation goes under its minimum threshold. Here starts the 60 s acquisition of physiological parameters (from \(t=15\) s to \(t=75\) s). Finally, subject 2 removes his hands from the robot's shoulders and moves backwards.  
During the experiment, we noticed two fundamental results: (1) although the two subjects got lost by the SA at a distance of one second only, the IDTm managed to successfully estimate who was touching the TMP patches, as the last closest subject's ID that has been detected. Indeed, during the physio data acquisition, the IDTm variable \(id_{touch}\) was set at 2, and this was correctly the ID to which the gathered signals had to be assigned; (2) only a few seconds after the physio data acquisition, thanks to the increase of the pixels' standard deviation over the threshold, the system realised that the camera was no longer occluded (occluded \(=N\)) and no one was touching the patches (\(id_{touch}=NaN\)), but what is more important is that, when the scene came back clear to the vision system, QRCm managed to recognise again both subjects assigning the two correct IDs.