# Review of Studies on Emotion Recognition and Judgment Based on Physiological Signals
## 4 Research on and Application of Multimodal Physiological Signals
### Two Psychological Signals  
[PERSON] et al. [58] presented an approach to detect human emotions according to heart beat and facial expressions. Through game experiments using their presented approach, they found that it needed less computing time and had high accuracy in detecting the emotions of excitement, sadness, anger and calmness. The intensity of emotion could be evaluated with the values of heart beat. [PERSON] et al. [59] conducted experiments of emotion classification for different types of physiological signals and combinations of EEG, and the results showed that skin electrical signals and facial EMG signals yielded better classification results. [PERSON] et al. [60] built a system of emotion judgment and a system of emotion visualization based on changes in pictures and scenes, and compared the change in emotional trends detected with systems based on sets of optimal signal features and machine learning-based methods through skin electricity and pulse waves of 20 subjects who were listening to music. The results indicated that the change in emotional trends detected using the system of emotion visualization based on changes in picture and scene were consistent with those detected using the system of emotion judgment. Meanwhile, the interactive quality and real-time ability of the emotion visualization system based on scene changes were better than those based on picture changes; the interactive quality of the emotion judgment based on a set of optimal signal features was better than that based on a machine learning-based method.  
[PERSON] et al. [61] first presented a technique according to a fusion model of the Gaussian mixed model with the Butterworth and Chebyshev signal filter, and calculated the features extracted from EEG signals and eye-tracking signals, then used a max-min approach for vector normalization, and finally proposed a deep gradient neural network for classifying EEG and eye-tracking signals. The experimental results showed that the presented neural network could predict emotions with 88.10% accuracy, and the presented approach also performed with high efficiency in several indices compared with other typical neural networks. [PERSON] et al. [62] used ECG and EEG signals to assess the impact of music on the generation of ideas by subjects with design experience under individual and group conditions. The results indicated that the subjects produced more but worse design ideas in response to music that aroused positive emotions, and pleasant emotions were more likely to be evoked under group conditions. At the initial stage of design, the positive music was beneficial for producing more design ideas, but non-music was better when designers needed to think deeply. [PERSON] et al. [63] put forward a neural network driven by a solution to learning driving-induced stress modes, related it with time-frequency changes observed from galvanic skin response and photoplethysmography, and obtained precision of 89.23%, specificity of 94.92% and sensitivity of 88.83%. They summarized that the layer recurrent neural network was most suitable for the detection of stress degree. [PERSON] and [PERSON] [64] built a multimodal database of EEG and ECG signals obtained during emotional arousal through audio-visual stimuli from 23 subjects along with the subjects' self-evaluations of their respective emotional states based on valence, arousal and dominance, and established a baseline for subject-wise emotion identification using the features extracted from EEG and ECG signals as well as their fusion. They also assessed the self-evaluations of the subjects by comparing them with other self-evaluations. The results showed that the classification effect for valence, arousal and dominance based on the presented database could be matched with databases obtained using expensive and medical-grade devices, pointing out the prospects of using low-cost equipment for emotion recognition.