###### Abstract  
This study aimed to classify emotion based on facial expression and physiological signals using deep learning and to compare the analyzed results. We asked 53 subjects to make facial expressions, expressing four types of emotion. Next, the emotion-inducing video was watched for 1 min, and the physiological signals were obtained. We defined four emotions as positive and negative emotions and designed three types of deep-learning models that can classify emotions. Each model used facial expressions and physiological signals as inputs, and a model in which these two types of input were applied simultaneously was also constructed. The accuracy of the model was 81.54% when physiological signals were used, 99.9% when facial expressions were used, and 86.2% when both were used. Constructing a deep-learning model with only facial expressions showed good performance. The results of this study confirm that the best approach for classifying emotion is using only facial expressions rather than data from multiple inputs. However, this is an opinion presented only in terms of accuracy without considering the computational cost, and it is suggested that physiological signals and multiple inputs be used according to the situation and research purpose. emotion classification; deep learning; physiological signal; facial expression 2022122022021220220220220220220220222022202  
of valence, arousal, or intensity of emotion. In general, the two-dimensional circumplex model of [PERSON] [10] and the three-dimensional PAD emotional state model [11] are used. There are many methods for measuring human emotions. A survey-based measurement is basically used to figure out the emotion. However, since the results of emotions cause physiological reactions in the human body, a hormone-based measurement as well as measurements using a non-invasive method are used. A diagnosis by a psychiat is necessary when medical care is required based on the results of the measurement [12]. Representatively, methods such as the Cornell Medical Index (CMI) [13], saliva and urinalysis cortisol test for physiological responses, and hormone-level test through a blood test were generally used. Each measurement method has advantages and disadvantages. For the method using a questionnaire, the error in measurement accuracy cannot be ignored as the answer varies depending on the subjectivity. The hormone-measurement method has the disadvantage that it cannot grasp the status in real time [14]. In contrast, the non-invasive measurement method can complement real-time performance and is considered to be fairly reliable because it is based on physiological signals that the subject cannot change artificially [3]. Conventionally, research on emotion classification using physiological signals and facial expressions has been performed based on simple computational calculations. As these studies gradually develop, methods using machine learning and pattern recognition have been proposed, and more recently, studies to classify emotions using deep learning with computer hardware performance have been proposed. This type of research is widely used not only in emotion recognition [3,15,16], posture recognition [17,18] and medical diagnosis and therapy [19,20], but also in finance [21] and even agriculture [22]. Studies have shown that physiological signals and emotion are closely related. In the study of [PERSON] [23], emotion and the autonomic nervous system (ANS) response specificity were reviewed, and [PERSON] [24] presented how emotions and cognitive processes are physiologically affected. In addition, a study by [PERSON] [25], who conducted four different experiments to confirm the correlation between emotion-based facial expressions and physiological signals, was also conducted. Negative emotions can be expressed as stress and anxiety and are closely related. Stress can be recognized for its cause, but anxiety often involves not recognizing what the object of anxiety is.