# The Novel EfficientNet Architecture-Based System and Algorithm to Predict Complex Human Emotions
## 3 Materials and Methods  
### Data Acquisition  
The facial image open data used in the study were grayscale FER2013 of resolution size of 48 \(\times\) 48 pixels. In the experiments, we changed the labels using four classes: angry, happy, neutral, and sad. Figure 1 shows the most distinctive changes in facial expressions among the four classes. For the experiment, we produced bio-signal data for each class by referencing a study [39], which identified six HRV index values that vary with different emotions. In the research, the ECG signals were obtained from 60 subjects aged 20 to 26 with 50% male and 50% female students. Signals from participants were recorded using the RM6240B system. All signals were recorded on the desktop computer. The experiment was conducted in isolation from the operator to allow participants to freely experience their emotions. The purpose of the experiment was to record the ECG signal of subjects in Neutral, Fear, Sadness, Happiness, Anger, and Disgust states. For this, the subjects were shown a video corresponding to these emotions. Each video was 7 min long. All participants were healthy and had no complaints of heart disease. The experiment was recorded in a quiet room, and a 5 min break was taken after each video to increase the emotional accuracy of the ECG signal. To obtain the signal associated with the intended emotion, only the data from the last 5 min of each 7 min video were extracted and used in the analysis. These six HRV index values are calculated from the R-R interval data as shown in Figure 2, which are obtained by measuring and pre-processing the ECG. First, it is necessary to f  
Figure 1: FER2013 dataset picture examples. the R peak is the most important feature for ECG data analysis. Accurate detection of R peaks and R-R intervals is critical in diagnosing cardiac conditions. In the time domain, we computed two key metrics: SDNN, which is the standard deviation of normal-to-normal R-R intervals, and RMSSD, the square root of the mean of the squared differences between successive R-R intervals. In the frequency domain analysis, we utilized the normalized low-frequency power (LF), high-frequency power (HF), and their ratio (LF/HF). Additionally, we employed sample entropy (SampEn), a non-linear index, to analyze the complexity of HRV and the time series. In Figure 3, the ECG signals for each emotion, including angry, happy, sad, and neutral, are illustrated. The most commonly used non-linear method for analyzing HRV is the Poincare plot. Figure 4 illustrates the differences in emotions and the variations in successive R-R intervals. The Poincare graph is also a very useful tool for identifying ectopic beats. Table 1 presents the values of the bio-signal data we used. Since we accessed the data from an existing paper, which had categorized them according to emotion class, we assumed that the facial expression images in the FER2013 image data were included in the range of the emotion class data. \begin{table}
\begin{tabular}{c c c c c c} \hline \hline  & & & & & & \\ \hline
**Index** & & **Neutral** & **Sad** & **Happy** & **Angry** \\ \hline SDNN & Mean \(\pm\) SD & 50 \(\pm\) 18 & 47 \(\pm\) 16 & 55 \(\pm\) 19 & 51 \(\pm\) 18 \\ \hline RMSSD & Mean \(\pm\) SD & 41 \(\pm\) 19 & 40 \(\pm\) 18 & 43 \(\pm\) 18 & 44 \(\pm\) 21 \\ \hline LF & Mean \(\pm\) SD & 0.54 \(\pm\) 0.17 & 0.54 \(\pm\) 0.16 & 0.66 \(\pm\) 0.15 & 0.53 \(\pm\) 0.18 \\ \hline HF & Mean \(\pm\) SD & 0.46 \(\pm\) 0.17 & 0.46 \(\pm\) 0.16 & 0.34 \(\pm\) 0.15 & 0.47 \(\pm\) 0.18 \\ \hline LF/HF & Mean \(\pm\) SD & 1.6 \(\pm\) 1.2 & 1.5 \(\pm\) 1.0 & 2.9 \(\pm\) 3.6 & 1.5 \(\pm\) 1.1 \\ \hline SampEn & Mean \(\pm\) SD & 1.78 \(\pm\) 0.25 & 1.90 \(\pm\) 0.262 & 1.90 \(\pm\) 0.28 & 1.97 \(\pm\) 0.24 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Example of bio-signal data used in training. Figure 3: ECG signal visualization. Figure 2: ECG R peak. In the case of facial expression image data, we used the FER2013 open dataset.