# Identification System Based on Resolution Adjusted 2D Spectrogram of Driver’s ECG for Intelligent Vehicle

Authors: Gyu Ho Choi; Kiho Lim; Sung Bum Pan
Journal: Mobile Information Systems
 (2022)

---

## Abstract

Recently, traditional vehicles are being developed into intelligent vehicles as information is exchanged among various devices inside and outside the vehicles. In the connected car environment, the need for vehicle security is growing due to vehicle hacking accidents and possible threats to human life. Driver identification technology using electrocardiogram (ECG) signals has been studied to address vehicle security issues and driver-specific services. Existing driver identification systems tried to address the issues using a multidimensional feature extraction method. However, there are remaining issues, including accuracy concerns, because the resolution was adjusted without considering the ECG’s P, QRS Complexes, and T waves feature when analyzing the time-frequency multidimensional features. In this paper, we propose a driver identification system using a 2D spectrogram. It identifies a section where the resolution is optimally adjusted using a spectrogram that can simultaneously analyze the time-frequency features of an ECG. The experimental results show that the proposed method improved the identification performance compared to the existing multidimensional feature extraction methods such as EEMD and MFCCs. Besides, with a 2D spectrogram of 1/4 image size, the recognition performance is maintained in a CNN network and the training time is significantly reduced.


---

## Body

## 1. Introduction

Recently, smart infotainment systems based on biometrics have been introduced for driver convenience [1, 2]. Smart vehicles are actively researching in-vehicle-infotainment (IVI) or audio-video navigation (AVN) device technologies that can provide driver-centered lifelogging and 3D user interface (UI) services [3, 4]. To provide driver-customized services, an authentication technology that uses the driver’s bioinformation to prove identity with a biometrics system is being studied [5]. If the driver is authenticated inside and outside the vehicle using the driver’s biological information, a customized telematics service is provided to the driver [6]. Verma and Choudhary [7] studied a system in which vehicle doors open when a driver is recognized, and seats, mirrors, air conditioning, and audio are customized to the driver with automatic control. Choi et al. [8] studied a system in which various entertainments are provided to the driver by transmitting a signal that judges the driver’s situation or health condition, stress and drowsiness, etc.Various recognition technologies using the driver’s biological information are being studied to provide driver-customized applications. The technologies learned to apply the recognition technology in the vehicle include facial recognition, gesture action classification, emotion condition classification, voice word classification, and driver’s identification. Shojaeilangari et al. [9] studied the driver-customized music list recommendation system by judging drowsiness and mood by using driver face information through a camera in a vehicle. Dehzangi and Taherisadr [10] acquired the driver’s arm motions and movements as electric skin response (GSR) signals during driving and studied the inattention recognition system. He et al. [11] acquired bioinformation from various sensors in a vehicle and studied the emotional and health status recognition system. Sarala et al. [12] studied an infotainment system that can easily access vehicle entertainment using driver voice signals. Xing et al. [13] studied the driver identification and driving task performance system using biological information for the intelligent vehicle.Among the driver’s recognition technologies, the driver’s identification step is the first required step to provide driver-customized applications. Driver’s identification progress is a crucial step for driver security because it is linked to a driver-customized biometrics system. Xiao [14] described the process of a biometric system that registers an individual’s unique physical and behavioral biological information and recognizes the user. Physical biological features include external information of the body, such as fingerprint and face. Behavioral biological features include external signals of the body, such as voice, gait, handwriting, and internal signals of the body such as an electrocardiogram (ECG), electromyography (EMG), and an electroencephalogram (EEG). Dantcheva et al. [15] explained the problem that biometrics using external body signals had lower accuracy and higher error rate than bioinformation. Nazmy et al. [16] described the forgery incident and problems using silicon fingerprints, although the biometrics using external bioinformation is analyzed with high accuracy. Kim et al. [17] described the case of cloning and hacking using images of the iris of the German defense minister. Accordingly, Choi et al. and Kim et al. [18, 19] studied the user recognition system using the ECG generated as an electrical signal inside the body. Das et al. [20] studied the user recognition system using the EEG generated as an electrical signal inside the body.Among the biological signals generated by electrical signals inside the human body, an ECG acquired without user intervention has individual biological features depending on the heart’s electrophysiological location, size, factors, and condition. The driver identification performance by multidimensional features is analyzed and is found to be more stable than the single-dimensional features as the number of subjects increases. However, it is analyzed and is found to be lower than the identification performance using the single-dimensional features. To solve this problem, existing methods for extracting multidimensional features in the frequency domain have been studied. However, there is a problem of using multidimensional features that do not consider the P, QRS Complexes, and T waves features of the ECG and only considering the network structure.In this paper, we proposed a driver identification system by extracting multidimensional features with optimized time-frequency resolution from 2D spectrograms considering the ECG’s morphological feature. The existing multidimensional feature extraction method converts and extracts the Ensemble Empirical Mode Decomposition (EEMD) and Mel Frequency Cepstral Coefficients (MFCCs) without considering the morphological features of the ECG signal.


## 2. Related Work

### 2.1. Biometrics Using ECG on Vehicles

A driver’s recognition system using an ECG signal determines the driver’s status and provides recognition information through stress, fatigue, cognitive distraction, health, and emotion step classification, as shown in Table1.Table 1
Biometrics system according to ECG acquisition in vehicles.
Electrode positionAcquisition signal, positionFeature analysisClassificationRef.Seat beltECG, chestMorphological featureStress recognition[21]Seat beltECG, chestHRVStress level recognition[22]Seat beltECG, chestR-R intervalFatigue recognition[23]Seat beltECG/EEG, chest/headMultidimensional featureFatigue recognition[24]Seat beltECG/chestR-R intervalCognitive distraction recognition[25, 26]Seat beltECG/chestMultidimensional featureHeart condition recognition[27]Seat beltECG/chest—Emotion recognition[28]Steering wheelECG/wrist—Identification[29]Steering wheelECG/wristFiducial featureIdentification[30]Sihem et al. [21] studied a system for classifying stress steps by acquiring an ECG from the driver’s chest close the belt in a vehicle and morphological feature data. Munla et al. [22] studied a driver’s stress step monitoring system by acquiring an ECG from the driver’s chest close to the belt while driving. Wang et al. [23] studied a system for recognizing driver’s fatigue by analyzing R-R interval information from ECG signals acquired from the chest close to the seat belt in a driving. In biometrics’ vehicles for a driver, a recognition using an ECG signal can identify the driver’s status as stress, drowsiness, cognitive distraction, health, and emotion. Shiwu et al. [24] studied a system that acquires ECG and EEG signals while driving and identifies driver fatigue by analyzing multidimensional feature data. Miyaji et al. [25, 26] studied a warning notification system to prevent preaccidents by identifying driver’s cognitive distraction. The features used in the system were the R-R interval of the ECG signal, the pupil diameter, the gaze angle, and the head rotation angle information. Singh et al. [27] studied a heart condition system for monitoring the driver’s heart condition by acquiring multichannel ECG signals from the driver’s chest close to the belt. Wang et al. [28] studied the driver’s calm and anxious emotion identification system using the ECG signal acquired from the chest close to the seat belt in the driving.In order to apply the driver status recognition system while driving, the driver identification progress should be prioritized. Silva et al. [29] studied the driver identification system using the ECG signal acquired from the driver’s wrist. The driver’s ECG lead-I was acquired from the steering wheel and identified with an error rate of 3 to 5% in a stable status and a 30% error rate in a moving status. Santos et al. [30] studied a driver identification system using ECG signals from stress recognition in automobile drivers’ database at PhysioBank ATM. This system is analyzed with 95% driver identification accuracy by segmenting and feature extraction of fiducial point-based one cycle from the ECG signal. In order to identify a driver with high accuracy in a driving environment, a multidimensional feature extraction method from morphological features is required.


### 2.2. ECG Feature Extraction for Identification

Feature data extraction methods used for user identification using an ECG are factors that directly affect the accuracy improvement, and the morphological features of the ECG were studied in one dimension to improve accuracy [31–33]. A single-dimensional extraction method using morphological features of one period of an ECG, a multidimensional extraction method by dimensional transformation, and a network method are being studied, as shown in Table 2.Table 2
ECG feature extraction method for identification.
Feature domainFeature extractionIdentification accuracy (%)AuthorSingle-dimensionalMorphological95.9Choi et al. [34]Morphological98Shen et al. [35]Morphological100Nobunaga et al. [36]MultidimensionalEEMD95.56Zhao et al. [37]MFCCs80Chen et al. [38]Spectrogram98Chun et al. [39]NetworkCNN99.4Giannakakis et al. [40]CNN + LSTM96Xu et al. [41]The method of extracting features in single dimension proceeds in one cycle of the ECG. Choi et al. [34] extracted PQ, QS, and ST intervals and P, Q, R, S, and T amplitudes as features between the peaks of one cycle of the ECG. The identification performance of 175 subjects using the extracted single-dimensional features was analyzed as 95.9% by the Support Vector Machine (SVM). Shen et al. [35] extracted the features of the amplitude as RQ, RS, RS2, PQ, QS, RP, RT, and TS peaks, duration of QS, QT interval, the area of the QRS Complexes, the slope of RS, ST, and the angle with the y-axis of Q, R, and S peak in one cycle of the ECG. The identification performance of 49 subjects using the extracted single-dimensional features was analyzed as 98% by the Euclidean distance. Nobunaga et al. [36] extracted the amplitude and duration term of the P, Q, R, S, and T waves of the vertices in one cycle of the ECG and the time of the section where the waveform of one cycle intersects the zero point. The identification performance of 10 subjects using the extracted single-dimensional features was analyzed as 100% by K-Nearest Neighbors (KNN).The single-dimensional feature extraction method is analyzed as high identification performance in a low number of subjects but has a disadvantage of analyzing low accuracy as the number of subjects increases. In order to solve this problem, multidimensional feature extraction techniques have been proposed that can be extracted as anM×N matrix feature and analyzed with higher accuracy than a single-dimensional feature. Zhao et al. [37] extracted multidimensional features using EEMD through converting ECG signals for each frequency band. In the PTB database and the long-term ST database by KNN, the identification performance was analyzed as 96% and 95.75%, and it was analyzed as 95.56% when both databases were used. Chen et al. [38] extracted multidimensional features using MFCCs which analyze ECG signals in each Mel Frequency band with the cepstral coefficients of DCT. The identification performance for the PTB database was analyzed as 80% by KNN. Chun et al. [39] extracted multidimensional features using a spectrogram that can analyze data simultaneously in the time and frequency domains. The identification performance for the ECG-ID database was analyzed as 98% by Euclidean distance.Although the identification performance using the multidimensional feature extraction method has the advantage of being maintained compared to the single-dimensional feature even when the number of subjects increases, there is a disadvantage that the identification performance is analyzed by low accuracy. Deep learning-based identification systems have been studied to compensate for the shortcomings analyzed with low accuracy. Giannakakis et al. [40] analyzed 99.4% of the identification performance of 24 subjects using a Convolutional Neural Network (CNN). Xu et al. [41] analyzed 96% of the MIT-BIH database identification performance using a single-dimensional CNN, a Bidirectional Long Short-Term Memory (Bi-LSTM) neural network, and a deep trust neural network. Although feature extraction methods have been studied to improve the identification performance, data analysis is in progress in a complex network without considering the P, QRS Complexes, and T waves features of the ECG. We propose a driver identification system in a single network using a spectrogram image that time and frequency resolutions are optimized in consideration of the morphological features of the ECG.


## 2.1. Biometrics Using ECG on Vehicles

A driver’s recognition system using an ECG signal determines the driver’s status and provides recognition information through stress, fatigue, cognitive distraction, health, and emotion step classification, as shown in Table1.Table 1
Biometrics system according to ECG acquisition in vehicles.
Electrode positionAcquisition signal, positionFeature analysisClassificationRef.Seat beltECG, chestMorphological featureStress recognition[21]Seat beltECG, chestHRVStress level recognition[22]Seat beltECG, chestR-R intervalFatigue recognition[23]Seat beltECG/EEG, chest/headMultidimensional featureFatigue recognition[24]Seat beltECG/chestR-R intervalCognitive distraction recognition[25, 26]Seat beltECG/chestMultidimensional featureHeart condition recognition[27]Seat beltECG/chest—Emotion recognition[28]Steering wheelECG/wrist—Identification[29]Steering wheelECG/wristFiducial featureIdentification[30]Sihem et al. [21] studied a system for classifying stress steps by acquiring an ECG from the driver’s chest close the belt in a vehicle and morphological feature data. Munla et al. [22] studied a driver’s stress step monitoring system by acquiring an ECG from the driver’s chest close to the belt while driving. Wang et al. [23] studied a system for recognizing driver’s fatigue by analyzing R-R interval information from ECG signals acquired from the chest close to the seat belt in a driving. In biometrics’ vehicles for a driver, a recognition using an ECG signal can identify the driver’s status as stress, drowsiness, cognitive distraction, health, and emotion. Shiwu et al. [24] studied a system that acquires ECG and EEG signals while driving and identifies driver fatigue by analyzing multidimensional feature data. Miyaji et al. [25, 26] studied a warning notification system to prevent preaccidents by identifying driver’s cognitive distraction. The features used in the system were the R-R interval of the ECG signal, the pupil diameter, the gaze angle, and the head rotation angle information. Singh et al. [27] studied a heart condition system for monitoring the driver’s heart condition by acquiring multichannel ECG signals from the driver’s chest close to the belt. Wang et al. [28] studied the driver’s calm and anxious emotion identification system using the ECG signal acquired from the chest close to the seat belt in the driving.In order to apply the driver status recognition system while driving, the driver identification progress should be prioritized. Silva et al. [29] studied the driver identification system using the ECG signal acquired from the driver’s wrist. The driver’s ECG lead-I was acquired from the steering wheel and identified with an error rate of 3 to 5% in a stable status and a 30% error rate in a moving status. Santos et al. [30] studied a driver identification system using ECG signals from stress recognition in automobile drivers’ database at PhysioBank ATM. This system is analyzed with 95% driver identification accuracy by segmenting and feature extraction of fiducial point-based one cycle from the ECG signal. In order to identify a driver with high accuracy in a driving environment, a multidimensional feature extraction method from morphological features is required.


## 2.2. ECG Feature Extraction for Identification

Feature data extraction methods used for user identification using an ECG are factors that directly affect the accuracy improvement, and the morphological features of the ECG were studied in one dimension to improve accuracy [31–33]. A single-dimensional extraction method using morphological features of one period of an ECG, a multidimensional extraction method by dimensional transformation, and a network method are being studied, as shown in Table 2.Table 2
ECG feature extraction method for identification.
Feature domainFeature extractionIdentification accuracy (%)AuthorSingle-dimensionalMorphological95.9Choi et al. [34]Morphological98Shen et al. [35]Morphological100Nobunaga et al. [36]MultidimensionalEEMD95.56Zhao et al. [37]MFCCs80Chen et al. [38]Spectrogram98Chun et al. [39]NetworkCNN99.4Giannakakis et al. [40]CNN + LSTM96Xu et al. [41]The method of extracting features in single dimension proceeds in one cycle of the ECG. Choi et al. [34] extracted PQ, QS, and ST intervals and P, Q, R, S, and T amplitudes as features between the peaks of one cycle of the ECG. The identification performance of 175 subjects using the extracted single-dimensional features was analyzed as 95.9% by the Support Vector Machine (SVM). Shen et al. [35] extracted the features of the amplitude as RQ, RS, RS2, PQ, QS, RP, RT, and TS peaks, duration of QS, QT interval, the area of the QRS Complexes, the slope of RS, ST, and the angle with the y-axis of Q, R, and S peak in one cycle of the ECG. The identification performance of 49 subjects using the extracted single-dimensional features was analyzed as 98% by the Euclidean distance. Nobunaga et al. [36] extracted the amplitude and duration term of the P, Q, R, S, and T waves of the vertices in one cycle of the ECG and the time of the section where the waveform of one cycle intersects the zero point. The identification performance of 10 subjects using the extracted single-dimensional features was analyzed as 100% by K-Nearest Neighbors (KNN).The single-dimensional feature extraction method is analyzed as high identification performance in a low number of subjects but has a disadvantage of analyzing low accuracy as the number of subjects increases. In order to solve this problem, multidimensional feature extraction techniques have been proposed that can be extracted as anM×N matrix feature and analyzed with higher accuracy than a single-dimensional feature. Zhao et al. [37] extracted multidimensional features using EEMD through converting ECG signals for each frequency band. In the PTB database and the long-term ST database by KNN, the identification performance was analyzed as 96% and 95.75%, and it was analyzed as 95.56% when both databases were used. Chen et al. [38] extracted multidimensional features using MFCCs which analyze ECG signals in each Mel Frequency band with the cepstral coefficients of DCT. The identification performance for the PTB database was analyzed as 80% by KNN. Chun et al. [39] extracted multidimensional features using a spectrogram that can analyze data simultaneously in the time and frequency domains. The identification performance for the ECG-ID database was analyzed as 98% by Euclidean distance.Although the identification performance using the multidimensional feature extraction method has the advantage of being maintained compared to the single-dimensional feature even when the number of subjects increases, there is a disadvantage that the identification performance is analyzed by low accuracy. Deep learning-based identification systems have been studied to compensate for the shortcomings analyzed with low accuracy. Giannakakis et al. [40] analyzed 99.4% of the identification performance of 24 subjects using a Convolutional Neural Network (CNN). Xu et al. [41] analyzed 96% of the MIT-BIH database identification performance using a single-dimensional CNN, a Bidirectional Long Short-Term Memory (Bi-LSTM) neural network, and a deep trust neural network. Although feature extraction methods have been studied to improve the identification performance, data analysis is in progress in a complex network without considering the P, QRS Complexes, and T waves features of the ECG. We propose a driver identification system in a single network using a spectrogram image that time and frequency resolutions are optimized in consideration of the morphological features of the ECG.


## 3. Driver Identification System Using Resolution Adjusted 2D Spectrogram

The system flow chart for driver identification is shown in Figure1. The identification system process consists of the step of acquiring the ECG signal, the preprocessing step to remove noise, the step of the proposed adjustment of time and frequency resolution in the spectrogram, and the step of identifying the final driver from the CNN after 2D resized.Figure 1
Driver identification system based on resolution adjusted 2D spectrogram.


### 3.1. Identification System Using the 2D Spectrogram Image

In the identification system for an intelligent vehicle, the registration and test data using the driver’s ECG signal are independently extracted through the proposed time-frequency resolution adjusted 2D spectrogram image. The registration and the test processes are the same in the rest of the steps except for P, QRS Complexes, and T waves cycle normalization using nonoverlapping ECG signals until the feature extraction step and the registration data are trained and modeled. The test data is matched with the modeled registration data and classified in the softmax layer of CNN. An ECG signal acquired in a static condition is used as registration data, and an ECG signal acquired in a complex condition while driving is used as test data. ECG signals used in the system were acquired with the lead-I signal from both wrists based on international standard 12 lead and built DB. The ECG lead-I signal typically includes power line noise generated from measuring equipment, baseline fluctuation noise caused by the subject’s breathing, and muscle noise caused by the subject’s movement. Representative noise included in ECG lead-I was removed through a frequency filtering process. The average moving filter is applied to the section excluding the QRS Complexes interval containing unique individual features for fine noise. The high- and low-frequency component noise in ECG lead-I signal are removed by the Butterworth and Notch bandpass filter. The ECG’s R peak is detected from the threshold value by the Pan–Tompkins R peak detection algorithm [42]. The detected R wave peak is used to segment the fiducial point and set the morphological feature section. The baseline wandering noise caused by the driver’s breath is not removed after applying the frequency filter and moving average filter to remove the noise. Choi et al. [43] used a continuous first-order regression analysis method to remove baseline wondering noise, maintaining morphological features and adjusting them to zero. In addition, a normalization study was conducted in which the morphological features of the ideal ECG cycle P, QRS Complexes, and T waves were clearly generated [44]. Rehman et al. [45] confirmed the fiducial point segmentation method using the P, QRS Complexes, and T waves features of the ECG’s one cycle with higher identification accuracy than the nonfiducial point segmentation method. We applied the fiducial segmentation method with excellent identification accuracy. ECG’s one cycle is set from the peak of the R wave detected by the Pan–Tompkins algorithm to the P wave on the left and the T wave on the right. The segmented ECG one cycle is transformed into a 2D spectrogram that time-frequency resolution is optimized to extract features from the proposed multidimensional features. The 2D spectrogram features are inputted to a single CNN to identify the driver in the softmax layer.


### 3.2. Resolution Adjusted 2D Spectrogram for Multidimensional Features

The feature extraction step is a process that directly affects identification performance. Existing feature extraction techniques have been studied in single-dimensional and multidimensional networks to improve identification performance and are not affected by identification performance as the number of subjects increases and identification performance is improved. Features extracted from a single dimension are analyzed with high identification performance, but there are disadvantages of decreasing as the number of subjects increases. Features extracted from multiple dimensions maintain identification performance even when the number of subjects increases but are analyzed with low identification performance. The performance of feature extraction and identification classification in the network has been analyzed with high performance by overcoming the disadvantages of the existing multidimensional feature extraction method. Due to the complex structure of a single network and multiple networks, there is a disadvantage in that the learning time is delayed due to high time complexity (HTC), which has a high learning process. In this paper, we propose a multidimensional feature extraction method that solves the problem of HTC and has a higher identification performance than existing feature extraction techniques even when the number of subjects increases. The proposed method consists of a step to convert an ECG’s one cycle into a spectrogram by optimizing the time-frequency resolution, a 2D image resizing step, and a driver identification step by a low hidden layer in a single network composed of a CNN.The spectrogram with the adjusted time-frequency resolution is a process of converting one cycle of an ECG into a spectrogram, adjusting time and frequency resolution, and converting it into an image. Conversion into a spectrogram is determined by the following equation using an ECG signalx(t). The transformed spectrogram X is composed of real and imaginary values s, and the ECG signal and FFT are calculated according to τ where the window function wt is moved and calculated as s value for τ and each frequency w.(1)XR,ω=∫−∞∞xtwt−Re−iwtdt=s.The spectrogram converted using ECG’s one cycle of each subject is shown in Figure2. For each subject, the P, QRS Complexes, and T waves of the ECG signal are converted into a spectrogram so that the size distributed over a frequency band over time can be confirmed.Figure 2
ECG one cycle and spectrogram. (a) Subject 35. (b) Subject 57.
(a)(b)The spectrogram in Figure2(a) is composed of higher magnitude values in a similar frequency band about the P wave and QRS Complexes interval than in (b). In the frequency domain of the spectrogram, the magnitude values of the frequency bands for each subject are composed of unique features. Spectrogram can be output with time and frequency resolution adjusted according to window movement. When the window is moved, the time and frequency resolutions are adjusted by the window movement variable, which is the number of overlapping data amounts. The window shift variable is adjusted by the difference between the FFT length and the number of overlapping data amounts. Because of the inverse relationship between time and frequency resolution, when the window shift variable increases, the time resolution decreases and the frequency resolution increases. It shows the spectrogram with the time and frequency resolution adjusted according to the window movement variable in Table 3. As the time resolution increases, it is possible to check the frequency value of the ECG signal according to time at a narrow interval. As the time resolution decreases, the frequency magnitude values can be checked according to a wider interval of time. In this paper, the spectrogram is extracted as multidimensional features when the time resolution is maximum by considering that the morphological features of the ECG signal are composed of time series data.Table 3
Spectrogram according to window movement adjustment.
Window parameter,R2102030The 2D image resizing process is performed in order to convert the spectrogram extracted as a multidimensional feature into an image and reduce the dimension of the converted image size. The resize method is compressed by an interpolation method without data loss by bicubic interpolation. The resized 2D spectrogram maintains the time resolution, and the image size is reduced, as shown in Figure3 [18]. Bicubic interpolation is obtained after performing cubic interpolation on the x-axis and y-axis, respectively. For any point in the square with four points (0, 0), (0, 1), (1, 0), and (1, 1) as vertices (x, y) 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1, the pixel value is calculated at (i, j) −1 ≤ i ≤ 2 and −1 ≤ j ≤ 2, which is 16 points around the pixel value. The pixel value f(x, y) is determined by the first following equation as a cubic function of x and y. Then, interpolation proceeds after determining the 16 coefficients of a. Four equations can be obtained, like the second following equation using the f(x, y) function.(2)fx,y=∑i=03∑j=03aijxiyj,(3)f0,0=a00,f1,0=a00+a10+a20+a30,f0,1=a00+a01+a02+a03,f1,1=∑i=03∑j=03aij.Figure 3
2D resized spectrogram.Then, the data is reinforced by calculating the perimeter information of the pixel using a 16 dimensional vector, which is a coefficient of 16 polynomials using 8 equations differentiated in the x- and y-directions and 4 equations differentiated in thexy mixture.


## 3.1. Identification System Using the 2D Spectrogram Image

In the identification system for an intelligent vehicle, the registration and test data using the driver’s ECG signal are independently extracted through the proposed time-frequency resolution adjusted 2D spectrogram image. The registration and the test processes are the same in the rest of the steps except for P, QRS Complexes, and T waves cycle normalization using nonoverlapping ECG signals until the feature extraction step and the registration data are trained and modeled. The test data is matched with the modeled registration data and classified in the softmax layer of CNN. An ECG signal acquired in a static condition is used as registration data, and an ECG signal acquired in a complex condition while driving is used as test data. ECG signals used in the system were acquired with the lead-I signal from both wrists based on international standard 12 lead and built DB. The ECG lead-I signal typically includes power line noise generated from measuring equipment, baseline fluctuation noise caused by the subject’s breathing, and muscle noise caused by the subject’s movement. Representative noise included in ECG lead-I was removed through a frequency filtering process. The average moving filter is applied to the section excluding the QRS Complexes interval containing unique individual features for fine noise. The high- and low-frequency component noise in ECG lead-I signal are removed by the Butterworth and Notch bandpass filter. The ECG’s R peak is detected from the threshold value by the Pan–Tompkins R peak detection algorithm [42]. The detected R wave peak is used to segment the fiducial point and set the morphological feature section. The baseline wandering noise caused by the driver’s breath is not removed after applying the frequency filter and moving average filter to remove the noise. Choi et al. [43] used a continuous first-order regression analysis method to remove baseline wondering noise, maintaining morphological features and adjusting them to zero. In addition, a normalization study was conducted in which the morphological features of the ideal ECG cycle P, QRS Complexes, and T waves were clearly generated [44]. Rehman et al. [45] confirmed the fiducial point segmentation method using the P, QRS Complexes, and T waves features of the ECG’s one cycle with higher identification accuracy than the nonfiducial point segmentation method. We applied the fiducial segmentation method with excellent identification accuracy. ECG’s one cycle is set from the peak of the R wave detected by the Pan–Tompkins algorithm to the P wave on the left and the T wave on the right. The segmented ECG one cycle is transformed into a 2D spectrogram that time-frequency resolution is optimized to extract features from the proposed multidimensional features. The 2D spectrogram features are inputted to a single CNN to identify the driver in the softmax layer.


## 3.2. Resolution Adjusted 2D Spectrogram for Multidimensional Features

The feature extraction step is a process that directly affects identification performance. Existing feature extraction techniques have been studied in single-dimensional and multidimensional networks to improve identification performance and are not affected by identification performance as the number of subjects increases and identification performance is improved. Features extracted from a single dimension are analyzed with high identification performance, but there are disadvantages of decreasing as the number of subjects increases. Features extracted from multiple dimensions maintain identification performance even when the number of subjects increases but are analyzed with low identification performance. The performance of feature extraction and identification classification in the network has been analyzed with high performance by overcoming the disadvantages of the existing multidimensional feature extraction method. Due to the complex structure of a single network and multiple networks, there is a disadvantage in that the learning time is delayed due to high time complexity (HTC), which has a high learning process. In this paper, we propose a multidimensional feature extraction method that solves the problem of HTC and has a higher identification performance than existing feature extraction techniques even when the number of subjects increases. The proposed method consists of a step to convert an ECG’s one cycle into a spectrogram by optimizing the time-frequency resolution, a 2D image resizing step, and a driver identification step by a low hidden layer in a single network composed of a CNN.The spectrogram with the adjusted time-frequency resolution is a process of converting one cycle of an ECG into a spectrogram, adjusting time and frequency resolution, and converting it into an image. Conversion into a spectrogram is determined by the following equation using an ECG signalx(t). The transformed spectrogram X is composed of real and imaginary values s, and the ECG signal and FFT are calculated according to τ where the window function wt is moved and calculated as s value for τ and each frequency w.(1)XR,ω=∫−∞∞xtwt−Re−iwtdt=s.The spectrogram converted using ECG’s one cycle of each subject is shown in Figure2. For each subject, the P, QRS Complexes, and T waves of the ECG signal are converted into a spectrogram so that the size distributed over a frequency band over time can be confirmed.Figure 2
ECG one cycle and spectrogram. (a) Subject 35. (b) Subject 57.
(a)(b)The spectrogram in Figure2(a) is composed of higher magnitude values in a similar frequency band about the P wave and QRS Complexes interval than in (b). In the frequency domain of the spectrogram, the magnitude values of the frequency bands for each subject are composed of unique features. Spectrogram can be output with time and frequency resolution adjusted according to window movement. When the window is moved, the time and frequency resolutions are adjusted by the window movement variable, which is the number of overlapping data amounts. The window shift variable is adjusted by the difference between the FFT length and the number of overlapping data amounts. Because of the inverse relationship between time and frequency resolution, when the window shift variable increases, the time resolution decreases and the frequency resolution increases. It shows the spectrogram with the time and frequency resolution adjusted according to the window movement variable in Table 3. As the time resolution increases, it is possible to check the frequency value of the ECG signal according to time at a narrow interval. As the time resolution decreases, the frequency magnitude values can be checked according to a wider interval of time. In this paper, the spectrogram is extracted as multidimensional features when the time resolution is maximum by considering that the morphological features of the ECG signal are composed of time series data.Table 3
Spectrogram according to window movement adjustment.
Window parameter,R2102030The 2D image resizing process is performed in order to convert the spectrogram extracted as a multidimensional feature into an image and reduce the dimension of the converted image size. The resize method is compressed by an interpolation method without data loss by bicubic interpolation. The resized 2D spectrogram maintains the time resolution, and the image size is reduced, as shown in Figure3 [18]. Bicubic interpolation is obtained after performing cubic interpolation on the x-axis and y-axis, respectively. For any point in the square with four points (0, 0), (0, 1), (1, 0), and (1, 1) as vertices (x, y) 0 ≤ x ≤ 1 and 0 ≤ y ≤ 1, the pixel value is calculated at (i, j) −1 ≤ i ≤ 2 and −1 ≤ j ≤ 2, which is 16 points around the pixel value. The pixel value f(x, y) is determined by the first following equation as a cubic function of x and y. Then, interpolation proceeds after determining the 16 coefficients of a. Four equations can be obtained, like the second following equation using the f(x, y) function.(2)fx,y=∑i=03∑j=03aijxiyj,(3)f0,0=a00,f1,0=a00+a10+a20+a30,f0,1=a00+a01+a02+a03,f1,1=∑i=03∑j=03aij.Figure 3
2D resized spectrogram.Then, the data is reinforced by calculating the perimeter information of the pixel using a 16 dimensional vector, which is a coefficient of 16 polynomials using 8 equations differentiated in the x- and y-directions and 4 equations differentiated in thexy mixture.


## 4. Experimental Results

Experiments to analyze the proposed resolution adjusted 2D spectrogram multidimensional feature extraction method and the accuracy of the identification system were conducted in MATLAB R2021a on a PC with an Intel Core i7 processor, NVIDIA GeForce RTX 2060. The ECG signals for analyzing the identification accuracy were acquired from 100 subjects (64 males and 36 females) affiliated with Chosun University (CU-DB), and the age of the subjects ranged from 23 to 34 years old. The protocol for acquiring ECG signals is defined as shown in Table4.Table 4
ECG acquisition protocol.
TimesECG acquisition environmentSit (sec)Slide touch (times)Postexercise (sec)160101802–3-day break260101802–3-day break36010180As shown in Figure4, the ECG data were acquired for each subject three times in the sitting state, slide touch, and postexercise states, respectively. V is the voltage value of the ECG signal. n, m, and l are different numbers of data because the acquisition time is different for each state. The acquired ECG signals consist only of voltage values and are not associated with class labels in the full data.Figure 4
ECG DB description.All subjects acquired a total of 3 times at intervals of 2–3 days. In consideration of the driving status, it was performed in a seated stable condition, a slide touch, and a postexercise status. The ECG acquisition equipment was BIOPAC MP160 and Vernier’s EKG Sensor, and ECG signals were acquired from both wrists using Covidien’s wet electrode at a 2,000 Hz sampling rate. ECG acquisition of the actual driving status was carried out in the afternoon and at night while driving, as shown in Figure5. In the evaluation of the proposed feature extraction method using the acquired ECG signal, the identification accuracy was analyzed according to the resolution adjustment of the spectrogram and compared with the existing multidimensional feature extraction method.Figure 5
ECG lead-I acquisition in driving environment. (a) ECG acquisition environment before driving. (b) ECG acquisition environment while driving.
(a)(b)The driver identification accuracy was measured by the accuracy rate calculation method. The identification rate evaluates the degree of matching predicted by the recognized subject class from the entire enrolled subject class. The classification rate is calculated as the following Equation by calculating the values of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) predicted by 1:N matching. TP: actual “true one subject” correct answer is identified as “true.” FP: actual “false other subject” correct answer is identified as “true.” FN: actual “true one subject” correct answer is identified as “false.” TN: actual “false other subject” correct answer is identified as “false.” Identification accuracy can probabilistically analyze the numerical values corresponding to “true positive” and “true negative” in the confusion matrix through 1:N matching.(4)Accuracy=TP+TNTP+FN+FP+TN.The 2D resized spectrogram image for driver identification is input to a CNN composed of a single network. A CNN designed with a resolution adjusted 2D spectrogram as an input image is composed of 2D convolution, batch normalization, and activation function ReLU with 3 layers, max-pooling with 2 layers, fully connected, and softmax layers as shown in Table5. The input data used for training via CNN is a spectrogram 2D of ECG signals. Feature extraction is carried out by convolution layer in CNN, but resolution adjusted spectrogram 2D as an input data preprocessing step is fed to the network. Subjects are classified for identification in the softmax layer (the last layer of the CNN) through 1:N matching of the test (recognition) data with the trained model. In the spectrogram, the time resolution is analyzed contrary to the frequency resolution, so the time resolution is increased, and the identification performance is evaluated. Time resolution is determined by the number of frequency components and Time Overlap (TO) when the window function moves, so the higher the number of frequency components and the lower the TO and the higher the time resolution. Chun et al. [39] converted according to a fixed 0.02 seconds TO without adjusting the spectrogram resolution. The sampling rate is 200 Hz, and the TO of 0.02 seconds is set to 4 by 200 Hz × 0.02 seconds. Abdeldayem and Bourlai [46] converted according to a fixed 0.015-second TO without adjusting the spectrogram resolution. The sampling rate of the electrocardiogram is 200 Hz, and the TO of 0.015 seconds is set to 3 by 200 Hz × 0.015 seconds.Table 5
CNN structure.
LayerDescription1Input65 × 23 × 322D convolution8, 3 × 3 conv., stride [1 1]3Batch normalizationOutput value nor.4ReLUIfx > 0 gradient 1 and if x < 0 f(x) = 05Max-pooling2 × 2 max value, stride [2 2]62D convolution16, 3 × 3 conv., stride [1 1]7Batch normalizationOutput value nor.8ReLUIfx > 0 gradient 1 and if x < 0 f(x) = 09Max-pooling2 × 2 max value, stride [2 2]102D convolution32, 3 × 3 conv., stride [1 1]11Batch normalizationOutput value nor.12ReLUIfx > 0 gradient 1 and if x < 0 f(x) = 013Fully connected100 fully connected layer14SoftmaxSoftmaxFigure6 compares and analyzes the identification accuracy according to the increase in the number of ECG cycles using a 2D spectrogram whose resolution is not adjusted with each fixed TO (when the time resolution is not maximal) and a 2D spectrogram whose time resolution is adjusted with optimization (when the time resolution is maximal). The driver identification accuracy using the proposed resolution adjusted 2D spectrogram image was analyzed with higher identification accuracy at all ECG cycles than the conventional unadjusted TO resolution method. In the ECG signal, the class was more rightly distinguished when the time resolution of the spectrogram was maximal through the spectral frequency features generation of ECG’s morphological features for each TO section.Figure 6
Comparison of identification performance before and after resolution adjustment.In order to analyze the identification accuracy according to the resolution, the TO and the number of frequency components were constantly adjusted for all four DBs. Using self-acquired CU-DB and public DBs, MIT-BIH NSR, QT, and European, it was composed of learning data and recognition data at 7 : 3. In the following equation of the spectrogram conversion, the time resolution is adjusted by the parameterN that determines the number of frequency components at the sampling rate and the TO according to the sample movement parameter of the ECG cycle signal.(5)XδTs,ωN=∑t=0L−1ωtxδ+te−jωt.The graph on the left in Figure7 is a graph that analyzes the identification accuracy of each DB as TO decreases. When the value of TO is constantly adjusted and the minimum value is 2, all four DBs have the highest identification performance, and when the time resolution is high, and the frequency resolution is low, the identification accuracy is improved and analyzed. The identification accuracy of each DB is analyzed as 97.1% in CU-DB, 100% in MIT-BIH NSR DB, 98.1% in QT DB, and 98.7% in European DB. By increasing the number of frequency components that affect the time resolution, the identification accuracy of each DB was analyzed, as shown in the graph on the right of Figure 7. When the number of frequency components N was 4, the identification accuracy was improved in all 4 DBs, and the analysis was performed constantly from 5. In particular, when the number of frequency components increases, the identification accuracy is improved, and the identification accuracy is converged and analyzed above a certain frequency component.Figure 7
Comparison and analysis of recognition performance according to changes in the number of TO and frequency components.
(a)(b)Using the resolution adjusted spectrogram, the identification accuracy and elapsed learning time were analyzed simultaneously, as shown in Figure8, according to the 2D image size change. The size of the image is adjusted to 8 times, 4 times, 2 times, original image, 1/2, 1/4, and 1/8 at a constant rate. Using a 2D spectrogram original image, the resolution was adjusted by max-pooling by CNN; it was analyzed with 97.1% identification accuracy and 29 seconds of elapsed learning time. By adjusting the 2D spectrogram image size, the identification accuracy from 8 times to 1/4 image size is kept constant without data loss by the convolutional feature of the 2D spectrogram and max-pooling and the elapsed learning time is shortened by 6 seconds. After the size of the 1/4 image, the elapsed learning time was shortened by 2 seconds from the 1/8 size of the 2D spectrogram image, but the identification performance deteriorated. This paper uses a 2D spectrogram of 1/4 image size that maintains identification accuracy and shortens the elapsed learning time.Figure 8
Recognition performance and elapsed learning time according to image size adjustment.The proposed feature extraction method using a resolution adjusted 2D spectrogram compares and analyzes the identification accuracy using the existing multidimensional feature extraction method. Existing multidimensional feature extraction methods are performed in EEMD and MFCCs and to verify the identification accuracy CU-DB was used to analyze the average identification accuracy as the number of ECG cycles increased [37, 38].Figure9 shows a comparative analysis of the identification accuracy of the existing multidimensional feature extraction and the proposed multidimensional feature extraction method. The identification accuracy using the method proposed in ECG cycle 2 and the EEMD feature extraction method was analyzed the highest, the MFCCs feature extraction method was analyzed the highest in cycle 4, and all decreased little by little from cycle 5. The identification accuracy using the proposed feature extraction method was 97.1% in cycle 2, which was higher than that of the existing feature extraction method. EEMD was analyzed as 95.1% in cycle 2 and MFCCs as 96.7% in cycle 4.Figure 9
Comparison and analysis of identification performance according to existing and proposed multidimensional feature extraction methods.


## 5. Conclusions

Driver identification using an ECG signal results in low identification accuracy due to multidimensional features and causes HTC problems due to complex networks. Existing multidimensional feature extraction methods are analyzed with low identification accuracy, and the learning time is delayed by HTC due to a complex network structure. In this paper, we propose a driver identification system in a single CNN designed with a low hidden layer structure of 2D spectrogram multidimensional features adjusted for resolution by optimization. The proposed method consists of a process of converting a segmented ECG cycle into a spectrogram, a process of adjusting time-frequency resolution by optimization, a process of resizing a 2D image, and a process of identifying the driver in a single CNN.The problem of low accuracy was solved as the identification accuracy using the proposed resolution adjusted 2D spectrogram was analyzed as 2% and 0.4% higher in CU-DB than in the existing multidimensional feature extraction methods of EEMD and MFCCs. The identification accuracy using the proposed 2D spectrogram with the optimized resolution was analyzed the highest at 97.1% in CU-DB, 100% in MIT-BIH NSR DB, 98.1% in QT DB, and 98.7% in European DB. In addition, the identification performance was maintained in a single network CNN using a 2D spectrogram of 1/4 image size. It shortens the elapsed learning time by 6 seconds and improves the high time complexity.



---

*Source: 5404343-2022-01-17.xml*