# Electroencephalogram of Happy Emotional Cognition Based on Complex System of Music and Image Visual and Auditory

Authors: Lin Gan; Mu Zhang; Jiajia Jiang; Fajie Duan
Journal: Complexity
 (2020)

---

## Abstract

People are ingesting various information from different sense organs all the time to complete different cognitive tasks. The brain integrates and regulates this information. The two significant sensory channels for receiving external information are sight and hearing that have received extensive attention. This paper mainly studies the effect of music and visual-auditory stimulation on electroencephalogram (EEG) of happy emotion recognition based on a complex system. In the experiment, the presentation was used to prepare the experimental stimulation program, and the cognitive neuroscience experimental paradigm of EEG evoked by happy emotion pictures was established. Using 93 videos as natural stimuli, fMRI data were collected. Finally, the collected EEG signals were removed with the eye artifact and baseline drift, and thet-test was used to analyze the significant differences of different lead EEG data. Experimental data shows that, by adjusting the parameters of the convolutional neural network, the highest accuracy of the two-classification algorithm can reach 98.8%, and the average accuracy can reach 83.45%. The results show that the brain source under the combined visual and auditory stimulus is not a simple superposition of the brain source of the single visual and auditory stimulation, but a new interactive source is generated.


---

## Body

## 1. Introduction

The study of audiovisual synchrony induced electroencephalogram (EEG) is an important part of the brain-computer interface (BCI) system. The BCI system provides a new way for humans to communicate without relying on peripheral nerves and muscles, that is, to communicate and control the outside world by measuring brain waves or other electrophysiological signals in the human brain. By studying the cognitive mechanism of the brain, exploring its efficient way of information processing, and popularizing the research results in the fields of intelligent systems such as computers, the intelligent processing ability of computers can be improved, so as to promote the rapid development of information science.By converting the original brainwave signal into characteristic space in the form of string to a certain extent, the signal noise can be reduced, the change mode of brainwave signal can be abstracted, and the local information of brainwave signal can be kept to the maximum extent. In order to overcome the shortcomings of traditional methods, we extract the modal functions related to specific EEG signal tasks, which greatly improves the performance of emotion classification tasks based on EEG signals.In the practical application of EEG signal analysis and processing, signal analysis methods such as time domain and frequency domain are successively introduced. Lawhern believes that the BCI uses neural activity as a control signal to communicate directly with the computer. For a given BCI example, the feature extractor and classifier are customized for the different features of the EEG control signal expected to limit its application to this specific signal. Here, he proposed whether it is possible to design a CNN architecture to accurately classify EEG signals from different BCI paradigms while making it as compact as possible. In this work, he introduced EEGNet, a compact convolutional network based on EEG’s BCI. He introduced the use of deep convolution and separable convolution to construct an EEG-specific model, which encapsulates the famous EEG feature extraction concept of BCI. He compared EEGNet with the current state-of-the-art methods of the four BCI paradigms: P300 visual evoked potential, error-related negative response (ERN), motor-related cortical potential (MRCP), and sensorimotor rhythm (SMR). Although the conclusions of his research are correct, the research objects are rather vague [1]. Zhang believes that regularization has become a way to prevent overfitting of the brain-computer interface EEG classification. The effectiveness of regularization is usually highly dependent on the choice of regularization parameters usually determined by cross-validation (CV). However, CV imposes two main restrictions on BCI: (1) the user needs a large amount of training data; (2) it takes a relatively long time to calibrate the classifier. These restrictions will greatly reduce the practicality of the system and may cause users to be reluctant to use BCI. They introduced the sparse Bayesian method to classify EEG by using the Laplacian prior, SLaplace. Under the framework of Bayesian evidence, they use Laplacian priors to learn sparse discriminant vectors in a hierarchical manner. All required model parameters can be automatically estimated from the training data without the need for CV. Although their research is more targeted, it is not comprehensive enough [2]. Van believes that many variables in social sciences, physical sciences, and biological sciences (including neuroscience) are nonnormally distributed. In order to improve the statistical properties of such data or to allow parameter testing, logarithmic or logarithmic transformations are usually used. Box-Cox transformation or ad hoc method is sometimes used for parameters, and for these parameters, there is no known transformation that approximates normality. However, these methods are not always consistent with Gaussian. They discussed a transformation that maps the probability distribution to the normal distribution as much as possible and has precise consistency for the continuous distribution. To illustrate this point, the transformation was applied to the theoretical distribution and applied to quantitative electroencephalogram (qEEG) measurements from repeated recordings of 32 highly abnormal subjects. Consistency with Gauss is better than using logarithm, logarithm, or Box-Cox transformation. Their research lacks experimental data [3].The main contributions of this paper are as follows: (1) an effective experimental paradigm of cognitive neuroscience is established, and the cognitive law of visual emotion is obtained, which improves the classification recognition rate; (2) the analysis and processing process of EGG signal is improved to promote the better integration of computer science and medicine; (3) the feature extraction and classification algorithm improvement of EEG data are carried out by using ERP technology.


## 2. Emotional Cognition and EEG Signals

### 2.1. Emotional Cognition Based on Complex System

Emotion is a kind of psychological evaluation made by the body for the things in the surrounding environment relative to its own needs. It can be seen that the cognitive process has a very important meaning for the generation and adjustment of emotions. The body constantly uses cognitive mechanisms to evaluate and judge whether things in the surrounding environment can meet the body’s adaptation needs, and on this basis, there are positive or negative emotional reactions. When the body adopts a more concise and clear cognitive structure and strategies, the way the body evaluates things will be relatively simple, and the emotional experience generated at this time is more likely to be in a strong state; and when the human body adopts more complex and changeable cognitive structures and cognitive strategies, the body will evaluate the surrounding things from multiple aspects and multiple levels, and the emotional experience generated at this time is more likely to tend to a mild state; that is, the complexity of the cognitive structure and the difference in cognitive strategies can greatly affect the generation and experience of emotions [4].Complex system theory is a frontier direction in system science, and its main purpose is to reveal some dynamic behaviors that are difficult to explain with existing scientific methods. Different from the traditional reductionist method, the complex system theory emphasizes the combination of holism and reductionism to analyze the system. Complex systems are very sensitive to changes in individual parameters and partial structures in the system, while the human brain and nervous system are nonlinear and extremely complex systems. Therefore, when conducting research on emotional cognition, people will use the theory of complex systems to better feel the changes in the human body and achieve a high degree of correlation in many fields such as psychology, physiology, and neuroscience.


### 2.2. EEG Signal

When the external stimulation acts on the neurons, the potential difference between the inner and outer sides of the cell membrane decreases, and the excitability is enhanced. As the active potential is generated, a peak pulse will be generated on both sides of the cell membrane, which changes the positive and negative values from the inside to the outside. Synapses are the processes needed to transmit excitability between different neurons, and synapses play a very important role in transmitting excitability. In the process of transmission is excited, excited feeling of cells before contact position after pulse synapses, capsule into the hot spring active, after the release of the neurotransmitter substance synaptic relaxation, like a receiver cell membrane, it can be felt on the surface of the neurotransmitter substance and catch the body cause a series of changes of cell membrane ion channels, to cause the membrane potential of, namely the trigger synapses potential [5].Brain waves can be roughly regarded as the dominant waveform of sine waves, so the waveform of brain waves can be represented by parameters such as frequency, amplitude, and phase. (1) Alpha waves appear when you are awake and close your eyes quietly. The frequency is 8∼13 Hz, and the amplitude is 20∼100μv. This is the most rhythmic waveform in brain waves. During visual stimulation or related cognition, the alpha wave will be immediately replaced by the beta wave. (2) Beta waves appear when the brain is excited, which is related to mental tension and emotional excitement. The frequency is 14∼30 Hz and the amplitude is 5∼20 μv, which is a fast wave. (3) When Cida waves indicate sleepiness or mental anxiety, the frequency is 4∼7 Hz and the amplitude is 10∼50 μv when satisfied. (4) Delta waves are under deep anesthesia, hypoxia, or organic brain diseases, with a frequency of 0.5 to 3 Hz and an amplitude of 20 to 200 μv. The actual measured brain wave is a signal composed of the above-mentioned multiple frequency components, which usually contains a lot of background noise [6].The characteristics of EEG are as follows: (1) EEG is weak and interference noise is strong. Generally, the amplitude of EEG is only about 50μV. In the observation, the signal of the nonstudy object is very strong; for example, some unavoidable interference factors will cause strong interference noise. Due to these interference factors, the requirements for EEG extraction and processing devices need to be increased. For example, EEG detection systems and analysis systems require high input impedance, high common mode removal ratio, and low noise amplification technology. (2) EEG is unstable and random. The instability of the signal means that the statistical characteristics of the signal have nothing to do with the time of statistical analysis. In fact, the rhythm of brain waves is related to changes in mental state. The nonstationarity of brain waves is caused by certain changes in the physiological factors that constitute brain waves, and it has a relatively strong ability to automatically adapt to the outside world. (3) The frequency domain characteristics of EEG are clear, and power spectrum analysis and various frequency processing technologies decide to occupy a more important position than other physiological telecommunications. (4) There is very important mutual information between each read signal. This is because EEG generally uses multichannel signals obtained by a multielectrode measuring device [7].


### 2.3. Emotional Classification of EEG Signals

Emotional brainwave signals are generated by test subjects under specific emotional stimuli. Compared with sleep-related brain wave analysis tasks, the brain wave signal generated by emotional stimulation is longer. All points of the entire brain wave signal have nothing to do with a specific emotion, and most of the emotions are generated in local parts [8].The Gaussian process is a collection of probability variables, which are distributed according to the combined Gaussian. In the Gaussian process regression, these probability variables represent the values of independent variable functions. Gaussian process regression assumes that the average value of the independent variable function distribution is 0, and the correlation between them is represented by the covariance function. The commonly used covariance function is as follows [9]:(1)Ki,j=KXi,Xj=θ0exp−12∑d=1D1ηdxi,d−xj,d.The fully connected layer usually uses the Softmax model to solve multiclassification problems. The loss function of Softmax is as follows:(2)Jθ=−1m∑i=1m∑j=1klyi=jlogeθjl∑keθkl.In the formula,θjl represents the input of the jth neuron node of the lth layer (usually the last layer), and ∑keθkl represents the sum of the inputs of all the neuron nodes of the entire l layer. In order to prevent the local optimization of J(θ), the weight attenuation term is introduced. The specific expression is as follows:(3)Jθ=−1m∑i=1m∑j=1klyi=jlogeθjl∑keθkl+λ2∑i=1k∑j=0nθi,j2.Human beings do not recognize things by pixel by pixel but get local information from a part of the area, collect all the local information, and finally integrate it into the global information. In general, in any image, the closer the distance between two pixels is, the greater the correlation between them will be, while the correlation between two pixels that are far away is relatively small [10]. In fact, human neurons can only capture the local information of a picture and do not respond to global information. But in the end, the currently selected attribute is classified according to the impurity function. If selecting this attribute can reduce the impurity, then this attribute can separate the data. If the impurity function is denoted as i(t), the purity gain is denoted as(4)Δit=itp−Eitc=itp−Plitl+Pritr.Dice similarity coefficient calculation formula is as follows:(5)DICE=2SEG∩GTSEG+GT,(6)y=αWy+β1X−Wβ2X+ε,(7)lngdpit=a0+a1du∗dt+∑i=1NbjXu+εu.Video coefficient calculation formula is as follows:(8)SEN=SEG∩GTGT,(9)lssim=1−2μxμy+C12σxy+C2μx2+μy2+C1σx2+σy2+C2,(10)ψ=∑x=1θVx=∑x=1ϑWx∑1nWℑSx.VOE coefficient calculation formula is as follows:(11)VOE=1−SEG∩GTSEG∪GT,(12)U2=s−p1−kx2,x−p2−k1−x2.RVD coefficient calculation formula is as follows:(13)RVD=100%×1−SEG−GTGT.Jaccard coefficient calculation formula is as follows:(14)Jaccard=SEG∩GTSEG∪GT,(15)xH=p2−p1+12.


### 2.4. Physiological Basis of Music- and Image-Induced EEG

There is a close relationship between emotion and emotion, which is both difference and connection. The process of cognition is accompanied by human emotions and emotions. It comes from the process of cognition and affects the conduct of cognition and activities.When we have different emotions such as joy, anger, and sadness, we usually have emotions first and then emotions. The generation of emotion does not need conditioned reflex, while emotion is gradually acquired and evolved in the society. Emotions are extremely unstable and emotions are relatively stable. Generally, the emotion is not stable, which is situational and temporary. Compared with emotion, emotion is more stable, which is an embodiment of essence and will not change at any time. Inducing different emotions is the most important premise of emotion research. Emotion can be induced by external stimulation and internal response. At present, the common methods of emotion induction can be divided into two types: subjective induction and event induction. The subject elicitation is to make the subjects recall the memory fragments with emotional color or imagine the scene with a specific emotional state to induce the specific emotions of the subjects. The disadvantage of this method is that it cannot ensure that the participants can recall the corresponding memory fragments or imagine the scene of a specific emotional state, so it is difficult to ensure that the subjects can successfully induce specific emotions; even if the induction is successful, it is difficult to measure the duration of the corresponding emotions. Event elicitation is based on the mirror neuron theory, using external means to induce subjects to produce corresponding emotions. It is the most common event inducing method to induce the subjects’ different emotions through external stimuli such as pictures, music, and video, and it is also the most commonly used emotion-inducing method by researchers. Emotion induction is the precondition of emotion recognition research. If the subjects’ emotion cannot be induced successfully, the follow-up research will not be conducted or the wrong results will be obtained [11, 12].


## 2.1. Emotional Cognition Based on Complex System

Emotion is a kind of psychological evaluation made by the body for the things in the surrounding environment relative to its own needs. It can be seen that the cognitive process has a very important meaning for the generation and adjustment of emotions. The body constantly uses cognitive mechanisms to evaluate and judge whether things in the surrounding environment can meet the body’s adaptation needs, and on this basis, there are positive or negative emotional reactions. When the body adopts a more concise and clear cognitive structure and strategies, the way the body evaluates things will be relatively simple, and the emotional experience generated at this time is more likely to be in a strong state; and when the human body adopts more complex and changeable cognitive structures and cognitive strategies, the body will evaluate the surrounding things from multiple aspects and multiple levels, and the emotional experience generated at this time is more likely to tend to a mild state; that is, the complexity of the cognitive structure and the difference in cognitive strategies can greatly affect the generation and experience of emotions [4].Complex system theory is a frontier direction in system science, and its main purpose is to reveal some dynamic behaviors that are difficult to explain with existing scientific methods. Different from the traditional reductionist method, the complex system theory emphasizes the combination of holism and reductionism to analyze the system. Complex systems are very sensitive to changes in individual parameters and partial structures in the system, while the human brain and nervous system are nonlinear and extremely complex systems. Therefore, when conducting research on emotional cognition, people will use the theory of complex systems to better feel the changes in the human body and achieve a high degree of correlation in many fields such as psychology, physiology, and neuroscience.


## 2.2. EEG Signal

When the external stimulation acts on the neurons, the potential difference between the inner and outer sides of the cell membrane decreases, and the excitability is enhanced. As the active potential is generated, a peak pulse will be generated on both sides of the cell membrane, which changes the positive and negative values from the inside to the outside. Synapses are the processes needed to transmit excitability between different neurons, and synapses play a very important role in transmitting excitability. In the process of transmission is excited, excited feeling of cells before contact position after pulse synapses, capsule into the hot spring active, after the release of the neurotransmitter substance synaptic relaxation, like a receiver cell membrane, it can be felt on the surface of the neurotransmitter substance and catch the body cause a series of changes of cell membrane ion channels, to cause the membrane potential of, namely the trigger synapses potential [5].Brain waves can be roughly regarded as the dominant waveform of sine waves, so the waveform of brain waves can be represented by parameters such as frequency, amplitude, and phase. (1) Alpha waves appear when you are awake and close your eyes quietly. The frequency is 8∼13 Hz, and the amplitude is 20∼100μv. This is the most rhythmic waveform in brain waves. During visual stimulation or related cognition, the alpha wave will be immediately replaced by the beta wave. (2) Beta waves appear when the brain is excited, which is related to mental tension and emotional excitement. The frequency is 14∼30 Hz and the amplitude is 5∼20 μv, which is a fast wave. (3) When Cida waves indicate sleepiness or mental anxiety, the frequency is 4∼7 Hz and the amplitude is 10∼50 μv when satisfied. (4) Delta waves are under deep anesthesia, hypoxia, or organic brain diseases, with a frequency of 0.5 to 3 Hz and an amplitude of 20 to 200 μv. The actual measured brain wave is a signal composed of the above-mentioned multiple frequency components, which usually contains a lot of background noise [6].The characteristics of EEG are as follows: (1) EEG is weak and interference noise is strong. Generally, the amplitude of EEG is only about 50μV. In the observation, the signal of the nonstudy object is very strong; for example, some unavoidable interference factors will cause strong interference noise. Due to these interference factors, the requirements for EEG extraction and processing devices need to be increased. For example, EEG detection systems and analysis systems require high input impedance, high common mode removal ratio, and low noise amplification technology. (2) EEG is unstable and random. The instability of the signal means that the statistical characteristics of the signal have nothing to do with the time of statistical analysis. In fact, the rhythm of brain waves is related to changes in mental state. The nonstationarity of brain waves is caused by certain changes in the physiological factors that constitute brain waves, and it has a relatively strong ability to automatically adapt to the outside world. (3) The frequency domain characteristics of EEG are clear, and power spectrum analysis and various frequency processing technologies decide to occupy a more important position than other physiological telecommunications. (4) There is very important mutual information between each read signal. This is because EEG generally uses multichannel signals obtained by a multielectrode measuring device [7].


## 2.3. Emotional Classification of EEG Signals

Emotional brainwave signals are generated by test subjects under specific emotional stimuli. Compared with sleep-related brain wave analysis tasks, the brain wave signal generated by emotional stimulation is longer. All points of the entire brain wave signal have nothing to do with a specific emotion, and most of the emotions are generated in local parts [8].The Gaussian process is a collection of probability variables, which are distributed according to the combined Gaussian. In the Gaussian process regression, these probability variables represent the values of independent variable functions. Gaussian process regression assumes that the average value of the independent variable function distribution is 0, and the correlation between them is represented by the covariance function. The commonly used covariance function is as follows [9]:(1)Ki,j=KXi,Xj=θ0exp−12∑d=1D1ηdxi,d−xj,d.The fully connected layer usually uses the Softmax model to solve multiclassification problems. The loss function of Softmax is as follows:(2)Jθ=−1m∑i=1m∑j=1klyi=jlogeθjl∑keθkl.In the formula,θjl represents the input of the jth neuron node of the lth layer (usually the last layer), and ∑keθkl represents the sum of the inputs of all the neuron nodes of the entire l layer. In order to prevent the local optimization of J(θ), the weight attenuation term is introduced. The specific expression is as follows:(3)Jθ=−1m∑i=1m∑j=1klyi=jlogeθjl∑keθkl+λ2∑i=1k∑j=0nθi,j2.Human beings do not recognize things by pixel by pixel but get local information from a part of the area, collect all the local information, and finally integrate it into the global information. In general, in any image, the closer the distance between two pixels is, the greater the correlation between them will be, while the correlation between two pixels that are far away is relatively small [10]. In fact, human neurons can only capture the local information of a picture and do not respond to global information. But in the end, the currently selected attribute is classified according to the impurity function. If selecting this attribute can reduce the impurity, then this attribute can separate the data. If the impurity function is denoted as i(t), the purity gain is denoted as(4)Δit=itp−Eitc=itp−Plitl+Pritr.Dice similarity coefficient calculation formula is as follows:(5)DICE=2SEG∩GTSEG+GT,(6)y=αWy+β1X−Wβ2X+ε,(7)lngdpit=a0+a1du∗dt+∑i=1NbjXu+εu.Video coefficient calculation formula is as follows:(8)SEN=SEG∩GTGT,(9)lssim=1−2μxμy+C12σxy+C2μx2+μy2+C1σx2+σy2+C2,(10)ψ=∑x=1θVx=∑x=1ϑWx∑1nWℑSx.VOE coefficient calculation formula is as follows:(11)VOE=1−SEG∩GTSEG∪GT,(12)U2=s−p1−kx2,x−p2−k1−x2.RVD coefficient calculation formula is as follows:(13)RVD=100%×1−SEG−GTGT.Jaccard coefficient calculation formula is as follows:(14)Jaccard=SEG∩GTSEG∪GT,(15)xH=p2−p1+12.


## 2.4. Physiological Basis of Music- and Image-Induced EEG

There is a close relationship between emotion and emotion, which is both difference and connection. The process of cognition is accompanied by human emotions and emotions. It comes from the process of cognition and affects the conduct of cognition and activities.When we have different emotions such as joy, anger, and sadness, we usually have emotions first and then emotions. The generation of emotion does not need conditioned reflex, while emotion is gradually acquired and evolved in the society. Emotions are extremely unstable and emotions are relatively stable. Generally, the emotion is not stable, which is situational and temporary. Compared with emotion, emotion is more stable, which is an embodiment of essence and will not change at any time. Inducing different emotions is the most important premise of emotion research. Emotion can be induced by external stimulation and internal response. At present, the common methods of emotion induction can be divided into two types: subjective induction and event induction. The subject elicitation is to make the subjects recall the memory fragments with emotional color or imagine the scene with a specific emotional state to induce the specific emotions of the subjects. The disadvantage of this method is that it cannot ensure that the participants can recall the corresponding memory fragments or imagine the scene of a specific emotional state, so it is difficult to ensure that the subjects can successfully induce specific emotions; even if the induction is successful, it is difficult to measure the duration of the corresponding emotions. Event elicitation is based on the mirror neuron theory, using external means to induce subjects to produce corresponding emotions. It is the most common event inducing method to induce the subjects’ different emotions through external stimuli such as pictures, music, and video, and it is also the most commonly used emotion-inducing method by researchers. Emotion induction is the precondition of emotion recognition research. If the subjects’ emotion cannot be induced successfully, the follow-up research will not be conducted or the wrong results will be obtained [11, 12].


## 3. EEG Experiment on Emotional Cognition under Visual and Auditory Synergy Stimulation

### 3.1. Experimental Data Set

The DEAP data set includes a preprocessed version of the original EEG signal, and its main content is shown in Table1. This version downsamples the original signal, and the sampling frequency is after 128 Hz. The signal is filtered at 4 to 45 Hz through a bandpass filter, and then the traces of the electrical signal are removed by blind source separation. In this experiment, 32 induced EEG signals in the data set were set to 5.0, and joy, arousal, and excitement were classified as low (score <5.0) and high (score ≥5.0), respectively; in 3 emotional dimensions, two classifications were made separately [13].Table 1
Experimental data set.
Number of participants32Number of videos40Video duration60 sEmotional dimensionPleasureArousalLike degreeDominanceFamiliarityValue rangeFamiliarity: 1.0 to 5.0Others: 1.0 to 9.0Recorded signalNumber of channels: 40Number of participantsSampling frequency: 128 Hz


### 3.2. Experimental Platform

The experiment uses presentation to write the experimental stimulation program. Presentation can interact well with ERP, MEG, fMRI, and so on and is often used for stimulus presentation and experimental process control in cognitive experiments. It runs under the Windows environment and can reach millisecond time accuracy [14].


### 3.3. Experimental Process

(1)
fMRI data collection was performed using 93 videos as natural stimuli. These videos were divided into eight large segments and played to three subjects using an MRi-compatible VR eyewear device, while they were scanned by fMRI. Parameters were 30-axis slice, matrix size 64 × 64, layer thickness 4 mm, 220 mmFOV, TR = 1.5 s, TE = 25 ms, and ASSET = 2 [15].(2)
In order to prevent subjects from being stimulated by strong picture colors during the experiment, the experiment adopted a black background with a resolution of 640∗480 and a size equal to half of the screen. Moreover, the experimental pictures had the same brightness and contrast. The pictures were randomly divided into 5 sections and presented in a random manner to effectively avoid the practice effect and fatigue effect. Each image presented 3 s, and images were continuously presented between stimuli in each bar. After the end of each section of the experiment, the subjects could choose whether to take a rest. The experiment lasted for 5 minutes [16].(3)
EEG data acquisition: firstly, set the storage path of EEG data; then the EEG signals recorded on the screen of the subjects were observed. When the EEG signals stabilized, the audiovisual stimulation experimental paradigm was presented on the screen of the subjects. Finally, in accordance with the designed experimental paradigm flow, the subjects were induced by emotional pictures and sounds to collect EEG signals [17].


### 3.4. Data Processing

(1)
Data format conversion (convert .cnt to .mat format): since the original data is a .cnt format file collected by the Scan software, this article uses the EEGlab toolbox to import it into Matlab and then process it. The data saved after processing is in the .mat format, and the data in the .mat format can be used. The data generated by subsequent feature extraction and other works can also use the .mat format.(2)
Complete the independent component analysis processing of the data. Use the ICA algorithm in the EEGlab toolbox to decompose the data, and remove the artifact signals such as ocular electricity and myoelectricity to achieve denoising [18].


## 3.1. Experimental Data Set

The DEAP data set includes a preprocessed version of the original EEG signal, and its main content is shown in Table1. This version downsamples the original signal, and the sampling frequency is after 128 Hz. The signal is filtered at 4 to 45 Hz through a bandpass filter, and then the traces of the electrical signal are removed by blind source separation. In this experiment, 32 induced EEG signals in the data set were set to 5.0, and joy, arousal, and excitement were classified as low (score <5.0) and high (score ≥5.0), respectively; in 3 emotional dimensions, two classifications were made separately [13].Table 1
Experimental data set.
Number of participants32Number of videos40Video duration60 sEmotional dimensionPleasureArousalLike degreeDominanceFamiliarityValue rangeFamiliarity: 1.0 to 5.0Others: 1.0 to 9.0Recorded signalNumber of channels: 40Number of participantsSampling frequency: 128 Hz


## 3.2. Experimental Platform

The experiment uses presentation to write the experimental stimulation program. Presentation can interact well with ERP, MEG, fMRI, and so on and is often used for stimulus presentation and experimental process control in cognitive experiments. It runs under the Windows environment and can reach millisecond time accuracy [14].


## 3.3. Experimental Process

(1)
fMRI data collection was performed using 93 videos as natural stimuli. These videos were divided into eight large segments and played to three subjects using an MRi-compatible VR eyewear device, while they were scanned by fMRI. Parameters were 30-axis slice, matrix size 64 × 64, layer thickness 4 mm, 220 mmFOV, TR = 1.5 s, TE = 25 ms, and ASSET = 2 [15].(2)
In order to prevent subjects from being stimulated by strong picture colors during the experiment, the experiment adopted a black background with a resolution of 640∗480 and a size equal to half of the screen. Moreover, the experimental pictures had the same brightness and contrast. The pictures were randomly divided into 5 sections and presented in a random manner to effectively avoid the practice effect and fatigue effect. Each image presented 3 s, and images were continuously presented between stimuli in each bar. After the end of each section of the experiment, the subjects could choose whether to take a rest. The experiment lasted for 5 minutes [16].(3)
EEG data acquisition: firstly, set the storage path of EEG data; then the EEG signals recorded on the screen of the subjects were observed. When the EEG signals stabilized, the audiovisual stimulation experimental paradigm was presented on the screen of the subjects. Finally, in accordance with the designed experimental paradigm flow, the subjects were induced by emotional pictures and sounds to collect EEG signals [17].


## 3.4. Data Processing

(1)
Data format conversion (convert .cnt to .mat format): since the original data is a .cnt format file collected by the Scan software, this article uses the EEGlab toolbox to import it into Matlab and then process it. The data saved after processing is in the .mat format, and the data in the .mat format can be used. The data generated by subsequent feature extraction and other works can also use the .mat format.(2)
Complete the independent component analysis processing of the data. Use the ICA algorithm in the EEGlab toolbox to decompose the data, and remove the artifact signals such as ocular electricity and myoelectricity to achieve denoising [18].


## 4. EEG Analysis of Emotional Cognition under Visual and Auditory Costimulation Based on Complex System

### 4.1. Analysis of Global Feature Difference Results

First, select the initial value of the weight vector in the convolutional layer, the learning rate is 1e−5, and the momentum factor is 0.9. Under the eight probability distribution conditions of uniform distribution, zero-point distribution, and normal distribution, the effects of different initialization weight vectors on the accuracy of emotion recognition are shown in Figure 1. It can be found from the table that the classification accuracy is the highest when the initialization weight vector is uniformly distributed. The power spectrum entropy of P area, PT area, and O area does not change significantly with the level, which also shows that these areas have a small relationship with emotions.Figure 1
The influence of different distribution forms on the region.As shown in Figure2 and Table 2, the power spectrum entropy of F zone, AT zone, and C zone fluctuates greatly with the level change, especially in F zone. And the power spectrum entropy of the F zone has an upward trend with the level change. This may be due to the fact that the brain is in a highly stressed state when viewing lower-level pictures, which makes the regularity of the brain waves stronger, so that the power spectrum entropy is higher.Figure 2
Spectral entropy fluctuates greatly with changes in energy level.Table 2
The influence of different energy fluctuation forms on the accuracy of emotion recognition.
NumberFATCPowerS-entropyEle-wave10.161.320.060.360.120.0721.33.053.492.0441.2232.255.255.843.165.892.2744.463.042.332.935.393.0851.982.573.782.652.383.4462.612.235.76.156.131.68As shown in Table3, as the level of the picture increases, the state of the subjects is more relaxed when watching the picture, which makes the law of brain waves weaker, and thus the power spectrum entropy becomes larger. The importance of emotion to human life is self-evident, and it affects the individual’s evaluation of external things and affects the individual’s behavior mode in dealing with external things. The relationship between emotion and executive function has also become a research hotspot in recent years. Through research on patients with anxiety and depression, people have found that emotions affect individual working memory, renewal, transformation, and other abilities [19, 20].Table 3
The influence of different initialization weight vectors on the accuracy of emotion recognition.
Data distributionAccuracyVarianceUniform0.6368270.013589Lecun_uniform0.6012380.010331Normal0.6192660.094318Zero0.5583420.024774Glorot_normal0.5710850.043303Glorot_uniform0.5719850.025202He_normal0.5938920.012603He_uniform0.5908750.051937Figure3 shows the accuracy comparison of the two-category emotion recognition of the five algorithms. The horizontal axis in the figure represents the tester number, and the vertical axis represents the accuracy rate. It can be seen from the figure that when using statistical features as input, two algorithms, RBF-SVM and Linear-SVM, are used for emotion recognition, and their accuracy is generally lower than that of the algorithm in this paper.Figure 3
The impact of different initialization weight vectors on the accuracy of emotion recognition.As shown in Table4, through the comparison of the accuracy of the algorithm in this paper and the algorithm in the literature, it can be found that this paper uses the preprocessed brain wave signal as input. By adjusting the parameters of the convolutional neural network, the highest accuracy rate of the two-class algorithm can reach 98.8%, and the average accuracy rate is 83.45%; compared to the emotion recognition algorithm of the convolutional neural network that uses statistical features as input, it is significantly improved [21].Table 4
The parameters of the convolutional neural network.
ItemStimulateNeuronsCell membranePolarizationElimination statePositive1.631.760.81.991.02Overlapping2.651.943.323.681.28Galvanic4.152.184.583.72.83pF2.333.534.261.775.86LO3.424.443.612.123.86PPA6.322.843.095.634.65As shown in Figure4, with the increase in sparsity, the clustering coefficient C under different sparsity levels increases monotonically, and the clustering coefficient can measure the clustering characteristics and tightness of the brain function network. Under high arousal conditions, the clustering coefficient of the brain network with low pleasure is smaller than the brain network of high pleasure; under the condition of low pleasure, the clustering coefficient of low arousal is greater than the clustering coefficient of high arousal.Figure 4
the clustering coefficient of high arousal.As shown in Figure5 and Table 5, this may indicate that, under higher pleasure conditions, the brain nerves are more excited, local connections between brain regions increase, and brain function connections are enhanced, and under the same conditions of low pleasure, the brain function of the brain with low arousal is higher than the high arousal. The brain function of the brain network has more connections between brain regions, and the local connection function is stronger. The research results on local efficiency are consistent with the research results of clustering coefficients. Both of these two attributes can reflect the differentiation ability of local brain functions. Together, the two indicate that high-pleasure emotions promote the local information processing and processing of the brain [22].Figure 5
Comparison of accuracy of two-category emotion recognition of five algorithms.Table 5
The influence of different emotional states on brain nerves.
NSparsityClusteringCoefficientBrain functionInternetPleasureC-coefficient10.210.81.291.241.91.151.1321.772.782.441.593.213.771.7334.345.023.223.664.355.615.8843.44.284.343.085.734.055.5751.282.224.231.574.854.774.2563.765.535.215.635.736.92.5474.282.112.623.274.812.483.51


### 4.2. Analysis of EEG Feature Extraction Results

The average accuracy of discrete dimensions is shown in Table6. As can be seen from the figure, the average accuracy has improved. When using the convolutional neural network to solve the problem of image classification, the input is unstructured image data; that is, we do not need to extract all the features of an image. At the same time, the operation of each layer in the convolutional neural network is equivalent to extracting the image features with a feature operator, and the parameters of the feature operator can be adjusted and updated continuously during the whole network training process, so as to optimize the classification results.Table 6
Average accuracy of discrete dimensions.
FBIF + BF + IB + IF + B + ID28.7224.3620.1817.6724.8616.2021.7021.0426.8840.5829.4937.7146.1838.2352.1531.7130.3830.9045.3535.6928.8657.2826.9045.6640.2643.9353.8532.4447.6030.0430.6845.1447.2356.5257.8743.7946.1033.7844.5745.1228.5448.6826.6832.2249.4047.7927.0443.0027.4054.3659.0934.2126.5527.6958.4035.4238.6158.3957.0845.2059.0243.6441.7260.0947.4443.0526.5048.3131.8757.5152.1438.4153.4349.1241.0730.8353.1338.8545.4345.77As shown in Figure6, the number of features that can be extracted in the whole network is closely related to the scale of the network model. The larger the scale of the network model is, the more features can be extracted and the more types can be effectively distinguished [23]. In addition, since the convolution kernel parameters are constantly updated and adjusted throughout the training process, there is no need to pay special attention to the processing results of each layer, but to adjust the parameters of each layer within a reasonable range based on the output error of the network, and finally complete. From autonomous learning to abstract feature expression, image classification is realized. Instead, we use the average intensity of all voxels in the selected brain area as a benchmark. Such a choice has an advantage over the same conversion factor in the expansion result.Figure 6
Average accuracy of discrete dimensions.As shown in Figure7 and Table 7, compared to the gray matter intensity calculated separately, it affects the change in signal percentage. A brain with 35% white matter is 15% brighter than gray matter, and the average signal percentage using the whole brain is reduced by 5% of the baseline using only gray matter. Generally, the magnitude of this difference is small compared to other errors. Similarly, the threshold used to cover the whole brain only has a small effect on the reasonable range of the threshold [24].Figure 7
The number of features that can be extracted and the size of the network model.Table 7
Comparison of the effects of gray matter intensity and white matter on the brain.
NAngleEdgeInformation gainAreaRidgeNLP10.60.860.290.950.690.1421.623.953.083.271.463.4435.815.382.65.735.523.8142.53.571.034.44.523.6154.213.193.814.73.253.2866.795.392.974.042.166.99


### 4.3. Synchronous Analysis of EEG Induced by Visual and Auditory Perception

The results of the EEG phase synchronization indext-test are shown in Figure 8 and Table 8. The results show that there are significant differences between the EEG synchronicity index of happy emotion cognition (positive emotion) and anger, sadness, surprise, disgust, and fear emotion cognition (negative emotion). When receiving pleasant facial expression pictures and sound synchronization stimulation, the parietooccipital lobe and the left frontal lobe have a high EEG phase synchronization; that is, the visual area and the left emotional area have a synchronized oscillation phenomenon, and the synchronization of the glume lobe and the left frontal lobe is not high; that is, the phenomenon of synchronous oscillation is not obvious.Figure 8
EEG synchronicity index of happy emotion cognition.Table 8
Synchronized oscillation phenomenon.
Visual channelPleasureSound synchronizationGlume leafRight frontal lobeAuditory channel1.140.461.461.611.781.692.581.882.082.083.572.15.794.935.462.93.723.984.434.413.594.974.612.263.871.151.194.81.883.273.64.131.756.873.033.77As shown in Figure9 and Table 9, it can be inferred that the visual channel plays a dominant role in the perception of pleasure. When receiving the stimulus of the sad expression picture and the sound synchronization, the occipital lobe and glume lobe have a high EEG phase synchronization with the right frontal lobe; that is, the visual area and auditory area have synchronized oscillations with the right emotional area, but the leaves are displayed. Compared with the synchronization of the right frontal lobe, the synchronization between the occipital lobe and the right frontal lobe is higher. From this, it can be inferred that, in the cognition of sad emotions, the auditory channel plays a leading role and the visual channel plays a supporting role. When neurons are in a resting state, positive and negative charges are evenly distributed inside and outside the cell membrane, but due to the overlap of the positive and negative centers, all neurons neither will exhibit electrical properties on the outside nor will form galvanic couples [25,26].Figure 9
Visual channels play a leading role in pleasure.Table 9
Visual channels play a leading role in pleasure.
ItemStimulateNeuronsCell membranePolarizationElimination statePositive1.631.760.81.991.02Overlapping2.651.943.323.681.28Galvanic4.152.184.583.72.83pF2.333.534.261.775.86LO3.424.443.612.123.86PPA6.322.843.095.634.65As shown in Figure10 and Table 10, when a neuron is stimulated, the cell membrane is in a state of polarization cancellation, the internal and external charge distribution are uneven, and the centers of positive and negative charges do not overlap, thus forming a pair of galvanic couples. Under sound conditions, a similar situation occurs. The three brain regions pF, LO, and PPA form a network, but they have nothing to do with STS. STS can perceive three modal stimuli: single vision, single hearing, and audiovisual integration, so it is possible that this mode of sound is only processed in STS and has nothing to do with the network formed by those three brain regions. According to the results of multivoxel mode analysis, we see that the STS brain area is also involved in representing the semantic similarity between the scene and the sound, so we further explored the whole brain that is functionally connected to the STS under the sound and scene tasks.Figure 10
Network state formed by brain regions.Table 10
The effect of charge neuron stimulation on vision.
STSCommon partPerceiveIsland leafOcciputMiddle temporalBrain1.540.840.061.820.170.230.6813.551.83.882.262.063.553.4353.182.393.332.154.65.192.95.234.494.453.831.814.411.81.763.112.243.832.091.884.793.126.062.065.652.1As shown in Figure11, the results found that there are common parts of the brain areas that are jointly responsible for processing scenes and sound tasks with STS: perception related (insular lobes), spatial imagery and memory (middle occipital region), semantic related (back of middle temporal gyrus) and object perception related brain area (LOC), and audiovisual integration. As the number of decomposition layers increases, the smoother the waveform is, the more the information is lost; on the contrary, the less the number of decomposition layers, the more the interference in the waveform. It can be obtained by plotting the approximation coefficients and detail coefficients of the above layers, and the preprocessed EEG signal is subjected to 5-layer wavelet decomposition, and the obtained waveform has good smoothness while retaining the details more completely. Therefore, the wavelet coefficients of the fifth layer are algorithmically reconstructed to extract the P300 characteristics of evoked EEG in the visual, auditory, and visual and auditory target stimulation modes. Whether it is visual, auditory, or visual-auditory joint stimulation mode, the wavelet transform method can effectively extract the P300 characteristics of the evoked EEG signal, so that the target stimulus in the three modes can be effectively distinguished from the nontarget stimulus [27, 28].Figure 11
The effect of the number of decomposition layers on the lost information.As shown in Figure12 and Table 11, in the ERP induced by different valence emotion pictures as a stimulus, the middle component and late component also showed significant differences in amplitude. In the right brain, the negative wave amplitude induced by high-pleasure pictures is significantly higher than the negative wave amplitude induced by low pleasure pictures. This phenomenon can indicate that positive stimulation can get more cognitive resources in the right brain, and emotions are in the right brain. Subject to more refined evaluation processing, this cognitive process involves attention and memory cognitive processes. Different processing of visual information with different valence exists in every stage of mental processing.Figure 12
EEG phase synchronization indext-test results.Table 11
The emotional effect of ERP fluctuations.
Emotional processingEmotional priceERP roleUp negativeNegative stimulusLeft brain1.420.551.431.210.46Right brain1.442.11.242.793.82Vi-info5.74.514.055.715.76Fron-lobe4.283.93.462.725.03Cent-area1.73.753.582.433.79As shown in Figure13, whether under high arousal conditions or low arousal conditions, after 300 ms, the left and right brains have different processing methods for different valence of visual information, but they are all involved in the emotional processing process, both in the frontal and central regions. Reflect this difference in processing. The research on the ERP effect of emotional valence mainly focuses on negative bias, but there is no obvious negative bias in this experiment. This is because negative bias mainly occurs when attention resources are relatively lacking, and negative stimuli are allocated to more psychological resources. When psychological resources are sufficient, there is no difference in advantage between positive and negative emotional information processing.Figure 13
Processing under different wakeup conditions.


## 4.1. Analysis of Global Feature Difference Results

First, select the initial value of the weight vector in the convolutional layer, the learning rate is 1e−5, and the momentum factor is 0.9. Under the eight probability distribution conditions of uniform distribution, zero-point distribution, and normal distribution, the effects of different initialization weight vectors on the accuracy of emotion recognition are shown in Figure 1. It can be found from the table that the classification accuracy is the highest when the initialization weight vector is uniformly distributed. The power spectrum entropy of P area, PT area, and O area does not change significantly with the level, which also shows that these areas have a small relationship with emotions.Figure 1
The influence of different distribution forms on the region.As shown in Figure2 and Table 2, the power spectrum entropy of F zone, AT zone, and C zone fluctuates greatly with the level change, especially in F zone. And the power spectrum entropy of the F zone has an upward trend with the level change. This may be due to the fact that the brain is in a highly stressed state when viewing lower-level pictures, which makes the regularity of the brain waves stronger, so that the power spectrum entropy is higher.Figure 2
Spectral entropy fluctuates greatly with changes in energy level.Table 2
The influence of different energy fluctuation forms on the accuracy of emotion recognition.
NumberFATCPowerS-entropyEle-wave10.161.320.060.360.120.0721.33.053.492.0441.2232.255.255.843.165.892.2744.463.042.332.935.393.0851.982.573.782.652.383.4462.612.235.76.156.131.68As shown in Table3, as the level of the picture increases, the state of the subjects is more relaxed when watching the picture, which makes the law of brain waves weaker, and thus the power spectrum entropy becomes larger. The importance of emotion to human life is self-evident, and it affects the individual’s evaluation of external things and affects the individual’s behavior mode in dealing with external things. The relationship between emotion and executive function has also become a research hotspot in recent years. Through research on patients with anxiety and depression, people have found that emotions affect individual working memory, renewal, transformation, and other abilities [19, 20].Table 3
The influence of different initialization weight vectors on the accuracy of emotion recognition.
Data distributionAccuracyVarianceUniform0.6368270.013589Lecun_uniform0.6012380.010331Normal0.6192660.094318Zero0.5583420.024774Glorot_normal0.5710850.043303Glorot_uniform0.5719850.025202He_normal0.5938920.012603He_uniform0.5908750.051937Figure3 shows the accuracy comparison of the two-category emotion recognition of the five algorithms. The horizontal axis in the figure represents the tester number, and the vertical axis represents the accuracy rate. It can be seen from the figure that when using statistical features as input, two algorithms, RBF-SVM and Linear-SVM, are used for emotion recognition, and their accuracy is generally lower than that of the algorithm in this paper.Figure 3
The impact of different initialization weight vectors on the accuracy of emotion recognition.As shown in Table4, through the comparison of the accuracy of the algorithm in this paper and the algorithm in the literature, it can be found that this paper uses the preprocessed brain wave signal as input. By adjusting the parameters of the convolutional neural network, the highest accuracy rate of the two-class algorithm can reach 98.8%, and the average accuracy rate is 83.45%; compared to the emotion recognition algorithm of the convolutional neural network that uses statistical features as input, it is significantly improved [21].Table 4
The parameters of the convolutional neural network.
ItemStimulateNeuronsCell membranePolarizationElimination statePositive1.631.760.81.991.02Overlapping2.651.943.323.681.28Galvanic4.152.184.583.72.83pF2.333.534.261.775.86LO3.424.443.612.123.86PPA6.322.843.095.634.65As shown in Figure4, with the increase in sparsity, the clustering coefficient C under different sparsity levels increases monotonically, and the clustering coefficient can measure the clustering characteristics and tightness of the brain function network. Under high arousal conditions, the clustering coefficient of the brain network with low pleasure is smaller than the brain network of high pleasure; under the condition of low pleasure, the clustering coefficient of low arousal is greater than the clustering coefficient of high arousal.Figure 4
the clustering coefficient of high arousal.As shown in Figure5 and Table 5, this may indicate that, under higher pleasure conditions, the brain nerves are more excited, local connections between brain regions increase, and brain function connections are enhanced, and under the same conditions of low pleasure, the brain function of the brain with low arousal is higher than the high arousal. The brain function of the brain network has more connections between brain regions, and the local connection function is stronger. The research results on local efficiency are consistent with the research results of clustering coefficients. Both of these two attributes can reflect the differentiation ability of local brain functions. Together, the two indicate that high-pleasure emotions promote the local information processing and processing of the brain [22].Figure 5
Comparison of accuracy of two-category emotion recognition of five algorithms.Table 5
The influence of different emotional states on brain nerves.
NSparsityClusteringCoefficientBrain functionInternetPleasureC-coefficient10.210.81.291.241.91.151.1321.772.782.441.593.213.771.7334.345.023.223.664.355.615.8843.44.284.343.085.734.055.5751.282.224.231.574.854.774.2563.765.535.215.635.736.92.5474.282.112.623.274.812.483.51


## 4.2. Analysis of EEG Feature Extraction Results

The average accuracy of discrete dimensions is shown in Table6. As can be seen from the figure, the average accuracy has improved. When using the convolutional neural network to solve the problem of image classification, the input is unstructured image data; that is, we do not need to extract all the features of an image. At the same time, the operation of each layer in the convolutional neural network is equivalent to extracting the image features with a feature operator, and the parameters of the feature operator can be adjusted and updated continuously during the whole network training process, so as to optimize the classification results.Table 6
Average accuracy of discrete dimensions.
FBIF + BF + IB + IF + B + ID28.7224.3620.1817.6724.8616.2021.7021.0426.8840.5829.4937.7146.1838.2352.1531.7130.3830.9045.3535.6928.8657.2826.9045.6640.2643.9353.8532.4447.6030.0430.6845.1447.2356.5257.8743.7946.1033.7844.5745.1228.5448.6826.6832.2249.4047.7927.0443.0027.4054.3659.0934.2126.5527.6958.4035.4238.6158.3957.0845.2059.0243.6441.7260.0947.4443.0526.5048.3131.8757.5152.1438.4153.4349.1241.0730.8353.1338.8545.4345.77As shown in Figure6, the number of features that can be extracted in the whole network is closely related to the scale of the network model. The larger the scale of the network model is, the more features can be extracted and the more types can be effectively distinguished [23]. In addition, since the convolution kernel parameters are constantly updated and adjusted throughout the training process, there is no need to pay special attention to the processing results of each layer, but to adjust the parameters of each layer within a reasonable range based on the output error of the network, and finally complete. From autonomous learning to abstract feature expression, image classification is realized. Instead, we use the average intensity of all voxels in the selected brain area as a benchmark. Such a choice has an advantage over the same conversion factor in the expansion result.Figure 6
Average accuracy of discrete dimensions.As shown in Figure7 and Table 7, compared to the gray matter intensity calculated separately, it affects the change in signal percentage. A brain with 35% white matter is 15% brighter than gray matter, and the average signal percentage using the whole brain is reduced by 5% of the baseline using only gray matter. Generally, the magnitude of this difference is small compared to other errors. Similarly, the threshold used to cover the whole brain only has a small effect on the reasonable range of the threshold [24].Figure 7
The number of features that can be extracted and the size of the network model.Table 7
Comparison of the effects of gray matter intensity and white matter on the brain.
NAngleEdgeInformation gainAreaRidgeNLP10.60.860.290.950.690.1421.623.953.083.271.463.4435.815.382.65.735.523.8142.53.571.034.44.523.6154.213.193.814.73.253.2866.795.392.974.042.166.99


## 4.3. Synchronous Analysis of EEG Induced by Visual and Auditory Perception

The results of the EEG phase synchronization indext-test are shown in Figure 8 and Table 8. The results show that there are significant differences between the EEG synchronicity index of happy emotion cognition (positive emotion) and anger, sadness, surprise, disgust, and fear emotion cognition (negative emotion). When receiving pleasant facial expression pictures and sound synchronization stimulation, the parietooccipital lobe and the left frontal lobe have a high EEG phase synchronization; that is, the visual area and the left emotional area have a synchronized oscillation phenomenon, and the synchronization of the glume lobe and the left frontal lobe is not high; that is, the phenomenon of synchronous oscillation is not obvious.Figure 8
EEG synchronicity index of happy emotion cognition.Table 8
Synchronized oscillation phenomenon.
Visual channelPleasureSound synchronizationGlume leafRight frontal lobeAuditory channel1.140.461.461.611.781.692.581.882.082.083.572.15.794.935.462.93.723.984.434.413.594.974.612.263.871.151.194.81.883.273.64.131.756.873.033.77As shown in Figure9 and Table 9, it can be inferred that the visual channel plays a dominant role in the perception of pleasure. When receiving the stimulus of the sad expression picture and the sound synchronization, the occipital lobe and glume lobe have a high EEG phase synchronization with the right frontal lobe; that is, the visual area and auditory area have synchronized oscillations with the right emotional area, but the leaves are displayed. Compared with the synchronization of the right frontal lobe, the synchronization between the occipital lobe and the right frontal lobe is higher. From this, it can be inferred that, in the cognition of sad emotions, the auditory channel plays a leading role and the visual channel plays a supporting role. When neurons are in a resting state, positive and negative charges are evenly distributed inside and outside the cell membrane, but due to the overlap of the positive and negative centers, all neurons neither will exhibit electrical properties on the outside nor will form galvanic couples [25,26].Figure 9
Visual channels play a leading role in pleasure.Table 9
Visual channels play a leading role in pleasure.
ItemStimulateNeuronsCell membranePolarizationElimination statePositive1.631.760.81.991.02Overlapping2.651.943.323.681.28Galvanic4.152.184.583.72.83pF2.333.534.261.775.86LO3.424.443.612.123.86PPA6.322.843.095.634.65As shown in Figure10 and Table 10, when a neuron is stimulated, the cell membrane is in a state of polarization cancellation, the internal and external charge distribution are uneven, and the centers of positive and negative charges do not overlap, thus forming a pair of galvanic couples. Under sound conditions, a similar situation occurs. The three brain regions pF, LO, and PPA form a network, but they have nothing to do with STS. STS can perceive three modal stimuli: single vision, single hearing, and audiovisual integration, so it is possible that this mode of sound is only processed in STS and has nothing to do with the network formed by those three brain regions. According to the results of multivoxel mode analysis, we see that the STS brain area is also involved in representing the semantic similarity between the scene and the sound, so we further explored the whole brain that is functionally connected to the STS under the sound and scene tasks.Figure 10
Network state formed by brain regions.Table 10
The effect of charge neuron stimulation on vision.
STSCommon partPerceiveIsland leafOcciputMiddle temporalBrain1.540.840.061.820.170.230.6813.551.83.882.262.063.553.4353.182.393.332.154.65.192.95.234.494.453.831.814.411.81.763.112.243.832.091.884.793.126.062.065.652.1As shown in Figure11, the results found that there are common parts of the brain areas that are jointly responsible for processing scenes and sound tasks with STS: perception related (insular lobes), spatial imagery and memory (middle occipital region), semantic related (back of middle temporal gyrus) and object perception related brain area (LOC), and audiovisual integration. As the number of decomposition layers increases, the smoother the waveform is, the more the information is lost; on the contrary, the less the number of decomposition layers, the more the interference in the waveform. It can be obtained by plotting the approximation coefficients and detail coefficients of the above layers, and the preprocessed EEG signal is subjected to 5-layer wavelet decomposition, and the obtained waveform has good smoothness while retaining the details more completely. Therefore, the wavelet coefficients of the fifth layer are algorithmically reconstructed to extract the P300 characteristics of evoked EEG in the visual, auditory, and visual and auditory target stimulation modes. Whether it is visual, auditory, or visual-auditory joint stimulation mode, the wavelet transform method can effectively extract the P300 characteristics of the evoked EEG signal, so that the target stimulus in the three modes can be effectively distinguished from the nontarget stimulus [27, 28].Figure 11
The effect of the number of decomposition layers on the lost information.As shown in Figure12 and Table 11, in the ERP induced by different valence emotion pictures as a stimulus, the middle component and late component also showed significant differences in amplitude. In the right brain, the negative wave amplitude induced by high-pleasure pictures is significantly higher than the negative wave amplitude induced by low pleasure pictures. This phenomenon can indicate that positive stimulation can get more cognitive resources in the right brain, and emotions are in the right brain. Subject to more refined evaluation processing, this cognitive process involves attention and memory cognitive processes. Different processing of visual information with different valence exists in every stage of mental processing.Figure 12
EEG phase synchronization indext-test results.Table 11
The emotional effect of ERP fluctuations.
Emotional processingEmotional priceERP roleUp negativeNegative stimulusLeft brain1.420.551.431.210.46Right brain1.442.11.242.793.82Vi-info5.74.514.055.715.76Fron-lobe4.283.93.462.725.03Cent-area1.73.753.582.433.79As shown in Figure13, whether under high arousal conditions or low arousal conditions, after 300 ms, the left and right brains have different processing methods for different valence of visual information, but they are all involved in the emotional processing process, both in the frontal and central regions. Reflect this difference in processing. The research on the ERP effect of emotional valence mainly focuses on negative bias, but there is no obvious negative bias in this experiment. This is because negative bias mainly occurs when attention resources are relatively lacking, and negative stimuli are allocated to more psychological resources. When psychological resources are sufficient, there is no difference in advantage between positive and negative emotional information processing.Figure 13
Processing under different wakeup conditions.


## 5. Conclusions

This paper mainly studies the changes of EEG under the synergy of music and image visual and auditory stimuli based on complex systems. The event-related potential technology is used to analyze and study the rules of visual emotion cognition. The brain activity analysis method based on emotional valence and the brain activity analysis method based on emotional arousal are used to compare the tester’s emotions on the two indicators of valence and arousal. Marking as a label, by selecting music videos as the tester’s emotional inducing material, it effectively induces the tester to produce obvious emotional changes. Let the tester watch the video intermittently, collect the physiological signals generated by the body during the viewing process, preprocess the collected human physiological signals to construct a new data set, and finally use the data set for the algorithm test.Feature extraction is performed on the preprocessed visual, auditory, and visual-auditory joint evoked EEG signals. When a single channel (visual or auditory) stimulation is given to complete positive emotional cognition, the right frontal lobe has a higher synchronization with the visual or auditory areas of the brain, which is related to the brain receiving visual or auditory stimuli; while completing the negative during emotional cognition, the left frontal lobe and other functional areas did not show synchronized activities. The results of this study show that when receiving a single channel (visual or auditory) stimulation to complete emotional cognition, positive emotions can cause synchronized electrical activity in related areas of the brain more than negative emotions.The body’s emotional state affects the pattern of attention distribution. If the body is in a positive emotional state, it is more likely to adopt a top-down approach to information processing, which relies on the body’s already formed knowledge and experience structure. On the contrary, the organism in a negative emotional state is more likely to adopt a more systematic information processing strategy, which is a bottom-up information processing method, which does not depend on the knowledge and experience of the body. Meanwhile, the organism adopting this method will pay more attention to the details of the current stimulus.



---

*Source: 6631400-2020-12-30.xml*