# Heart Murmur Quality Detection Using Deep Neural Networks with Attention Mechanism
## 4 Method
### Neural Network Model for Segments' Detection  
The designed neural network model is based on two-dimensional convolutional neural networks (2D-CNNs), channel attention, and bidirectional gated recurrent units (Bi-GRU). The CNN-Block was used for initial image feature extraction from the log-Mel spectrograms. Subsequently, the feature maps were flattened into a long feature sequence to be fed into the Bi-GRU for temporal feature extraction. In particular, a Squeeze-and-Excitation (SE)-Block was added between the CNN-Block and the Bi-GRU for giving different weights to different channels of the feature map, since the information in different channels may have different importance for the detection of the murmur quality (the results verified this). Figure 3 shows the structure of the designed neural network model, which included the following:  
* CNN-Block: It contains three layers of 2D convolution; each layer of 2D convolution follows an activation function (ReLU) and a batch-normalization (BN) layer. The first layer of 2D convolution has a convolution kernel size of \(5\times 5\), a stride of 2, and a padding of 2; the second and third layers of 2D convolution both have a convolution kernel size of \(3\times 3\), a stride of 1, and a padding of 1. Bias is added to all convolutions, and the padding is zero padding. Through the CNN-Block, the 1-channel log-Mel spectrogram became a 32-channel feature map.
* SE-Block: This kind of block was proposed by [PERSON] et al. [38]. It can model the interdependencies between channels by the information of different channels of the feature map, that is, to generate corresponding weights for each channel of the feature map, so as to recalibrate the features by the importance of different channels. Now, the  
Figure 2: The waveforms and log-Mel spectrograms of two typical heart sound segments (two with systolic murmurs described as “Harsh” and “Blowing”). n.u. refers to normalized units.  
structure of the added SE-Block was explained. Firstly, there is global average pooling, which can change the feature map from a C \(\times\) H \(\times\) W matrix to a C \(\times\) 1 \(\times\) 1 matrix, which is called squeeze. Then, there are two fully connected (FC) layers (between which is the ReLU activation function) with the output sizes of C/2 and C and the Sigmoid activation function, which is called excitation. Finally, a C \(\times\) 1 \(\times\) 1 matrix was obtained, which means that each channel received a weight, and then the different channels were weighted by doing a channel-wise multiplication with the input feature map (Scale).
* Bi-GRU: It is a bidirectional GRU module for extracting features from long sequences. The size of the hidden state in this module is 64, and the module is added bias.
* Linear Prediction Head: It contains an FC layer and a Sigmoid activation function that produces the final predicted labels for each segment.  
When training the neural network, the Adam optimizer was used, the learning rate was set to \(3*10^{-5}\), \(\beta_{1}\) = 0.9, \(\beta_{2}\) = 0.98, and the weight decay was 0.01. \(\beta_{1}\) refers to the exponential decay rate of the first moment estimation, and \(\beta_{2}\) refers to the exponential decay rate of the second moment estimation. The weight decay setting allowed for the L2 regularization to be added to the loss function, thus reducing overfitting to some extent. The loss function used was the cross-entropy loss function. The batch size was set to 64.