# Intrusion Detection in IoT Using Deep Residual Networks with Attention Mechanisms
## 4 Residual Network Architecture  
The specific structure design of the residual module proposed in this paper is shown in Figure 2. Each residual module consists of a CNN unit and an LSTM unit. The CNN unit mainly consists of three convolutional layers: the first convolutional layer has a convolutional kernel size of \(1\times 1\), which is used to reduce the dimensionality of the input feature map; the second convolutional layer has a convolutional kernel size of \(3\times 3\), which is the core part of the CNN structure and is used for feature extraction and nonlinear transformation; the third convolutional layer has a convolutional kernel size of \(1\times 1\), which is used to restore the dimensionality of the feature map to its original size. The proposed model extracts the features of the input data through the convolutional and pooling layers, and then the feature map is transformed and input to the LSTM unit. Assuming that the input datum is x, the convolution layer in the proposed model extracts the spatial features of the given data and performs the convolution operation on the input data to obtain the output:  
\[X=f(w_{i}\otimes x_{i}+b_{i}), \tag{1}\]  
where \(f(\cdot)\) is the activation function, \(w_{i}\) is the convolution kernel weight, and \(b_{i}\) is the corresponding bias. The rectified linear unit (ReLU) is used as the activation function:  
\[f(x)=\max(0,X). \tag{2}\]  
The first convolution operation is performed in the residual unit, using a \(1\times 1\) convolution kernel, reducing the number of channels of the input feature map to C/4 and the size of the output feature map to H x W x C/4; batch normalization (BN) and ReLU activation operations are performed on the output feature map:  
\[y_{i}=\lambda x_{i}^{\prime}+\varphi, \tag{3}\]  
where \(\lambda\) and \(\varphi\) are learning parameters and \(x_{i}^{\prime}\) is calculated as follows:  
\[\mu_{\beta}=\frac{1}{m}\sum_{i=1}^{m}x_{i}. \tag{4}\]  
Figure 2: Residual network structure diagram. \[\sigma_{\beta}^{2}=\frac{1}{m}\sum_{i=1}^{m}(x_{i}-\mu_{\beta})^{2}. \tag{5}\] \[x_{i}^{\prime}=\frac{x_{i}-\mu_{\beta}}{\sqrt{\sigma_{\beta}^{2}- \varepsilon}}. \tag{6}\]  
In these equations, \(\mu_{\beta}\) represents the mean of the mini-batch, which is the average value of the \(\beta\)-th feature map element in a mini-batch; \(\sigma_{\beta}^{2}\) represents the variance of the mini-batch, which is the variance of the \(\beta\)-th feature map element in a mini-batch; \(x_{i}\) represents the \(i\)-th element of the input feature map; \(x_{i}^{\prime}\) represents the \(i\)-th element of the normalized input feature map; \(m\) represents the size of the mini-batch in batch normalization; that is, the number of elements in the input feature map; and \(\varepsilon\) represents a small constant in batch normalization used to prevent division by zero. The second and third convolution operations are performed using \(3\times 3\) and \(1\times 1\) convolution kernels, respectively, and each convolution operation is followed by a BN and ReLU activation operation. The output of the previous layer is then downsampled by the pooling layer to compress its features, making the data feature dimensionality reduced by:  
\[h_{j}^{l}=down\left(h_{j}^{(l-1)}M^{l}\right), \tag{7}\]  
where \(M^{l}\) denotes the size of the \(l\) pooling layer and \(down(\cdot)\) denotes the downsampling function. The maximum pooling layer is a common down-sampling technique in convolutional neural networks, which serves to reduce the feature dimension and improve the computational efficiency and generalization ability of the model. The specific roles of the maximum pooling layer are as follows: feature compression: the maximum pooling layer can reduce the size of the feature map to reduce the computation of the model, and it can retain the main features of the original data to avoid overfitting.