## 2 Methodology  
### Conventional Deep Neural Network and Dropout Mechanism  
A conventional deep neural network (DNN) is typically composed of three main parts: the input layer, the output layer, and the hidden layers [21], as illustrated in Figure 2a. The input layer and output layer contain only a single layer of units, whereas the hidden layers can contain multiple layers, with the number of hidden layers referred to as the \"depth\" of the DNN. Each layer is composed of multiple units, and the number of units in each layer is referred to as the \"width\" of the DNN model. The units in different layers are densely connected, and information is propagated forward from the lower to the upper layers, from the input to the output layers. As a result, this type of connection is also known as a feedforward neural network. This process is described as follows:  
\[h_{i}^{l}=\sigma\left(b_{i}^{l}+\sum_{j}w_{ij}^{l}h_{j}^{l-1}\right)=\sigma \left(\mathbf{b}^{l}+\mathbf{W}_{i}^{l}\mathbf{h}^{l-1}\right) \tag{1}\]  
where \(\mathbf{h}^{l}=\left[h_{1}^{l},\cdots,h_{m}^{l}\right]\) is the hidden state of the \(l\)-th hidden layer, \(h_{i}^{l}\) is the hidden state of the \(i\)-th unit, and \(\mathbf{h}^{0}=\mathbf{x}\) is the input dataset; \(w_{ij}^{l}\) is the weight of the \(i\)th unit in the \(l\)-th layer and the \(j\)-th unit in the \(l\)-1 th layer, and \(b_{i}^{l}\) is the bias; and \(\sigma(\cdot)\) is the nonlinear activation function. The information flow from the dataset \(\mathbf{x}\) in the input layer to the predicted \(\mathbf{\overset{\wedge}{y}}\) in the output layer for a DNN with L hidden layers can be expressed as follows:  
\[\mathbf{\overset{\wedge}{y}}=f(\mathbf{x};\mathbf{\theta})=o\left(f^{l}\left(\cdots f^{2} \left(f^{1}(\mathbf{x})\right)\right)\right) \tag{2}\]  
where \(f^{l}(\cdot)\) represents the nonlinear function of the \(l\)-th hidden layer listed in Equation (1), \(o(\cdot)\) is the output function of the output layer, \(\mathbf{\theta}=\left\{\mathbf{W}^{\mathbf{i}},\mathbf{b}^{1},\cdots,\mathbf{W}^{L},\mathbf{b}^{ L}\right\}\) is the parameter to be learned in DNN, and \(f(\mathbf{x};\mathbf{\theta})\) represents the \(\mathbf{\theta}\)-parameterized DNN model with inputs \(\mathbf{x}\). Given the target \(\mathbf{y}\), a loss function that evaluates the prediction performance of the DNN model can be defined, e.g., the mean square error (MSE, as illustrated in Equation (3). \[L(\mathbf{\hat{y}},\mathbf{y};\mathbf{\theta})=L(f(\mathbf{x};\mathbf{\theta}),\mathbf{y})=\frac{1}{2 }\sum_{n=1}^{N}\|f(x_{n};\mathbf{\theta})-y_{n}\|^{2} \tag{3}\]  
Figure 2: Conventional DNN and dropout (arrow denotes the information flow). The loss function \(L(\cdot)\) is a function of parameters \(\mathbf{\theta}\), and \(N\) is the total number of samples in the dataset. A DNN model learns to predict the target \(\mathbf{y}\) by adjusting its parameters \(\mathbf{\theta}\) to minimize the loss \(L(\overset{\star}{\mathbf{y}},\mathbf{y};\mathbf{\theta})\). The gradient descent method is usually adopted in parameter updating; the gradient \(\
abla_{\theta}L\) w.r.t. parameters \(\mathbf{\theta}=\left\{\mathbf{W}^{1},\mathbf{b}^{1},\cdots,\mathbf{W}^{L},\mathbf{b}^{L}\right\}\) of each layer are calculated based on the backpropagation algorithm by applying the chain rule to Equations (2) and (3). Parameters are updated in iterations listed in Equation (4), where \(\alpha\) is the learning rate. \[\mathbf{\theta}\leftarrow\mathbf{\theta}-\alpha\
abla L_{\mathbf{\theta}} \tag{4}\]  
Compared to shallow neural networks with hand-crafted features [40], deep learning is designed to learn more effective representations that extract the nonlinear relationships hidden in data through end-to-end training. However, the parameter space can become extremely large for deep neural networks with dense connections, which can lead to issues such as overfitting and training difficulties. To address this, the dropout mechanism was proposed and has since become a standard technique for training deep neural networks [41].