# Deep Learning Stranded Neural Network Model for the Detection of Sensory Triggered Events
## 3 Materials and Methods
### Proposed Stranded-NN Model
In real-time cases where the number of monitoring equipment sensors is limited (for example, \(k=2\) temperature sensors), small batch values (for example, \(m=12\)) are used. The stranded-NN algorithm cannot generate enough hidden layers for such small values. That is why a threshold value of \(m=32\) was added in order for the stranded-NN algorithm to be able to generate at least a two hidden layer model (\(L_{1}^{64}-L_{2}^{16}\)) of 64 and 16 perceptrons per layer accordingly. In order to eliminate degraded gradients, L1 regularization is performed over the cross entropy loss function in each hidden layer, according to Equation (3):  
\[MLF=-\frac{1}{N}\sum_{n=1}^{N}\sum_{i=1}^{K}pn_{i}\cdot log_{2}y_{ni}+\lambda \sum_{k=1}^{l}|W_{k}|, \tag{3}\]  
where \(MLF\) is the modified loss function for the NN layer, \(N\) is the number of samples, \(K\) is the number of detection classes, \(pn_{i}\) is the indicator value that the sample \(n\) belongs to class \(i\), \(y_{ni}\) is the probability that the strand associates the \(n\)th input with class \(i\), and \(\sum_{k=1}^{l}|W_{k}|\) is the sum of the layer absolute weight values. Parameter \(\lambda\) is set to 0.005 as derived by experimentation for all model strands. Table 1 summarizes the hyperparameters accessible for each strand and their tuned values. To eliminate over-fitting issues, especially for datasets with a limited number of batch data, dropout layers can be introduced among layers as follows:  
* For even numbers of \(q\), a dropout layer can be inserted after every even layer depth;
* For odd numbers of \(q\), a dropout layer can be inserted after every odd layer depth after the first hidden layer. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Hyperparameter** & **Description** & **Tuned Value** \\ \hline \(q\) & Number of hidden layers per strand & \(2\leq q\leq 12\) \\ \(\lambda\) & L1 regularization parameter & 0.005 \\ \(p_{drop}\) & Drop probability of dropout layers & \(0.05\leq p_{drop}\leq 0.1\) \\ \(m_{max}\) & Maximum number of input measurements per batch & 4096 \\ \(\lambda_{t}\) & Learning rate parameter & \(10^{-4}\leq\lambda_{t}\leq 0.01\) \\ \(\lambda_{d}\) & Learning rate decrease coefficient & 0.2 \\ \(n_{s}\) & Number of stranded model strands & \(2\leq n_{s}\geq 32\) \\ \(C_{max}\) & Maximum number of outputs—detection classes & 16 \\ \hline \(Epochs\) & Number of train epochs & 20–80 \\ \(Batch_{size}\) & Batch size of train data & 32, 64, 128, 256, 512 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Hyperparameters tuning per strand of the stranded-NN model. The use of dropout layers for the model is not obligatory. Nevertheless, in cases of over-fitting, dropouts can be set uniformly across all strands of the stranded-NN network if requested. The drop probability of each layer can be determined via fine-tuning experimentation to minimize catastrophic drops. As a guideline from the authors' experimentation, the drop probability may be a randomly set value per layer between 0.05 and 0.1. For values above 0.1, significant losses and accuracy degradation were observed. The stranded-NN model can be used with or without including dropout layers. For big data input cases, the use of dropouts is not recommended. To limit the number of constructed layers, as the value of batch measurements \(m\) increases, a layer limit was set, such as \(q\leq 12\). The value of \(m=4096\) measurements per input batch was set as a measurements' threshold value to cover even the cases of periodic checks. Nevertheless, it is considered a hyper-parameter by the stranded model. It can be altered if more frequent sensory probing is performed (less than 10 s) or a big set of sensory observations is collected per machinery asset (more than 128 observations per real-time interval). During the per-strand training process of the stranded-NN model, the Adam solver was used with the categorical cross entropy as a loss function.