# Deep-Learning Software Reliability Model Using SRGM as Activation Function
## 3 Software Reliability Model
### Software Reliability Model Using Deep Learning
#### 3.2.2 Recurrent Neural Network  
A recurrent neural network is a model with cells that remember information from previous points in time, and it is specialized for learning sequential data. It is designed so that information from past points in time can influence future points in time. Figure 2a shows the structure of the RNN model represented by Equation (6). \(h_{t-1}\) is information about the previous layer, \(x_{t}\) is information about the current layer, \(\theta_{h}\) is the weight and  
Figure 1: Structure of DNN model. bias of the previous layer, and \(\theta_{x}\) is the weight and bias of the current layer. The activation functions utilized are a hyperbolic tangent function (_tanh_) [44]. \[h_{t}=tanh(\theta_{h}h_{t-1}+\theta_{x}x_{t}) \tag{6}\]  
However, while recurrent neural networks can influence later time points, they suffer from the problem that, as the time horizon becomes longer, the information from farther back in time is not reflected as well. To solve this problem, a model that has the ability to train long-term dependence, LSTM, was developed. An LSTM has a memory cell \(c\). Memory cell \(c\) consists of a forget gate (\(\mathrm{f_{f}}\)), an input gate (\(i_{t}\)), a new memory cell (\(g_{t}\)), and an output gate (\(o_{t}\)). The \(\mathrm{f_{f}}\) determines how much previous information is reflected, the \(i_{t}\) contains new information, the \(g_{t}\) determines how much new information is reflected, and the \(o_{t}\) indicates the amount of previous information passed to the next layer. The \(\theta_{x}\), \(i_{t}\), \(\theta_{x}\), \(g_{t}\), \(\theta_{x,\,i}\) and \(\theta_{x,\,o}\) are the weight for each gate connected to the input data. The \(\theta_{h,\mathrm{f}}\), \(\theta_{h,g}\), \(\theta_{x,\,i}\) and \(\theta_{x,\,o}\) are the weight of each gate for the previous layer. The \(c_{t}\) contains information about the long-term state by combining previous information and current information, and the \(h_{t}\) contains information about the short-term state. The activation functions utilized are a sigmoid function (\(\sigma\)) and the \(\tanh\). Memory cells can be used to solve the problem of vanishing gradients. The structure of the LSTM follows Equation (7). Figure 2b shows the structure of the LSTM model [45; 46]. \[\mathrm{f_{f}}=\sigma(x_{t}\theta_{x,\,t}+h_{t-1}\theta_{h,\mathrm{f}}) \tag{7}\] \[g_{t}=\tanh\left(x_{t}\theta_{x,\,g}+h_{t-1}\theta_{h,g}\right)\] \[i_{t}=\sigma(x_{t}\theta_{x,\,i}+h_{t-1}\theta_{x,\,i})\] \[o_{t}=\sigma(x_{t}\theta_{x,\,o}+h_{t-1}\theta_{x,\,o})\] \[c_{t}=\mathrm{f_{f}}\circ c_{t-1}+g_{t}\circ i_{t}\] \[h_{t}=o_{t}\circ\tanh(c_{t})\]  
LSTM solved the problem of the vanishing gradient for RNN. However, the larger the data size, the more computational effort is required by multiplying the number of parameters, resulting in inefficiency. To improve this, a gated recurrent unit (GRU) model was proposed. It reduces the four operations of the LSTM to three, creates an update gate (\(z_{t}\)) that combines the \(\mathrm{f_{f}}\) with the \(i_{t}\) in the LSTM into one, and replaces the \(o_{t}\) with a reset gate (\(r_{t}\)). The \(r_{t}\) and \(z_{t}\) determine the size of previous information forgotten and how much previous information is retained. The \(g_{t}\) is the gate that determines the information candidates at time \(t\). \(\theta_{x}\), \(r\), \(\theta_{x,z}\), and \(\theta_{x,\,g}\) are the weight for each gate connected to the input data. \(\theta_{h,r}\), \(\theta_{h,z}\), and \(\theta_{h,g}\) are the weight of each gate for the previous layer. The \(h_{t}\), which integrates the \(c_{t}\) and \(h_{t}\) of the LSTM, contains information from the previous and current  
Figure 2: (a) Structure of RNN model, (b) structure of LSTM model, and (c) structure of GRU model. layers together. It learns as it transfers to the next layer through Equation (8). Figure 2c shows the structure of the GRU model [47, 48]. \[