# A Review on Deep-Learning-Based Cyberbullying Detection
## 5 Deep-Learning-Based Models
### Long Short-Term Memory  
LSTM stands for Long Short-Term Memory [109] network, which is basically the modified version of RNN. LSTM is used for remembering the past data for a long period, which is mainly possible for backpropagation during the training period. As we can see in Figure 18, three gates: Forget, Input, and Output gate, represent the LSTM network.  
Figure 16: Architecture of a generative adversarial network (GAN) [108].  
Figure 17: Recurrent Neural Network.  
Thus, the LSTM cell contains the following components:  
1. Forget Gate \"_f_\";
2. Cell State \"_C_\";
3. Input Gate \"_i_\" ;
4. Output Gate \"_o_\";
5. Hidden state \"_h_\";
6. Memory state \"_C_\".  
Here is the diagram for a LSTM cell at the time step t.  
Here,  
\(\backslash\) --element wise multiplication;  
\(\backslash\) --element wise addition;  
\(C_{t}\) = current cell memory;  
\(C_{t-1}\) = previous cell memory;  
\(o_{t}\) = output gate;  
\(f_{t}\) = forget gate;  
\(\sigma\) = sigmoid function;  
w, b = weight vectors;  
\(h_{t-1}\) = previous cell output;  
\(x_{t}\) = input vector;  
\(h_{t}\) = current cell output.  
Forget Gate: From the previous hidden state, we obtain some information. The forget gate decides which information is important and which is not based on the previous state information. It basically passes current input \(x_{t}\) and previous state output \(h_{t-1}\) into a sigmoid function, which gives the value between 0 and 1. If the value is important, then the sigmoid output gives the value closer to 1. Then, this output is passed to the cell state and will be multiplied with previous cell state values. The equation of forget gate.  
\[f_{t}=\sigma\Big{(}w_{f^{\prime}}[h_{t-1},x_{t}]+b_{f}\Big{)} \tag{1}\]  
Input Gate: The current \(x_{t}\) and the previous \(h_{t-1}\) are passed into a sigmoid activation function, which transform the value between 0 and 1, and these values are stored into a vector. In this case, 0 indicates important and 1 indicates not important. Again, the same \(x_{t}\) and \(h_{t-1}\) are passed into a tanh activation function, which transforms the value between \(-1\) and 1. A vector is created with all these possible values of the tanh function. Finally, the output of both the sigmoid function and tanh function will be multiplied and passed to the call state.  
\[i_{t}=\sigma(w_{i}*[h_{t-1},x_{t}]+b_{i}) \tag{2}\]  
\[\sim C_{t}=\tanh(w_{c^{\prime}}[h_{t-1}x_{t}]+b_{c}) \tag{3}\]  
Cell State: Now the network has both input and forget gate information, which is required in the cell state to decide and store the information from the new state. After that, the previous cell state and the output of the forget gate will be multiplied. Then, the values  
Figure 18: LSTM Cell.  
are dropped if the output of the multiply is 0. Next, the result of this multiplication will perform addition with the input gates' result and will generate a new cell state.  
\[C_{t}=f_{t}*C_{t-1}+i_{t}*\sim C_{t} \tag{4}\]  
Output Gate: The gate finds the value of the next state, which also includes the information of the previous state's input. Here, again the current \(x_{t}\) and the previous \(h_{t-1}\) are passed into another sigmoid function. On the other side, the new cell state value is passed into a tanh function. Then, the output of the tanh function and the sigmoid function are multiplied and the final result is generated, which is passed into the next hidden state.  
\[o_{t}=\sigma(w_{o}*[h_{t-1},x_{t}]+b_{o}) \tag{5}\]  
\[h_{t}=o_{t}*\tanh(C_{t}) \tag{6}\]  
Other Recurrent Neural Network-based architectures are: Hopfield [110], Bi-LSTM [10], GRU [111] etc.