# MaLang: A Decentralized Deep Learning Approach for Detecting Abusive Textual Content
## 3 Materials and Methods
### Recurrent Neural Networks  
Many text classification operations have been performed using recurrent neural networks (RNN) [57; 58]. They enable the modeling of sequence-based tasks; however, they are known to suffer from the problem of vanishing gradients that diminishes the learning of long sequences [59]. To avoid the problem of a vanishing gradient, an ensemble of Bi-directional LSTM and Bi-directional GRU are used.  
Bidirectional LSTM or Bi-LSTM allows the neural networks to preserve both forward and backward information received from input about the sequence at every step with respect to time, making it time-dependent. Inputs are used in two ways, past to future and future to past, allowing the Bi-LSTM to better understand the context of the text. Figure 5 shows the design of a Bi-directional LSTM layer along with the operations happening in each cell. As seen in the figure, LSTM has three gates, input, output, and forget gates. Unlike RNN, the cell state derivative of cell state vectors prevent gradient vanishing since the long-term dependencies are encoded in them.  
Figure 2: System-level Module of MaLang.  
Figure 4: Probability of the presence of different types of toxic content.  
Figure 5: (**a**) Architecture of the Bi-LSTM neural network (**b**) Functions inside a single LSTM cell.  
Figure 3: Cloud-level Module of MaLang.  
The operations that happen in LSTM are:  
Previous state:  
\[f_{t}=\sigma\Big{(}W_{f^{*}}[h_{t-1},x_{t}]+b_{f}\Big{)}\dots \tag{1}\]  
\[i_{t}=\sigma(W_{i^{*}}[h_{t-1},x_{t}]+b_{i})\dots \tag{2}\]  
where \(i_{t}\) input/update the gate's activation vector.  
\[\widetilde{C}_{t}=tanhtanh(W_{c^{*}}[h_{t-1},x_{t}]+b_{c})\dots \tag{3}\]  
\[C_{t}=f_{t}*C_{t-1}+i_{t}*\widetilde{C}_{t}\dots \tag{4}\]  
\[o_{t}=\sigma(W_{0}[h_{t-1},x_{t}]+b_{0})\dots \tag{5}\]  
\[h_{t}=o_{t}*tanhtanh(C_{t})\dots \tag{6}\]  
where \(W\) is the weight matrix, \(b\) is the bias vector parameter, \(o_{t}\) is the activation vector of output vector, \(\widetilde{C}_{t}\) is the cell input activation vector, \(C_{t}\) is the cell state vector, and \(h_{t}\) is the hidden state vector [60].  
Similar to BiLSTM, bidirectional GRUs predict the current state of a sequence using the information preserved from both the previous and successive time steps. Unlike LSTMs, GRUs are equipped with only two gates: an update gate and a reset gate. Information is transferred using the hidden state, and no memory unit is used. GRUs are more efficient and train faster on less amounts of data giving results on par with those of LSTMs. Figure 6 shows the architecture of the Bidirectional GRU or Bi-GRU and the operations happening in every cell.  
The operations that happen inside a GRU cell are:  
Reset Gate:  
\[R_{t}=\sigma(X_{t}W_{xx}+H_{t-1}W_{ht}+b_{r})\dots \tag{7}\]  
Update Gate:  
\[z_{t}=\sigma(X_{t}W_{xx}+H_{t-1}w_{ht}+b_{z})\dots \tag{8}\]  
where \(W_{\mathbf{x}}\), \(w_{\mathbf{xz}}\) and \(W_{\mathbf{hr}}\), \(w_{\mathbf{hz}}\) are weight parameters, whereas \(b_{r}\) and \(b_{z}\) are biases.  
Hidden State:  
\[\widetilde{H}_{t}=\tanh(x_{t}W_{xh}+(R_{t}\odot H_{t-1}W_{ht})+b_{h})\dots \tag{9}\]  
Figure 6: (**a**) Architecture of the Bi-GRU neural network. (**b**) Functions inside a single GRU cell.  
Update State:  
\[H_{t}=Z_{t}\odot H_{t-1}+(1-z_{t})\odot\widetilde{H}_{t}\ldots \tag{10}\]  
The Reset gates capture short-term sequences, whereas the update gates capture long-term sequences.