# Enhancing Attention's Explanation Using Interpretable Tsetlin Machine
## 3 Proposed Architecture: TM Initialized Attention Model
### Attention Based Neural Network  
Here we explain the attention-based neural network for text classification where we use conventional Bi-GRU as the language representation layer and attention on top of it. Because of its linked hidden layers, where the internal states are used to process data in a sequential fashion, recurrent neural networks (RNNs) [38] have lately become the standard for NLP. RNNs, on the other hand, have several drawbacks that have led to the creation of versions like LSTM and GRU. The GRU, like the LSTM unit, regulates the flow of information without using a memory unit, making it more efficient with near-lossless performance [39]. GRU also overcomes the issue of vanishing gradients and gradient explosions in vanilla RNN. Our selected model consists of a Bi-GRU layer on top of embedding layer initialized with Glove embedding. This layer consists of a attention layer on top of Bi-GRU. The overall architecture of proposed model is shown in Figure 2.  
Consider a sentence \"This is wonderful movie.\" which is fed to the embedding layer initialized by Glove embedding. On the other hand, we obtain the clause score for each word in the sentence and feed to the embedding layer to match the dimension of input sentence embedding. Then both the embedding layer is passed to multiplication layer, where both are multiplied element wise. The output of the multiplication layer is then fed to the Bi-GRU having multiple hidden layers. Let us assume that the input to Bi-GRU is given by \(X=[x_{1},x_{2},x_{3},\dots,x_{k}]\) where \(k\) is the padded length of the input sentence. This information is passed to Bi-GRU layer. In GRU, there are two types of gates: update gates and reset gates. The update gate determines how much previous data must be brought into the current state and how much new data must be introduced. On the other hand, reset gate decides how much information from the previous steps is passed into the current state \(h_{t}\). Here, \(h_{t}\) is the output from the GRU at time step \(t\) and \(z_{t}\) means the update gate. At a specific time step \(t\), the new state \(h_{t}\) is given by:  
\[h_{t}=(1-z_{t})\odot h_{t-1}+z_{t}\odot h_{t}, \tag{5}\]  
where \(\odot\) represents the element-wise multiplication. To update \(z_{t}\), we have  
\[z_{t}=\sigma(W_{z_{t}}x_{t}+U_{z_{t}}h_{t-1}+b_{z_{t}}). \tag{6}\]  
Here, \(x_{t}\) is each word of the sentence at time step \(t\) that is passed into the network unit which is then multiplied with its own weight \(W_{z_{t}}\). Similarly, \(h_{t-1}\) represents the information of previous unit and is multiplied with its own weight \(U_{z_{t}}\) and \(b_{z_{t}}\) is the bias associated with update state. The current state \(h_{t}\) is updated using reset gate \(r_{t}\) by \[h_{t}=\tanh\bigl{(}W_{h_{t}}x_{t}+r_{t}\odot\bigl{(}U_{h_{t}}\bigr{)}+b_{h_{t}} \bigr{)}. \tag{7}\]  
At \(r_{t}\), the candidate state of step \(t\) can get the information of input \(x_{t}\) and the status of \(h_{t-1}\) of step \(t-1\). The update function of \(r_{t}\) is given by  
\[r_{t}=\sigma(W_{r_{t}}x_{t}+U_{r_{t}}h_{t-1}+b_{r_{t}}), \tag{8}\]  
where \(W_{r}\), and \(U_{r_{t}}\) are the weights associated with the reset state and \(b_{r_{t}}\) is the bias. The Bi-GRU consists the forward GRU layer \((\overset{\rightarrow}{h_{t}})\) that models the input sentence from step 0 to \(t\) and the backward GRU \((\overset{\leftarrow}{h_{t}})\) from \(t\) to 0. \[\overset{\rightarrow}{h_{t}}=\overset{\rightarrow}{GRU}(x_{t}),\ t\in[1,T], \tag{9}\]  
\[\overset{\leftarrow}{h_{t}}=\overset{\leftarrow}{GRU}(x_{t}),\ t\in[T,1], \tag{10}\]  
\[h_{t}=\left[\overset{\rightarrow}{h_{t}},\overset{\leftarrow}{h_{t}} \right]. \tag{11}\]  
Figure 2: The two-action TA and its transition in TM. As we all know, not all of the words in the context contribute equally to text categorization. As a result, an attention layer is allocated to the context to prioritize significant words.