# Enhancing Attention's Explanation Using Interpretable Tsetlin Machine
## 4 Experiments and Results
### Performance Comparison with State-of-the-Arts  
Table 3 shows the comparison of performance for selected datasets. As we can see that traditional method such as TF-IDF with Logistic Regression (TF-IDF+LR) performs decently in MR with 74.59, R8 with 93.74, and R8 with 86.95. Some sophisticated language model such as CNN, and LSTM performs quite similarly. The only improvement seen among them is Bi-LSTM which incorporates both past and future information for better input representation thereby reaching 77.06% in MR, 96.68% in R8, and 90.54% in R52. Slightly different than language models PV-DBOW and PV-DM performs poorly in all three datasets. Similarly, Graph-based CNN and SWEM perform on par with the state-of-the-arts baselines. On the other hand, the rule-based method TM performs quite comparable to baseline by reaching 75.14% in MR, 96.16% in R8, and 84.62% in R52. The performance is slightly below Bi-LSTM/GRU-based model because of its restriction to use pre-trained word embedding. However, [PERSON] et al. [22] show that embedding similar words using a pre-trained word embedding significantly enhances the performance and outperforms the baselines. However, our proposed model only uses TM explainability to generate prerequisite word weightage to replace human attention input into neural network language models. Hence, this is demonstrated in the table as well. Even though the motive of this task does not necessarily impact the accuracy but there is a slight increase in performance anyway. This is due to the fact that the TM score gives additional weightage to the model's input thereby reaching 77.95% in MR, 97.53% in R8, and 95.71% in R52 for TM+Bi-GRU+Attn. This shows an increment of about 1% in average throughout the selected datasets. In addition to this, we also evaluate some more metrics that supports the performance of the proposed model. Since MR is only binary classification dataset, R8 and R52 is multiclass dataset. Hence, there is need of the evaluation of the performance of each class. Usually unbalanced or multiclass datasets sometimes suffers with low F-scores because the model greedily learns the majority classes. Hence to have a clear picture of our proposed model, we evaluate precision, recall, and f-scores of main baseline TM, Bi-GRU+Attn with our proposed model TM+Bi-GRU+Attn as shown in Tables 4-6 respectively. The results clearly indicate the our proposed model also performance superior on all three selected \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Models** & **MR** & **R8** & **R52** \\ \hline TF-IDF+LR & 74.59 & 93.74 & 86.95 \\ CNN & 74.98 & 94.02 & 85.37 \\ LSTM & 75.06 & 93.68 & 85.54 \\ Bi-LSTM & 77.68 & 96.31 & 90.54 \\ PV-DBOW & 61.09 & 85.87 & 78.29 \\ PV-DM & 59.47 & 52.07 & 44.92 \\ SWEM & 76.65 & 95.32 & 92.94 \\ Graph-CNN-C & 77.22 & 96.99 & 92.75 \\ Tsetlin Machine & 75.14 & 96.16 & 84.62 \\ Bi-GRU+Attn & 77.15 & 96.20 & 94.85 \\ TM+Bi-GRU+Attn & 77.95 & 97.53 & 95.71 \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Performance of the proposed model (TM+Bi-GRU+Attn) with selected baselines. For ease of illustration, we visualize the attention weight of the Bi-GRU model and the attention weight of the Bi-GRU model initialized with TM's word score. We use the red color gradient to demonstrate the weightage of each input word in the context. Dark color represents the higher weightage with light color representing lower weightage. As we can see from Figure 3, only using Bi-GRU, the model recognizes mostly important words for predicting correct sentiment class. However, it is not perfect as the human level. However, Figure 4 shows the visualization of attention weight using Bi-GRU and TM's score. Here we can see that the model focus on more significant words than the previous model. For instance, in the first example, the later model captures \"look\", \"away\" with higher weightage which is an important context for negative sentiment than \"directing\" and \"attempt\".