# Enhancing Attention's Explanation Using Interpretable Tsetlin Machine
## 4 Experiments and Results  
Here, we demonstrate the experiments and the result on the proposed model for enhancing the explanation of attention layer in text classification. We use two sentiment classification datasets for evaluation. They are:  
* **MR** is a movie review dataset for binary sentiment classification with just one sentence per review [40]. There are 5331 positive reviews and 5331 critical reviews in the corpus. In this study, we used a training/test splitfrom [41] ([[https://github.com/mnqu/PTE/tree/master/data/mr](https://github.com/mnqu/PTE/tree/master/data/mr)]([https://github.com/mnqu/PTE/tree/master/data/mr](https://github.com/mnqu/PTE/tree/master/data/mr)) (accessed on 24 February 2022).).
* **Reuters** The Reuters 21,578 dataset has two subsets: R52 and R83 (all-terms version). R8 is divided into eight categories, including 5485 training and 2189 exam papers. R52 is divided into 52 categories and 6532 training and 2568 test papers. We employ Keras [42] to implement our model. Adam [43] is used as the models' optimization method with the learning rate of \(1\times e^{-3}\). Additionally, we adopted Dropout [44] as the regularization strategy and the probability of Dropout was kept to be 0.25. Words are initialized with Glove [25] of 300-dimension word embedding. The batch size was 128 and was run for 100 epochs in the test datasets for obtaining the best results. Since, the main purpose of this paper is to enhance the explanation of the attention layer, we demonstrate the performance of the proposed model with the relatable models to show the impact of each model. The comparable state-of-the-arts are explained below:
* **TF-IDF+LR**: Bag-of-words model with inverse document frequency weighting for term frequency. The classifier is based on logistic regression.
* **CNN**: CNN-rand uses arbitrarily initialized word embeddings [45].
* **LSTM**: The LSTM model that we employ here is from [46], representing the entire text using the last hidden layer. We used both the model that is using pretrained embeddings and without using.
* **Bi-LSTM**: Bi-directional LSTMs are widely used for text classification that models both forward and backward information.
* **PV-DBOW**: PV-DBOW is a paragraph vector model where the word order is not considered and is trained with Logistic Regression used as a softmax classifier [47].
* **PV-DM**: PV-DM is a paragraph vector model, with word ordering taken into consideration [47].
* **fastText**: This baseline uses the average of the word embeddings provided by fastText as document embedding. The embedding is then fed to a linear classifier [48].  
* **SWEM**: SWEM applies simple pooling techniques over the word embeddings to obtain a document embedding [49].
* **Graph-CNN-C**: A graph CNN model uses convolutions over a word embedding similarity graph [50], employing a Chebyshev filter.
* **Tsetlin Machine**: Simple BOW model for Tsetlin Machine without feature enhancement.
* **Bi-GRU+Attn**: Bi-directional GRUs are widely used for text classification. We compare our model with Bi-GRU fed with pre-trained word embeddings along with attention layer on top of it.
* **TM+Bi-GRU+Attn**: Proposed model with Bi-GRU model with pretrained word embedding initialized with pretrained TM score in its input layer.