# Design and Investigation of Capsule Networks for Sentence Classification
## 4 Experiments, Results and Discussion
### CapsNet Model  
The proposed CapsNet model was implemented with the following hyperparameters: Adam optimizer with learning rate of 0.001, batch size of 30, routing iteration of 3, and 50 epochs. The choice of these values is based on a time-consuming hyperparameter tuning for best performance.  
Except for MPQA and TREC datasets (which have shorter sentence length compared to others), we used the following network parameters:  
* In the first convolution layer, which contains four parallel convolution layers, the kernel size is (7, 300), (8, 300), (9, 300), (10, 300) with a stride of 1 and the number of filters is set to 64.
* In the second convolution layer, which also contains four parallel convolution layers, the kernel size for each convolution layer is (7, 1) with a stride of three and the number of filters is set to 64.
* In the third, fourth convolution layers and primary caps layer, the kernel size is set to (5, 1) with stride of 2, 1 and 2 for each layer, respectively.
* The primary caps layer dimension is set to 8D with 32 channels and
* The Class Caps Layer is set to have a dimension of 16D with rows equal to the number of classes in each dataset.  
For MPQA and TREC datasets, for the first convolution layer kernel size is (3, 300), (4, 300), (5, 300) and (6, 300) with stride of 1. In the second convolution layer, kernel size of (3, 1) with stride of 3 is used. In the third, fourth convolution layers and primary caps layer, the kernel size is (5, 1) with stride of 2, 1 and 2, respectively.  
Similar to the CNN and LSTM models, in order to test whether the CapsNet learns the sequential order of words or not (if it can resist an adversarial attack), we test the network with normal and shuffled test data. The results of our experiment on this model are summarized in Table 3.  
By examining Table 3, we can observe the following points. The difference between the last two rows for every dataset is huge compared to that of the CNN model. Since the fourth row is added merely for the purpose of verifying if the network learns the sequential order of words or just the presence or absence of a given word, we can safely say that CapsNets learn the sequential order of words effectively than CNNs. Doing the same comparison between the CapsNet and the LSTM model reveals that both networks exhibit almost similar responses.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline \multirow{2}{*}{**Dataset**} & \multicolumn{2}{c}{**CNN Model**} & \multicolumn{2}{c}{**LSTM Model**} \\ \cline{2-5}  & **Test Normal** & **Test Shuffled** & **Test Normal** & **Test Shuffled** \\ \hline MR & 79.8 & 76.6 & 81.6 & 77.8 \\ TREC & 91.6 & 86.3 & 94.5 & 83.0 \\ SST-2 & 84.7 & 80.0 & 86.7 & 81.6 \\ Subj & 93.3 & 90.6 & 93.5 & 89.5 \\ MPQA & 90.2 & 89.5 & 90.0 & 86.7 \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy of the CNN and LSTM models on different datasets.  
In order to check if the architecture given in Figure 1 with parallel convolution layers performs better than a basic CapsNet architecture without parallel convolution like the one given in Figure 2, an experiment was done on the MR dataset with normal test data. While we got 83.8% accuracy for the architecture in Figure 1, we got an accuracy of 79.2% for the one in Figure 2. This result proves that the network with parallel convolution layers performs better since it can capture more characteristic features of sentences related to their semantic or/and syntactic meanings with multiple kernel sizes.