# Design and Investigation of Capsule Networks for Sentence Classification
## 4 Experiments, Results and Discussion
### 2-Phase CapsNet Training  
In an effort to enhance the performance of our model, we performed an experiment on the model given in Figure 1 using a training procedure we call _2-phase training_. In the first phase of training, we initialized the weights of the convolution layers and the translation matrix \(w_{ij}\) of the capsule layer using _glorot uniform initializer_ while the initial logits \(b_{ij}\) are initialized with zero producing uniform coupling coefficients. We trained the network using the given datasets until best test accuracy was obtained. In the second phase, we trained the network further after reinitializing the translation matrix of the capsule layer and capsules' coupling coefficients while the trained weights of convolution layers are retained. With these proper initial weight values of the convolution layers, the network could be optimized more by learning features effectively to produce better results. In our experiment, this 2-phase training approach resulted in a slightly better performance than the original single-phase training of the CapsNet model. This 2-phase training approach just requires a small additional training time but provides a high potential to improve the performance without the need of increasing the amount of training data. As a result, this approach can be a good option to train multi-module networks for improved performance in any area of application. Our CapsNet model with this 2-phase training is presented in Table 4 as 2-phase CapsNet.  
In Table 4, we present a performance comparison of our model with other latest works. In the first four rows, we present results of the above models we used, while in the remaining rows we present the results reported in the corresponding papers.  
\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Network** & **MR** & **TREC** & **SST-2** & **Subj** & **MPQA** \\ \hline CNN & 79.8 & 91.6 & 84.7 & 93.3 & 90.2 \\ LSTM & 81.6 & 94.5 & 86.5 & 93.5 & 90.0 \\ CapsNet & 83.8 & 96.0 & 86.2 & 94.2 & 91.1 \\
2-phase CapsNet & **84.53** & **96.46** & 86.4 & **95.1** & **91.32** \\ \hline CNN-static [1] & 81.0 & 92.8 & 86.8 & 93.0 & 89.6 \\ CNN-non-static [1] & 81.5 & 93.6 & 87.2 & 93.4 & 89.5 \\ CNN-multichannel [1] & 81.1 & 92.2 & 88.1 & 93.2 & 89.4 \\ Capsule-B [4] & 82.3 & 92.8 & 86.8 & 93.8 & - \\ Capsule-static [5] & 81.0 & 94.8 & - & - & 90.6 \\ MCapsNet [19] & 83.5 & 94.2 & 88.6 & 94.5 & - \\ SVMs [8] & - & 95.0 & - & - & - \\ DCNN [11] & - & 93.0 & 86.8 & - & - \\ LR-Bi-LSTM [28] & 82.1 & - & - & - & - \\ CCAE [9] & 77.8 & - & - & - & 87.2 \\ Bi-LSTM+SWN-Lex [29] & - & - & 89.2 & - & - \\ BLSTM-2 DCNN [18] & 82.3 & 96.1 & **89.5** & 94.0 & - \\ \hline \hline \end{tabular}
\end{table}
Table 4: Comparison: Accuracy of our CapsNet model and other models on different datasets.  
\begin{table}
\begin{tabular}{c c c c c c} \hline \hline \multirow{2}{*}{**CapsNet Model**} & \multicolumn{4}{c}{**Dataset**} \\ \cline{2-6}  & **MR** & **TREC** & **SST-2** & **Subj** & **MPQA** \\ \hline Test Normal & 83.8 & 96.0 & 86.2 & 94.2 & 91.1 \\ Test Shuffled & 77.7 & 86.5 & 81.7 & 89.7 & 89.2 \\ \hline \hline \end{tabular}
\end{table}
Table 3: Accuracy of the proposed CapsNet model on different datasets.  
As it can be seen from Table 4, while our CapsNet model shows comparable or better results than other networks for most datasets, our 2-phase CapsNet model shows much better results than most models. It is clear from the above results that the proposed CapsNet model can understand text data better than CNN models. Table 4 further shows that, compared to other CapsNet-based approaches given in [4; 5; 19], our CapsNet models exhibit superior performance.