# Design and Investigation of Capsule Networks for Sentence Classification
## 5 Conclusions  
In this work, we presented an innovative CapsNet architecture for sentence classification. The performance of our model was compared with CNN and LSTM based models and other state-of-the-art models on five datasets. In most cases, our CapsNet model shows either superior or a very comparable performance to the other models, even with relatively simple network architecture. Just using the same dataset, our CapsNet model with a 2-phase training process was able to achieve the best results and hence we conclude that this multi-phase training approach can be very helpful to increase performance for networks with combined modules in any area of application. By intentionally changing the order of words in the test input, we learned that our CapsNet and the LSTM-based models are sensitive to the order of words in the given sentence while the CNN-based models were found to be less sensitive to the changes. This in turn means that CapsNet and LSTM based models can capture spatial information better than CNN-based models. This can be further interpreted as CapsNets can prevent adversarial attacks or can not be tricked by introducing artificial noise like intentional reshuffling. In addition to the numerical results in terms of accuracy, the results of visualization in the first convolution layer have further proven that the proposed CapsNet model learns the sequential order of words well.  
In addition, we noticed from the experiments we performed that using pre-trained Word2 vec models for encoding text data is of great importance. Apart from its use of word-to-vector encoding, the Word2 vec embedding can also be used to generate synonym words for the purpose of data augmentation. Furthermore, due to the nature of the convolution operation used in CapsNet and CNN based models, the proposed network resulted in an increased processing speed compared to RNN-based models.  
Conceptualization, H.W.F. and T.-H.K.; Methodology, H.W.F.; Software, H.W.F.; Validation, H.W.F. and T.-H.K.; Investigation, H.W.F.; Resources, T.-H.K.; Data curation, H.W.F.; Visualization, H.W.F.; Supervision, T.-H.K.; Project administration, T.-H.K.; Funding acquisition, T.-H.K.  
This paper was supported by the Research Fund, Kumoh National Institute of Technology.  
\begin{table}
\begin{tabular}{l l l l} \hline \hline \multicolumn{3}{c}{**Prediction Confidence**} & \multicolumn{1}{c}{**Sample Sentence**} \\ \hline
**CapsNet** & **LSTM** & **CNN** & \\ \hline
0.695352 & 0.692398 & 0.999564 & a stuporously solemn film \\ (Positive) & (Positive) & (Negative) & \\ \hline
0.842644156 & 0.99852017 & 0.78954309 & though everything might be literate and smart, it never \\ (Negative) & (Negative) & (Positive) & took off and always seemed static \\ \hline
0.833856 & 0.998444 & 0.9367719 & Elvira fans could hardly ask for more \\ (Negative) & (Negative) & (Positive) & \\ \hline
0.719898 & 0.993055 & 0.999979 & there might be some sort of credible gender provoking \\ (Positive) & (Positive) & (Negative) & philosophy submerged here, but who the hell cares? \\ \hline \hline \end{tabular}
\end{table}
Table 7: Sample sentences incorrectly predicted by CapsNet but correctly predicted by CNN.  
Conflicts of Interest: The authors declare no conflict of interest. The funders had no role in the design of the study; in the collection, analyses, or interpretation of data; in the writing of the manuscript, or in the decision to publish the results.