# Design and Investigation of Capsule Networks for Sentence Classification
## 1 Introduction  
Sentence classification is a fundamental problem in natural language processing (NLP) which aims to organize a very large amount of textual information into groups so that users can access this bulk information with so much ease. Sentiment analysis, a sub-task of sentence classification, tries to identify and categorize opinions expressed in a piece of text to determine the overall contextual polarity. Because human language is complex and there are irregularities in usage, sentence classification is generally a difficult task which requires very careful modeling and analysis.  
Previous research on sentence classification or sentiment analysis relied on using different methods ranging from non-neural network based approaches like topical categorization and knowledge based models to neural network based techniques. With the immense amount of research and improvements in deep learning, however, recent works depend mainly on convolutional neural networks (CNNs) and recurrent neural networks (RNNs)--the two notable networks used in various kinds of computer vision and NLP tasks.  
While CNNs are usually employed in the area of image processing and analysis, [PERSON] [1] introduced the use of CNNs for sentence classification and the results are very promising especially when pre-trained word-vectors are available. Even though CNNs enjoyed enormous popularity in the deep learning community, the recent introduction of capsule networks (CapsNets) by [PERSON] et al. has opened a lot of investigation into the structure of CNNs, especially the pooling operation which causes loss of the important spatial relations between different features [2; 3].  
The motivation for CapsNets stems from the fact that neural networks needed better modeling of spatial relationships among parts instead of modeling the co-existence disregarding relative positioning. CapsNets accomplish this, _capturing of spatial information_, by using a set of nested layers in a networkcalled capsules and implementing an algorithm called routing-by-agreement. It is also reported that CapsNets can achieve better results for various kinds of image analysis tasks and are better in resisting adversarial attacks than their CNN counterparts [3]. Replacing CNNs by CapsNet, which can adequately capture the spatial relations between features, has higher potential for a better representation and understanding of a given NLP task. This performance improvement by using CapsNet for text classification is made evident by recent works of [PERSON] et al. [4] and [PERSON] et al. [5].  
In this work, which is an extension of work presented in [6], we present a CapsNet-based model designed for sentence classification, sentiment analysis in some cases, using a Word2 vec model as the vector encoding technique. We trained the proposed CapNet architecture in two successive phases for optimal performance. First, we did a normal training of the model using a _glorot uniform_ kernel initializer which was then followed by retraining of the same model with reinitialized weights of the CapsNet module in the second phase. The details of the network and approaches used are presented in Sections 3 and 4. Also presented is a comparative analysis of our CapsNet model against other models including CNNs and RNNs. The results of our experiment show that our well-tuned CapsNet model outperforms CNN models and can be a good, sometimes better and cheaper, substitute of much slower RNN-based models used in sentence classification.