# Design and Investigation of Capsule Networks for Sentence Classification
## 2 Related Work
The LSTM-RNN model by [PERSON] and [PERSON] [17] and the bidirectional LSTM model with 2D max-pooling by [PERSON] et al. [18] also reported great classification accuracy. Recent works by [PERSON] et al. [4], [PERSON] et al. [5], [PERSON] et al. [19], [PERSON] et al. [20] and [PERSON] et al. [21] employed CapsNets as the core network for text classification. [PERSON] et al. [4] adopted an ensemble of capsule layers whose output is fed to capsule average pooling layer to produce the final results. In their work, they modified the routing-by-agreement algorithm given in [3] by using _Leaky-Softmax_ and _coefficient amendment_. The results presented show a great improvement over previous CNN-based works. Similarly, [PERSON] et al. [5] presented a text classification with CapsNet-based architecture which uses an ELU-gate unit before the convolutional capsule layer. The authors experimented on both the dynamic routing and its simplified version they referred to as _static routing_. They argued that the simplified _static routing_ produces superior results to dynamic routing for text classification with respect to accuracy and robustness of classification. [PERSON] et al. [19] implemented CapsNets for text classification with multi-task learning. In multi-task learning, sharing as much knowledge between tasks while routing features to the appropriate tasks has been a challenge. The routing-by-agreement in CapsNets has the ability to cluster features that can help to solve the inherent problem of discriminating features for each task in multi-task learning. For this purpose, a modified routing algorithm called task routing was proposed to replace the dynamic routing algorithm so that choosing between tasks may be possible. CapsNets were also implemented with RNN and its variants for text classification tasks. [PERSON] et al. [21] used a combination of capsules and RNN for sentiment analysis on MR, Stanford Sentiment Treebank (SST) and Hospital Feedback datasets where all word vectors were initialized by _Glove_. In a similar fashion, [PERSON] et al. [20] designed a CapsNet with the LSTM to identify toxicity and aggression in comments. In this paper, inspired by the success of CapsNets, we present a novel architecture designed for sentence classification, sentiment analysis in some cases, based on CapsNets. Unlike the approaches followed in the above works, we resort to using a cascade of simple convolution operations in parallel before the capsule layer to reduce the high dimensional data into lower dimensions and to encode different features using varying kernel sizes. The cascade of parallel convolution layers is implemented with relatively larger kernel sizes compared to [4; 5; 19], as larger kernel sizes can be useful to incorporate more information with large receptive fields. While multi-task learning was employed in [19] to increase the training data size by sharing knowledge among tasks, we implemented a much cheaper alternative using the _Word2 vec_ model for data augmentation, in addition to its usual use of encoding text into vectors, to increase our dataset size. Even though combining CapsNets and RNNs in [20; 21] exhibits encouraging results, the added use of RNNs makes them more complicated compared to the approach we proposed here. On top of that, since CapsNets can capture contexts and encode the sequences well, the benefits one can get by incorporating RNNs with CapsNet would be of marginal importance. Authors in [5] argued that, due to high variability in sentence construction, the model should be flexible enough to accommodate modifications such as changing the order of words. One of the reasons why they chose static routing over dynamic routing was to bring about this flexibility. Even though this claim may be valid at times, we argue that the exact order of words playsan important role in the overall meaning of a sentence as changing the order of words in a sentence usually results in a completely meaningless sentence or another sentence with a totally different meaning.