# Design and Investigation of Capsule Networks for Sentence Classification
## 2 Related Work  
Over the years, a large body of research work was conducted on sentiment analysis, or more generally on sentence classification tasks. Earlier non-neural network based approaches focused on topical categorization, attempting to sort documents according to their subject matter. However, applications like business intelligence or other product review require more sophisticated and in-depth analysis of the opinions presented than topical categorization. Other approaches have partially been knowledge-based, using linguistic heuristics or a pre-selected set of seed words as in the work by [PERSON] et al. [7]. The authors used Naive Bayes classification, maximum entropy classification and support vector machines (SVMs) to perform sentence classification on a movie reviews (MR) dataset, producing moderate results. The problem with these approaches is that they require a priori knowledge or pre-selected seed words. Other popular approach for sentence classification, question classification specifically, is to use rule-based classifiers. [PERSON] et al. [8] used a rule-based classifier with an SVM to classify Text REtrieval Conference (TREC) questions. Even though they were able to record state-of-the-art performance using this approach, the use of manually built patterns makes it unfit for practical uses, especially when there are large and varied datasets. In the work by [PERSON] et al. [9], semantically meaningful vector representations from sentences and phrases of variable size were extracted using combinatorial category grammar (CCG) operators and each of these vector representations were fed to an autoencoder function for classification. Genetic algorithms (GAs) were also used for automatic classification of text sentiment in the work by [PERSON] et al. [10]. In their work, which the authors referred to as a genetic algorithm for sentiment analysis (GASA), they proposed a GA approach to classify sentences by optimizing unknown words as either a sentiment or an amplifier word which was able to outperform certain systems. [PERSON] [1] proposed a shallow CNN network for sentence classification which has single convolution and pooling layers. The author compared the performance by applying random embedding (CNN-rand), pre-trained Word2 vec embedding (CNN static), and pre-trained Word2 vec embedding with fine-tuning using back propagation (CNN non-static) on common datasets. The reported results show superior accuracy for most datasets used in the experiment. Moreover, the author argued that a pre-trained Word2 vec model can be used as a general-purpose choice for text embedding. Another notable work for sentence modeling is a work by [PERSON] et al. [11] which used a deep CNN with dynamic k-max pooling that can handle sentences of varying lengths. [PERSON] et al. [12] also adopted a CNN for multimedia sentiment analysis consisting of texts and images. Yet another CNN architecture by [PERSON] et al. [13] was used for text understanding from character-level inputs all the way up. Whilemost CNN models for text classification are shallow, [PERSON] et al. at Facebook-AI [14] came up with a very deep CNN architecture showing that the performance of this model increases with the depth. Though these CNN based models produced notable results, we proved that a well tuned CapsNet based models can do better. By combining a recurrent structure from RNNs and max-pooling from CNNs, [PERSON] et al. [15] proposed an RCNN model for text classification which produced good results. The recurrent structure, a bi-directional RNN with word embeddings, is utilized to capture contexts in both directions of a given word and the output is propagated through a max pooling layer. The long-short-term-memory-network (LSTMN) by [PERSON] et al. [16] was implemented to perform shallow reasoning with memory and attention. This work extended the LSTM architecture with a memory network in place of a single-memory cell which enables adaptive memory usage during recurrence with neural attention.