# The Influence of Feature Representation of Text on the Performance of Document Classification
## 2 Background
### Document Representation Models
#### 2.1.2 Continuous-Space Models  
Continuous-space word representations (word2 vec) capture syntactic and semantic regularities in language from observing the patterns of words co-occurrence in the same context of neighboring words [26]. More specifically, word2 vec employs neural networks to model the relation between a given word and its context of neighboring words in the given collection of documents. The continuous bag-of-word model (CBOW) predicts the context (neighboring words) for a given word, while the continuous skip-gram model predicts a word given the context. Both lead to a vector-space embedding (representation) of individual words. The neural network prediction model uses a hierarchical softmax function whose structure is a binary Huffman tree and is trained using stochastic gradient descent and back propagation algorithms. To improve the computational efficiency of model training, negative sampling is used to reduce the number of distributed context vectors considered. This type of model is also referred to as a neural language model [27]. Recently, word2 vec has been shown to be successful in many natural language processing tasks ranging from sentiment analysis [8; 28; 29; 30], topic modeling [31], through document classification [32; 33] and name entity recognition [34] to machine translation [35; 36].  
High dimensional vectors also proved to be efficient on larger linguistic units, such as pieces of text of variable lengths (sentences, paragraphs or documents) resulting in paragraph2 vec and doc2 vec models [7; 37]. The doc2 vec models are able to predict word occurrence in the context of paragraphs or documents. Hence, doc2 vec has been shown to be efficient in sentiment analysis [7; 38; 39], information retrieval [7], document classification [9; 10], summarization [40] and question answering [41]. In our study, we use the variants of the word2 vec and doc2 vec models that correspond to different sizes of the feature vectors extracted from the continuous space transformation.  
Recent studies of the continuous space models for document classification have been primarily focused on the exploration of one isolated aspect of the system, usually comparing different classifiers and contrasting different representation models of documents against bag-of-words as a baseline model and word2 vec (doc2 vec as well, although rarely) as the suggested improvement. For example, [PERSON] et al. [38] compare the TF and TFIDF variants of the bag-of-words model with a doc2 vec model in the task of hate-speech detection. They show that doc2 vec model outperforms both bag-of-words variants in terms of the area under the receiver-operating characteristic curve obtained with a classifier based on linear regression. Similarly, [PERSON] [39] shows that the doc2 vec model outperforms TF and TFIDF variants of the bag-of-words model in the sentiment analysis tasks in Thai and English languages, regardless of the used classifier (logistic regression, naive Bayes or support vector machines). [PERSON] et al. [42] show the combination of the bag-of-words and the continuous space models lead to a marginal performance improvement over the alternatives in the sentiment analysis task. [PERSON] et al. [10] show that multi-co-training of TFIDF, latent Dirichlet allocation and doc2 vec achieves more robust results in document classification tasks especially with very low dimensions of features' vectors. [PERSON] and [PERSON] [43] propose the fusing of different streams of features in a multi-stream neural model, while [PERSON] et al. [44] introduce the self-interaction attention mechanism for aggregating the integration strategies into a hierarchical structure for the document classification.