# A Tensor Space Model-Based Deep Neural Network for Text Classification
## 3 Concept-Driven Deep Neural Networks for Text Classification
### Embedding of Concepts and Documents
In other words, after creating a concept embedding vector for each concept, all concept embedding vectors are collected in two-dimensional form to create a document embedding. * The third architecture ('concept-wise TSM-CNN') is shown in Figure 5:3 This is similar to the second architecture, except that multiple convolutional layers are added after the document embedding layer. In general, convolutional layers afford high performance when used to classify two- or three-dimensional image data. As document embedding is two-dimensional, we expected that classification would be improved on inclusion of convolutional layers. The width of the first such layer equals the length of document embedding, and the filter size is set to 3. When the number of convolutional layers is excessive, the number of hyper-parameters is too large to optimize. Hence, it is desirable to control the number of convolutional layers using one-dimensional Max-pooling with the stride set to 2. In short, all three deep learning architectures perform text classification using fully connected layers that employ a document embedding matrix as an input. Unlike existing deep learning-based classification methods that employ units of terms, our proposed architectures fully utilize the conceptual information contained in the TSM. The TSM-DNN is connected to a basic neural network by concatenating the vectors of concept units, and the concept-wise TSM-DNN and concept-wise TSM-CNN create and combine new concept embedding vectors and then forward them to fully connected networks. ## 4 Experiments

### Empirical Setup

For reliable evaluation of the proposed method, we used three controlled subsets: 20 Newsgroups, Reuters-21578, and OHSUMED document corpora, which are known to be clean and are commonly used to evaluate various text classification methods. The 20 Newsgroups corpus contains approximately 20,000 documents partitioned across 20 different newsgroups. We selected 10,000 of the top-ranked largest documents. Next, we generated a subset of the Reuters-21578 corpus so that documents can be not skewed over classes. For this, we first selected documents belonging to the the five most common classes and then chose the 10,000 largest documents with a single class (thus, we avoided the ambiguities of documents on multiple classes). Finally, we used the OHSUMED corpus that is a subset of clinically-oriented MEDLINE database and includes medical abstracts from the 23 Medical Subject Headings (MeSH) categories. Many studies have shown that this corpus is difficult to classify properly; thus, it is very challenging to achieve a high classification performance. From this corpus, we selected 3600 documents of the nine most common classes. To ensure that performance evaluation was reliable, we performed 5-fold cross-validation. For comparison, we selected six conventional methods, of which two were traditional machine learning methods, including an SVM and naive Bayes, and the others were recent deep learning techniques, including a DNN [21], a CNN [22], and a VDCNN [3]. The traditional machine learning methods use TF-IDF values as inputs after pre-processing textual documents. For the SVM, the Radial Basis Function (RBF) served as the kernel, and the penalty parameter \(C\) (for misclassification) was set to 1.0. We used a Multinomial naive Bayes appropriate for textual data, and the parameter \(\alpha\) for Laplace smoothing was set to 1.0. When using the DNN, textual documents were transformed into Bag-of-Words inputs. The DNN architecture featured seven fully connected layers, with as many neurons as input words, 1024, 2048, 4096, 1024, 256, and two neurons, respectively. ReLU served as the activation function, and the Softmax function was applied by the last layer for text classification. The DNN architecture was trained using the Adam optimization method with a batch size of 64 over 10 epochs. The CNN architecture featured three fully connected layers lying after two convolutional layers and two Max-pooling layers (in that order).