# A Tensor Space Model-Based Deep Neural Network for Text Classification
## 3 Concept-Driven Deep Neural Networks for Text Classification
### Embedding of Concepts and Documents
The proposed neural network architectures, including concept embedding, contributed greatly to constructing a high-performance text classification model; the concept-wise TSM-DNN classification model and the TSM-DNN classification model were evaluated to have almost the same classification performance, and the concept-wise TSM-CNN classification model showed the best performance. This experimental result implies that, as the meaning of terms appearing in a given document became clearer through concept embedding, our deep learning architecture dramatically improved the text classification performance. Moreover, it suggests that the convolutional layers within the concept-wise TSM-CNN can further enhance the effectiveness of concept embedding and document embedding. Similar to why CNN architecture is very effective for image classification, we analyze that the concept-wise TSM-CNN model showed the best performance by extracting meaningful local patterns that help text classification from a document embedding matrix with a structure similar to image data. The accuracies of the comparative methods were over 90% for the Reuters-21578 data. In contrast, for the OHSUMED data, which is difficult to classify automatically, our methods were much better than the other methods. Furthermore, although we used fewer convolutional layers than the DNN, CNN, and VDCNN, our architectures exhibited high-level classification accuracy. The VDCNN classifications of the OHSUMED datasets were very poor; the classification accuracy severely decreased when the number of selected terms increased. When using the TSM-CNN architecture, it is necessary to define an optimal number of convolutional layers. Figure 10 shows the changes in classification accuracy when the number of convolutional layers for the OHSUMED dataset was varied in the concept-wise TSM-CNN. The classification accuracy for the OHSUMED data varied dramatically by the number of layers. A concept-wise TSM-CNN with eight convolutional layers afforded the best accuracy (85.2%); classification performance decreased with increasing numbers of layers. We conjecture that the classification model remains under-fitted because the training data do not increase when the classifier capacity increases to create a deeper network. A neural network with more than eight convolutional layers is of large capacity, associated with overfitting of the training data and, thus, poorer classification performance. The optimal layer number may differ for different sets of training data; however, we found that the optimal number was eight for all 3 datasets. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline  & **SVM** & \begin{tabular}{c} **Naïve** \\ **Bayes** \\ \end{tabular} & **DNN** & **CNN** & \begin{tabular}{c} **VDCNN** \\ **Depth-17** \\ \end{tabular} & \begin{tabular}{c} **VDCNN** \\ **Depth-29** \\ \end{tabular} & \begin{tabular}{c} **TSM** \\ **DNN** \\ \end{tabular} & \
\begin{tabular}{c} **Concept-Wise** \\ **TSM-DNN** \\ \end{tabular} &
\begin{tabular}{c} **Concept-Wise** \\ **TSM-CNN** \\ \end{tabular} \\ \hline Reuters-21578 & 0.9073 & 0.9104 & 0.9700 & 0.9823 & 0.9853 & 0.9853 & 0.9952 & 0.9956 & 0.9965 \\ \hline
20 Newsgroup & 0.9245 & 0.8792 & 0.7220 & 0.8653 & 0.9216 & 0.9290 & 0.9597 & 0.9562 & 0.9667 \\ \hline OHSUMED & 0.7665 & 0.7279 & 0.5546 & 0.6574 & 0.5954 & 0.4898 & 0.8400 & 0.8461 & 0.8522 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Performance comparisons in terms of accuracy. Figure 10: Changes in classification accuracy on variation of the number of convolutional layers in the concept-wise TSM-CNN. ## 5 Conclusions

We developed new, concept-driven deep learning architectures optimized for our semantic TSM of text representation, and we tried to improve text classification. We first introduce concept and document embedding using a Wikipedia-based representative TSM model.