# A Tensor Space Model-Based Deep Neural Network for Text Classification
## 3 Concept-Driven Deep Neural Networks for Text Classification
### Embedding of Concepts and Documents
The kernel size of the first convolutional layer was set to 3, the number of filters to 8, and the zero-padding to 1. For the second convolutional layers, the figures were 3, 32, and 1. For the Max-pooling layer, the kernel size was 3, the zero-padding 1, and the stride 2. The outcomes of the layers were flattened into one-dimensional forms and then submitted to the fully connected layers. The three such layers featured 512, 64, and two neurons (in that order). Text classification proceeded using ReLU as the activation function and with application of the Softmax function to the output of the last layer. The CNN architecture was trained using the Adam optimization method with a batch size of 32 over 10 epochs. The VDCNN [3] features four depths; we selected depth17 and depth29 because they afforded the best performance. The VDCNN classifies documents by inputting text data in character units; the data are not transformed into Bag-of-Words format (or the like). The maximum number of document characters is 1014; thus, the input data size is fixed. Momentum was employed for optimization; the learning rate was 0.01; and the batch size was 64 over 50 epochs. ### Tuning the Deep Learning Architectures

We now focus on various parameters of our new deep learning architectures for text classification. First, to create TSM tensors, we set the size of the concept space to 30, and extracted 1000\(\sim\)5000 meaningful terms from each dataset. For example, if 1000 terms were selected, each document was represented as a 1000 \(\times\) 30 matrix. In the TSM-DNN architecture, the fully connected layers featured 1024, 2048, 512, and two neurons (in that order). We employed the LeakyReLU activation function, and performed batch normalization (to avoid overfitting) after each fully connected layer generated an output. The output of the last fully connected layer was submitted to the Softmax function for text classification. In the concept-wise TSM-DNN architecture, the two fully connected layers for concept embedding featured 256 and 128 neurons, respectively. As a result, document embedding was in the form of a two-dimensional tensor of size 128 \(\times\) 30. Then, the document embedding tensor was flattened and connected to four fully connected layers with 2048, 512, 128, and two neurons (in that order). Batch normalization was performed after each fully connected layer created an output, and LeakyReLU served as the activation function. In the last fully connected layer, text classification was performed using the Softmax function. The concept-wise TSM-CNN employs convolutional layers to reduce the size of the feature map; thus, the concept embedding size was quadrupled to avoid information loss. Therefore, the numbers of neurons in the two fully connected layers for concept embedding were set to 2048 and 512, respectively, so that the size of document embedding became 512 \(\times\) 30. The document embedding then passed through a one-dimensional convolutional layer of kernel size 3, zero-padding 1, and with 64 and 128 filters. Each convolutional layer was connected to a one-dimensional Max-pooling layer with a kernel size of 3, a stride of 2, and zero-padding of 1. The four fully connected layers featured 2048, 512, 128, and two neurons, respectively. The concept-wise TSM-CNN featured batch normalization to avoid overfitting; the activation function was ReLU. In the last fully connected layer, text classification was performed using the Softmax function. All of our architectures used Adam for optimization; the learning rate was 0.01, and the mini-batch size was 32. ### Performance Evaluation

As shown in Table 1, the classification models trained using our architectures performed best; the classifications of the Reuters-21578 and 20 Newsgroup data were near-perfect.