# A Tensor Space Model-Based Deep Neural Network for Text Classification
## 3 Concept-Driven Deep Neural Networks for Text Classification
### Embedding of Concepts and Documents  
Broadly, our neural network for text classification features two front layers for concept and document embedding; each concept is embedded by inputting the corresponding concept vector from the term-by-concept matrix into two fully connected layers, as shown in Figure 5; document embedding then proceeds by collecting the concept embeddings (as when creating a concept space). Unlike the Bag-of-Words and Word2 Vec models often used for machine learning-based text classification, our TSM contains an additional concept space, as well as a term space. Thus, it is inappropriate to deliver document data collected by the TSM to  
Figure 4: Generating a concept-by-term-by-document tensor. Figure 5: The concept embedding process. conventional DNNs that handle only term-based data. Thus, we added concept embedding to our new concept-driven DNN. As shown in Figure 5, concept embedding proceeds by dividing the TSM text matrix by concept and then passing the split vectors through two fully-connected layers. The collection of all concept embeddings is termed 'document embedding', as shown in Figure 6. Below, we describe a new DNN featuring concept and document embedding. Let \(d_{i}\) be the term-by-concept matrix of a document \(i\). Then, a \(j\)th concept vector \(c_{j}\) within \(d_{i}\) is expressed as \(c_{j}=d_{i}[:,j]\). Concept embedding \(E_{c_{j}}\) of the concept \(c_{j}\) proceeds as follows:  
\[y_{c_{j}}^{(1)} =f^{(1)}(w_{c_{j}}^{(1)}c_{j}+b_{c_{j}}^{(1)}), \tag{1}\] \[y_{c_{j}}^{(2)} =f^{(2)}(w_{c_{j}}^{(2)}y_{c_{j}}^{(1)}+b_{c_{j}}^{(2)}),\] (2) \[\cdots\] \[E_{c_{j}} =f^{(n)}(w_{c_{j}}^{(n)}y_{c_{j}}^{(n-1)}+b_{c_{j}}^{(n)}), \tag{3}\]  
where \(y_{c_{j}}^{(n)}\), \(w_{c_{j}}^{(n)}\), \(b_{c_{j}}^{(n)}\), and \(f^{(n)}\) denote the output value, the weight, the bias, and the activation function of the \(n\)th layer when performing the \(j\)th concept embedding, respectively. The above formula does not connect concepts when engaging in concept embedding. Thus, as each concept of the tensor space model is independent, concept embedding is performed individually. Next, in the document embedding step, the concept embedding vectors generated in the previous step are combined to create a two-dimensional document embedding matrix. The document embedding \(E_{i}\) for document \(i\) is expressed as follows:  
\[E_{i}=[E_{c_{1}};E_{c_{2}};\cdots;E_{c_{n-1}};E_{c_{n}}]. \tag{4}\]  
Algorithm 1 demonstrates the pseudo code for the concept and document embeddings described above. Input: A term-by-concept matrix \(d_{i}\) for a document \(i\)  
Output: A document embedding \(E_{i}\) for a document \(i\)  
```
1:\(nc=d_{i}.numOfConcepts()\);
2:for\(j\gets 0\) to \(nc\)do
3:\(c_{j}\gets d_{i}[:,j]\); // \(c_{j}\) denotes the \(j\)th concept (column) in the matrix \(d_{i}\). 4:\(outL_{1}\gets L_{1}.output(c_{j})\); // \(L_{k}\) denotes the \(k\)th layer for concept embedding. 5:\(outL_{2}\gets L_{2}.output(outL_{1})\);
6:



### The TSM-Based Deep Learning Architecture for Text Classification

Basically, our proposed neural network embeds text documents expressed in a term-by-concept matrix for each concept, and the combined document embedding is submitted to fully connected layers. In this section, we describe three types of autonomous neural networks appropriate for TSMs. Figures 7-9 show the new concept-driven DNNs that best accommodate our semantic TSM for text classification. * The first architecture ('TSM-DNN') is shown in Figure 7. In this architecture, the term-by-concept matrix for a given document is flattened and connected to form a single one-dimensional vector and then passed through multiple fully connected layers. Figure 8: TSM-based deep neural network II: Concept-wise TSM-DNN. Figure 5:0 TSM-based deep neural network I: TSM-DNN. Figure 5:1 TSM-based deep neural network III: Concept-wise TSM-CNN. The second architecture ('concept-wise TSM-DNN') is shown in Figure 5:2 This includes both the concept and document embedding layers introduced in Section 3.2.