# A Tensor Space Model-Based Deep Neural Network for Text Classification
## 3 Concept-Driven Deep Neural Networks for Text Classification
### The Semantic Tensor Space Model  
Our semantic TSM is a human knowledge-based, text embedding model that represents a document as a two-dimensional matrix of terms and concepts (Figure 1). A term is a word in a document, and a concept is a newly defined feature, thus being semantic information that humans use when thinking about a word. We use Wikipedia articles as the semantic units of documents. Basically, a concept is defined by a combination of an intent and an extent in the theory of formal concept analysis (FCA) [19]; the 'extent' is the set of instances included in the concept, and the 'intent' is the set of common attributes of the instances included in the extent. In our work, the extent that represents a concept consists of a set of documents related to the concept; the intent is a set of keywords extracted from that set of documents. As shown in Figure 1, by combining the concepts of FCA theory and the conventional document representation term-by-document matrix, we build a concept-by-term-by-document 3 rd-order tensor. In other words, the 'concept' space is regarded as an independent space with the same status as the 'term' and 'document' spaces. As mentioned above, in our TSM model, a document is represented by a 2 nd-order tensor of term and concept; the tensor is implemented by creating additional concept vectors. A concept vector expresses the relevance of each term with respect to all concepts inherent within a single document. Figures 2-4 show the detailed creation of a concept vector. The first step is to create a concept window based on the Lesk algorithm (or the 'gloss overlap') [20]. Basically, terms that occur in similar contexts may have similar meanings, and terms occurring with a target term can be considered to enrich the sense of that term. Hence, we first create a sliding window that is used to evaluate all concepts for a target (center) term, while considering the surrounding terms. Figure 2 shows the concept window for the target term 'java' with its surrounding words. The concept window is composed of the 'java' with surrounding terms, such as 'application', 'bytecode', 'virtual', and'machine'. Figure 3 shows how we weight the concept vectors. Although the target term 'java' has several senses (i.e., a language, a coffee, an island), the concept window allows us to derive its current context by examining all relevant concepts. To weight all concepts for a target term, the overlap between the concept window of a target term in context and the terms in a Wikipedia article corresponding to each concept is calculated. If each of the terms in the concept window occurs in a specific concept, a pre-calculated TF-IDF value for that term is added to its weight; for example, the weight of the concept?ava (programming language)' for the term?ava' becomes the average value (i.e., \(\frac{\sum_{i=1}^{N}u_{t}}{N}=\frac{10.01}{5}\approx 2.0\), where \(N\) denotes the number of terms within the concept window, and \(u_{t}\) denotes the TF-IDF value of the \(t\)th term.) of the TF-IDF values for five words included in the concept window centered on the term?ava'. Then, as shown in Figure 4, the several senses of a target term can be quantitatively represented as a concept vector, and as a result, a document as a concept-by-term 2 nd-order, and a set of documents as a concept-by-term-by-document 3 rd-order tensor. Furthermore, to represent a particular set of documents as the proposed 3 rd-order tensor, an appropriate concept space should be automatically constructed. Firstly, we select only about 250,000 pages with more than 100 incoming links to eliminate the low quality of Wikipedia articles. The selected articles are then indexed by a search engine, such as ElasticSearch, which calculates the TF-IDF weight of each term occurring in the selected articles. Next, we generate a search query that consists of only highly weighted terms within each of the articles.