# Self-Interaction Attention Mechanism Based Text Representation for Document Classification
## 1 Introduction  
Document classification, as a challenging task in the field of Natural Language Processing (NLP), typically assigns one or more class labels to a document according to its subject or other attributes, e.g., author and topic. Generally, it has a broad application in the area of sentiment classification [1; 2], document ranking [3], genre classification [4] and topic labeling [5], etc. Traditional approaches on document classification mainly label the document according to a relevance of the document to a class label, which is estimated based on the statistical indicators, e.g., the frequency of co-occurrence words (the bag-of-words model [6]), the frequency of the word pair (the n-grams model [7]) and the weight scores of each word in different documents (the TF-IDF model [8]). However, such statistical-based methods typically suffer from the problem of data sparsity and dimensionality explosion when they are applied to a large-scale corpus. To deal with this, neural-based approaches are proposed by learning distributed representation [9; 10; 11; 12]. The neural-based models generally follow a so-called one-way action. That is to say, representations generated forthe preceding text segments are taken as the context to determine the representations of following text segments. Instead, we argue that the semantics of a text segment is a product of interactions of all text segments in a document. Restrictions to one-way action may result in a partial semantic loss. Although these interaction relations may be learned by neural networks with enough samples, modeling such interaction relations can directly make document representation more informative and effective. In order to learn such interaction, we propose a text representation model with Self-interaction Attention Mechanism (TextSAM)-based text representation for document classification. We illustrate our Self-interaction Attention Mechanism in Figure 1b. The idea of an _action_ of a text segment on another segment is that the former assigns a semantic weight to the latter. Standard attention mechanism [13] used in text representation (Figure 1a) typically introduces a context vector by a random initialization [14] (external input) as the _action controller_ to get source elements semantics weights that make up the document representation. That is to say, standard attention mechanism is a one-way _action_ between the context vector and source elements. In contrast, our self-interaction attention mechanism (Figure 1b) resorts to all source elements (without external input) in a document as the _action controllers_. All source elements that are regarded as the _action controllers_ are equivalent to the interaction between source elements. In detail, we design three aggregated strategies to integrate the interaction into a hierarchical architecture for document classification, i.e., averaging the interaction, maximizing the interaction and adding one more attention layer on the interaction, which leads to three models, i.e., TextSAM\({}_{AVE}\), TextSAM\({}_{MAX}\) and TextSAM\({}_{ATT}\), respectively. Our experimental results prove the effectiveness of our proposals. Figure 1: Comparison of Two Attention Mechanisms. The main contributions of our work are summarized as follows:  
* To the best of our knowledge, ours is the first attempt to model the interactions between source elements in a document;
* We propose a Self-interaction Attention Mechanism (TextSAM) to produce the interaction representation in a document for classification. * We introduce three aggregated strategies to integrate the interaction into a hierarchical structure, generating three models for document classification, i.e., TextSAM\({}_{AVE}\), TextSAM\({}_{MAX}\) and TextSAM\({}_{ATT}\), respectively. * We conduct comprehensive experiments on two large-scale public datasets (Yelp 2016 and Amazon Reviews (Electronics)) for document classification.