## 3 Model Design
### Our Model  
As shown in Figure 4, the modeling process of this paper is divided into text preprocessing, data enhancement, feature extraction, and training.  
1. The goal of text preprocessing is to turn the text into an analyzable and predictable form, and there are a large number of special symbols, tone auxiliaries, and other information irrelevant to classification in the textbook dataset. These noises increase the computational overhead of the model, so they should be removed as much as possible; thus, this paper uses the Jieba [18] word division tool and self-built deactivation word list to remove the noise from the textbook;
2. A balanced dataset has a very important impact on the classification performance of the model. In view of the serious imbalance of the Shanghai textbook dataset, this paper combines the ideas of SMOTE and EDA to achieve the oversampling of  
Figure 3: Mapping from a query to a set of key–value pairs.  
a few categories, and to randomly disrupt the order of sentences and words within sentences with a certain probability, so as to obtain a balanced new dataset;
3. In this paper, we use the negative sampling model Skip-Gram in Word2 vec [19; 20] to generate word vectors and map the reprocessed words into low-dimensional vectors;
4. The model structure of this paper is shown in Figure 5. Unlike the normal TextCNN structure, the attention layer is introduced between the input layer and the convolutional layer in order to create a context vector for each word. Then, the context vector and the original pre-training of the word are spliced as the new representation of the word, which is input to the convolutional layer and then passed through the pooling layer and the full concatenation layer to obtain the class of the text.  
The objective function in this paper uses the cross-entropy loss function with a faster convergence rate. The loss function is as follows:  
\[\text{Loss}=-\sum_{i=1}^{n}y_{i}\log(\hat{y}_{i}) \tag{6}\]  
Figure 4: The modeling flow of this paper.  
Figure 5: Model structure of this paper.  
where \(y_{i}\) is the true classification result of the \(i\)-th label and \(\hat{y}_{i}\) is the prediction result. Then, the loss of each batch is:  
\[\text{Loss}=\sum\limits_{j=1}^{m}\sum\limits_{i=1}^{n}\bigl{(}-y_{ji}\log \bigl{(}\hat{y}_{ji}\bigr{)}-\bigl{(}1-y_{ji}\bigr{)}\log\bigl{(}1-\hat{y}_{ji} \bigr{)}\bigr{)} \tag{7}\]  
where \(m\) is the number of samples in the current batch, \(n\) is the number of categories, and \(\hat{y}_{ji}\) is the predicted label of each node.  
Denoting the context vector in the graph by \(\mathbf{g}_{i}\), the attention mechanism is additionally an MLP (multilayer perceptron network) that determines which words should receive more attention when predicting sentences, i.e.,:  
\[\mathbf{g}_{i}=\sum\limits_{j\
eq i}\alpha_{i,j}\cdot x_{j} \tag{8}\]  
where \(\alpha_{i,j}\) is called the attention weight and requires \(\alpha\geq 0\) and \(\sum\limits_{j\
eq i}\alpha_{i,j}\cdot x_{j}=1\), which can be achieved here by softmax normalization:  
\[\alpha_{i,j}=\frac{\exp\bigl{(}\mathrm{score}\bigl{(}x_{i}\cdot x_{j}\bigr{)} \bigr{)}}{\frac{\sum\limits_{j}\exp\bigl{(}\mathrm{score}\bigl{(}x_{i}\cdot x _{j}\bigr{)}\bigr{)}}{\sum\limits_{j}\exp\bigl{(}\mathrm{score}\bigl{(}x_{i} \cdot x_{j}\bigr{)}\bigr{)}}} \tag{9}\]  
where the score value is calculated by the MLP mentioned above.