# Long Text Truncation Algorithm Based on Label Embedding in Text Classification
## 3 Method
### Label Embedding  
Label word representation methods mainly include one-hot embedding representation, static word vector representation (such as word2 vec [26]), and word vector extraction based on the pretrained BERT model. In comparison, one-hot embedding and static word vectors find it difficult to capture contextual information, while the word vector extraction method based on pretraining model finds it difficult to capture the interaction information between long text and label words. In order to better obtain the vector representation of label words and alleviate the problem of polysemy, the method of constructing a prefix template is improved on the basis of the BERT model. Figure 2 shows the whole framework of label embedding. As can be seen from Figure 2, this framework consists of two steps. **Step 1: Label word pretraining based on prefix template constructing**  
Firstly, we construct training sample pairs which shows as:  
\[T=[(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{i},y_{i}),\ldots,(x_{m},y_{m})],\]  
where \(x_{i}\) is the long text, and \(y_{i}\) is the corresponding label. Secondly, for each sample pair \((x_{i},y_{i})\), define the pretraining prefix template \(pt_{1}\) as:  
\"This is an article about: \([y_{i}].x_{i}\)\". Figure 2: The whole framework of label embedding. In this figure, [CLS] is the sentence vector representation slot, which outputs the sentence vector representation. [label] is the label word filling slot, which fills the label word to associate the label word with the long text. Thirdly, fill the training sample pair data into the prefix template to obtain the new text data \(\mathit{Text}_{new}\), which contains the original long text and label words. Finally, the new text data \(\mathit{Text}_{new}\) are input into the BERT model for in-domain pretraining to fully learn the interaction information between long text and label words. Through in-domain pretraining, we obtain a pretrained model \(H\), and the steps of in-domain pretraining will be described in detail in Section 3.3.1. **Step 2: Label word representation**  
In order to obtain the vector representation of each label word, the forward propagation method is used to predict the label word vector based on the BERT model trained in the previous step. Firstly, construct a new label word mapping prefix template \(\mathit{pt}_{2}\) as below:  
\"This is an article about: \([y_{i}]\).\"  
In this step, by constructing a label word mapping prefix template, richer context information is introduced into the label word, and the semantic range corresponding to the label word is limited, so that the problem of polysemy can be alleviated. Secondly, as shown in Figure 2, the prefix template containing the label word is input into the pretrained model \(H\). Then, we can obtain the embedding of each word in the prefix template. Finally, due to the unpredictability of the text category during word segmentation, a label word may occupy multiple characters, so we simultaneously record the start and end positions of the label word: \(label_{start}\) and \(label_{end}\). For all vectors in the range of \(label_{start}\) and \(label_{end}\), mean pooling is conducted to obtain the embedding of each label word. In addition, in order to make the reader understand this process more clearly, we summarize the process of label embedding in Algorithm 1. ```
0: The traning sample pairs \(T=[(x_{1},y_{1}),(x_{2},y_{2}),\ldots,(x_{i},y_{i}),\ldots,(x_{m},y_{m})]\); The pretraining prefix template \(\mathit{pt}_{1}\) and the label word mapping prefix template \(\mathit{pt}_{2}\). 0: Label word vector \(C=[C_{1},C_{2},\ldots,C_{i},\ldots,C_{m}]\). 1: Fill the sample pair consisting of \(x_{i}\) and \(y_{i}\) into the \(\mathit{pt}_{1}\) to obtain new text data \(\mathit{Text}_{new}\). 2: Input \(\mathit{Text}_{new}\) into the BERT model for in-domain pretraining, and obtain the pretrained model \(H\).