# Long Text Truncation Algorithm Based on Label Embedding in Text Classification
## 3 Method
### Long Text Filtering Based on Similarity Calculation
#### 3.2.2 Sentence Vector Filtering and Text Reorganization
Combining the dataset \(P\) and the original category label data, we can construct the finetuning dataset \(D=[(x_{1}^{\prime},y_{1}),(x_{2}^{\prime},y_{2}),\ldots,(x_{i}^{\prime},y_{i}) \ldots,(x_{m}^{\prime},y_{m})]\). There are three steps in the model finetuning:

1. Input finetuning dataset into BERT model loaded with the pretrained model \(H\). 2. Calculate the cross-entropy loss between the output [CLS] of the model hidden layer and the real category label. 3. Update model parameters according to step 2. In addition, the specific calculation equation of cross-entropy loss is:

\[H(p,q)=-\sum_{i=1}^{n}p(x_{i})\log(q(x_{i})), \tag{4}\]

where \(p(x_{i})\) and \(q(x_{i})\) refer to the probability distributions of the true category label and predicted category label of the \(i\)-th text, respectively, and \(H(p,q)\) is the value of cross-entropy. After finetuning, we obtain the finetuned model \(N\). #### 3.3.3 Prediction

The LTTA-LE algorithm needs to filter long text according to label data. However, due to the lack of label data in the test set, the label data cannot be used to filter the noise information in the prediction stage. Therefore, we need to take advantage of the pseudo-label method [28] to make predictions on the test set and then complete the filtering of the test set based on the obtained pseudo-labels. The specific implementation steps are as follows. In order to make the length of the test set meet the input requirements of the BERT model, we first shorten the test set by the head + tail truncation method, i.e., keep the fixed-length words in the head and tail of the test set as the input of the BERT model. Then, we use the model \(N\) trained in the finetuning stage to make predictions on the test set to obtain the predicted pseudo-labels. Based on our proposed LTTA-LE algorithm, the test set is filtered according to the predicted pseudo-labels, and new shorter texts are obtained after filtering. Use the model \(N\) trained in the finetuning stage to predict the new shorter text and obtain the final predicted category label of the test set. Through the above steps, the problem of missing labels in the test set can be solved and the performance of the model can be significantly improved. Furthermore, the model training process including the above three stages is summarized in Algorithm 3. ```
0: New shorter text \(P=[x_{1}^{\prime},x_{2}^{\prime},\ldots,x_{i^{\prime}}^{\prime},\ldots,x_{m}^ {\prime}]\); Label dataset \(Y=[y_{1},y_{2},\ldots,y_{i^{\prime}},\ldots,y_{m}]\). 0: The predict label. 1: Input \(P\) into the BERT model for in-domain pretraining, and obtain the pretrained model \(H\). 2: According to \(P\) and \(Y\), build the finetuning dataset \(D=[(x_{1}^{\prime},y_{1}),(x_{2}^{\prime},y_{2}),\ldots,\linebreak(x_{i^{ \prime}}^{\prime},y_{i})\ldots,(x_{m}^{\prime},y_{m})]\). 3: Input \(D\) into model \(H\) for finetuning, and obtain the finetuned model \(N\). 4: Use \(N\) to make predictions on the test set and obtain pseudo-labels. 5: Based on the LTTA-LE algorithm and the predicted pseudo-labels in step 4, the long texts in the test set are filtered to obtain new shorter texts. 6: Use \(N\) to predict the new shorter text in step 5 to obtain the final predict label of the test set. ```

**Algorithm 3** Model training algorithm. ## 4 Experiments

### Datasets

In order to verify the performance of our proposed model, we conduct algorithm validation on three public datasets: Arxiv Dataset [29], 20 NewsGroup [1], and IMDB Review [30]. #### 4.1.1 Arxiv Dataset

ArXiv Dataset is the largest open-source academic paper sharing platform, which has collected more than 1.7 million scientific and technological papers. We randomly select 50,000 sample data from it (90% for training and 10% for testing), and the number of data categories is 39. In order to verify the performance of our proposed model under the condition of long text, we only keep the abstract of each academic paper and its corresponding category label.