## 5 Conclusions  
In this study, we propose a text classification method APTrans based on a hierarchical attention pyramid architecture to cope with the problem of being unable to fully extract text position information and word association information at the same time. We reveal the key information relationship and word group inline relationship in statements. To take advantage of these features in the text, the XLNet model is first selected as our backbone to extract long-dependency semantic information and coarse-grained location information in sentences. The mapping semantics of embedding and the implicit semantic variables of each layer in the backbone are extracted as inputs, and attention computing is further used to mine associated semantic information. Thereafter, the feature fusion module FFCon is established to input semantic information of higher layers to the lower layer for fusion calculation with coarse-grained position information to obtain multilayer implicit semantic features from the text. Finally, the compression connection module of APTrans compresses and cuts the hierarchical features and completes the feature stitching input classification header to calculate the label corresponding to the text. We test APTrans on multiple text classification benchmark datasets. The experimental consequence clearly shows that our model, APTrans, can efficiently integrate word position information and semantic information in the text and achieve good results in text classification. Meanwhile, the success of APTrans illustrates the importance of lexeme relationships in texts, which have probably been overlooked in previous studies. We hope that our preliminary study can stimulate further research on the positional and semantic relationship of words in NLP and further promote the development of this field.  
Conceptualization, G.J. and Z.C.; methodology, G.J. and Z.C.; validation, G.J.; formal analysis, G.J., Z.C. and H.L.; investigation, G.J., Z.C. and H.L.; resources, G.J. and Z.C.; data curation, G.J., Z.C. and H.L.; writing--original draft preparation, G.J.; writing--review and editing, G.J., Z.C. and H.L.; visualization, G.J.; supervision, Z.C. and H.L.; project administration, G.J., T.L. and B.W.; funding acquisition, Z.C., H.L, T.L. and B.W. All authors have read and agreed to the published version of the manuscript.  
This study was supported by the National Natural Science Foundation of China (grant nos. 62277026, 62377037, 62277041, 62211530433, 62177018), in part by the National Natural Science Foundation of Hubei Province project (nos. 2022 CFB529, 2022 CFB971), the Jiangxi Provincial Natural Science Foundation under grant (no. 20232 BAB212026), the university teaching reform research project of Jiangxi Province (grant no. JXJG-23-27-6), and the Shenzhen Science and Technology Program under grant no. JCYJ20230807152900001).  
**Institutional Review Board Statement:** Not applicable.  
**Informed Consent Statement:** Not applicable.  
\begin{table}
\begin{tabular}{c c} \hline \hline
**Methods** & **Pad Size** & **Acc (\%)** \\ \hline BigBird [52] & 77.34 \\ DeBERTa [53] & 73.69 \\ ALBERT [47] & 73.70 \\ ELECTTA [49] & 80.47 \\ GPT-2 [36] & 82.02 \\ XLNet [12] & 82.03 \\ RoBERTa [11] & 83.06 \\ BART [46] & 83.07 \\ ERNIE [48] & 83.07 \\ \hline APTrans (ours) & **83.33** \\ \hline \hline \end{tabular} The top performing values are shown in **bold**, and those following the top values by the underlining.  
\end{table}
Table 4: A comparison with some T-PTLM methods. The results of our experiments are presented as follows.  
Data Availability Statement: The datasets used in this article are available upon request from the corresponding author.  
The authors declare no conflicts of interest.