### Contributions  
Based on the two key insights and T-PTLM, this study aims to achieve better text classification results by solving the problem that deep networks cannot fully extract location information from text expression and the feature fusion trouble of text location information and semantic information. Accordingly, this study proposes APTrans, a method designed to discover and utilize the internal semantic relationships and key information relationships in the text through the Transformer architecture. This method can discover the correlation between tokens by using the sub-attention method, and the core semantic features will be strengthened. Attention information learned about features can be visualized through vector similarity. Specifically, we have constructed a hierarchical feature extraction architecture. The backbone network extracts the associated information in the text and passes it to subsequent modules for processing. APTrans will fuse the semantic and positional information in the text to obtain the comprehensive representation. In summary, we have made the following contributions:  
* Three clues are obtained in the survey statements, including core relevance information, semantic location associations, and the mining characteristics of deep and shallow networks for different information. The core correlation information and semantic location associations confirm that the existence of key words in the text and the relative location information have an important impact on the effect of the final classifier. The other remaining clue ensures that we consider the lower layers' information as important as the high layers' information during the development of the model. Based on this, we propose a discriminant method, APTrans, for text classification to leverage our findings and address challenging environments.
* We reveal two key insights about sentences, namely, the key information relationship and word group inline relationship. In the proposed hierarchical deeper attention module, we improve the procedural computing mechanism of multi-head attention, enhancing the ability to extract the relationship between words by parallelizing attention computing processing. Moreover, the attention calculation of the text vector output from the backbone network is carried out again to ensure that the model can fully mine the key lexical semantic and location information in the text. The FFCon module transmits information through hierarchical iteration and shrinks text features through MLP, aiming to complete the fusion of multi-level semantic and positional information. Based on this, APTrans is able to solve the challenges caused by colloquial words or multi-semantic texts by mining and integrating the relative position information of texts, so as to achieve better classification results.
* APTrans has been shown to be efficiently applied to text classification in real NLP tasks through a large number of experiments and to perform comparisons on multiple text classification benchmark datasets. In addition, we demonstrate through modular ablation experiments that the two proposed modules, the hierarchical deeper attention module and FFCon module, have an important impact on the overall effect of the model. Furthermore, we conducted visualization experiments to prove that the model effectively leverages locational information for prediction.  
The rest of this article is structured as follows: Section 2 reviews some related work of text classification. Section 3 describes the proposed method APTrans, which has a feature mapping backbone, a hierarchical deeper attention module, a feature transfer fusion FFCon module, and a compressed connection classification head module. Section 4 presents the detailed experiments and results. Section 5 concludes our work.