## 1 Introduction
### Observation and Insights  
In this study, we identified two insights through observation to facilitate text classification by leveraging the fusion of critical local contextual semantic and global semantic information. First, crucial semantics determining text categories often manifest in different positions across various text genres. For instance, Figure 1a,b depict that the critical information determining the category attribution in news articles typically appears at the beginning of paragraphs. Emphasizing the use of the initial paragraph information aids in determining the text categories. Nonetheless, the distinctions between electronic technology products and agricultural technology applications depicted in Figure 1a,b, respectively, are significant, necessitating a further category differentiation. Figure 1d,e show that the meanings conveyed in written and spoken expressions are roughly the same. In such cases, genre classification based solely on the overall semantics poses significant challenges, as discerning genre categories without a careful comprehension of each sentence becomes difficult. Without integrating full-text semantics with sentence-level semantics to confirm text categories in extensive samples, classification errors are prone to occur. Thus, confirming the approximate classification of texts through holistic semantic learning, combined with fine-grained learning based on sentence-level associations, facilitates the accurate prediction in samples with implicit nested label associations.  
Second, when critical text keywords are erroneous or missing, correction can be achieved by leveraging other semantic information within the entire text. This notion suggests that, in long-text learning, individual keywords alone cannot determine the overall semantics. Guiding classification with overall semantics can enhance accuracy. For example, Figure 1c demonstrates that the first word guiding the overall semantics in the original text is incorrectly recognized after being transcribed by a speech recognition system. Upon reading the entire text, the existence of this error becomes apparent and does not result in any difficulty in understanding the semantics.  
**Key insight I: Overall semantics guide understanding**  
In specific texts, situations arise where critical keywords are missing or erroneous. At such times, the overall semantics of the text are crucial for comprehension and are more robust than word-level and sentence-level semantic learning (Figure 1c). Although various methods exist for text learning, only the overall semantics can guide one's understanding, rectifying errors in word and sentence vectors that may disrupt classification due to internal critical information errors.  
**Key insight II: Joint semantic learning is robust**  
We find that word vector semantic learning and overall semantic learning have advantages in text feature learning. Accordingly, these two learning modes can serve as parallel structures to assist each other in feature learning. We term this mechanism joint semantic learning. When the sentence-level semantics and overall semantics coexist, extreme situations in text data can be addressed, as depicted in the three parts of Figure 1. In comparison with CNN and RNN architectures, Transformers possess notable long-distance dependency learning capabilities, rendering them suitable carriers for joint semantic learning.  
In this section, we define three levels of semantic representation units--\"overall semantics of the text\", \"word-level semantics\", and \"sentence-level semantics\"--and the specific meanings and purpose of these three levels of units will be described in Section 3.