# Research on Feature Fusion Method Based on Graph Convolutional Networks
## 2 Related Work  
Due to the remarkable success of deep learning neural networks [15; 16], text categorization methods based on deep learning have garnered significant attention from researchers. Among these, the BERT-based text classification model segments the text into words, converts each word into a corresponding word vector or embedding vector, and incorporates positional encoding into each word vector before inputting it into the Transformer model. On the other hand, Graph-Neural-Network-based text classification models extract rich and effective relational structure information by constructing a heterogeneous graph representation of the text. This section will explore the mainstream models of these approaches. BERT-based text classification model: BERT-based text classification models focus on leveraging the self-attention mechanism within the Transformer architecture of BERT to capture semantic information in text sequences and apply it to text classification tasks. These models typically encode text sequences into context-aware representations and then utilize these representations for classification prediction. Throughout this process, the models can efficiently handle long text sequences and often achieve good performance after pre-training on large corpora. The model structure is depicted in Figure 1.  
The Transformer model, proposed by [PERSON] et al. in 2017 [17], revolutionized natural language processing. This model computes the weight of each word in a sentence in parallel via a self-attention mechanism, learning word representations with attentional weights, and implements the text classification task by appending an output layer at the end of the encoder. Leveraging the attention mechanism, the Transformer achieves higher parallelization compared to a CNN and RNN [18], enabling efficient training of very large models on GPU clusters. Subsequently, pre-trained language models (PLMs) emerged, leveraging the robust word representation capabilities of pre-trained language models to achieve promising results in text categorization tasks. Existing pre-trained language models fall into two categories: autoregressive pre-trained language models and self-encoding pre-trained language models. GPT [19] stands as one of the earliest autoregressive pre-trained language models, comprising unidirectional Transformers renowned for their adeptness in capturing contextual semantic information. BERT, on the other hand, diverges from GPT's unidirectional structure, adopting a bidirectional pre-training approach coupled with the Mask masking mechanism during training. Owing to BERT's potent representational prowess, numerous researchers have endeavored to enhance the model to suit diverse learning tasks. RoBERTa [20] employs augmented training data, larger parameter sizes, and batch sizes to refine the model. It also integrates a dynamic masking mechanism, resulting in a more robust final model compared to BERT. ALBERT [21], on the other hand, streamlines the BERT architecture to boost training speed and reduce memory consumption through parameter reduction. DistillBERT [22] employs knowledge distillation techniques during pre-training, eliminating token-like word embeddings and pooling layers to halve the model's parameters while retaining vital information, thus enhancing training speed and simplifying the BERT network's structure. ERBERT [23] utilizes a dynamic masking mechanism in model training, whereas ERNIE integrates domain knowledge from external knowledge bases into pre-trained language models. ALUM [24] introduces adversarial loss during model pre-training, bolstering the model's generalization capabilities for new tasks and robustness against adversarial attacks. Finally, XLNet [25] amalgamates the strengths of autoregressive and autoencoder models, incorporating sorting operations during pre-training to imbue the model with sensitivity to location information.