# GoalBERT: A Lightweight Named-Entity Recognition Model Based on Multiple Fusion
###### Abstract
Nevertheless, CNNs are not without limitations; when applied to text processing, their focus on local information within a segment may lead to a failure in capturing long-range dependencies between sentences in lengthy texts, resulting in the loss of important sentence-level features. Consequently, both convolutional neural networks (CNNs) and recurrent neural networks (RNNs) have shown limitations as text datasets grow larger. While incorporating attention mechanisms is effective in enhancing the handling of long texts, it does not fundamentally resolve the issue of inadequate processing capabilities during the word embedding phase [12]. The Transformer, however, utilizes self-attention mechanisms to capture dependencies at any position in a sequence, improving computational efficiency and better-capturing dependencies in long-text sequences [13]. Additionally, in the Transformer, each sub-layer (self-attention and feed-forward networks) is equipped with skip connections and layer normalization, which prevents gradient explosion or vanishing gradients, stabilizes the training process, and accelerates convergence [14]. Nevertheless, the BERT model, which is based on the Transformer, is not without its flaws. Its exceptional performance is due to its high model complexity, leading to substantial computational resources and memory consumption during training. Moreover, BERT requires a vast amount of data for training to effectively learn the complex patterns within the self-attention mechanism, which makes it prone to overfitting in the absence of sufficient datasets. In response to these challenges, many scholars have sought to improve the current NER (Named-Entity Recognition) environment by proposing a series of lightweight models. Among these, the ALBERT model is one of the most used [15]. ALBERT reduces the total parameter count by employing parameter decomposition and sharing mechanisms, thus decreasing memory requirements during training. However, parameter sharing restricts each layer from independently learning different feature representations, which can limit the model's expressiveness in complex language tasks. On the other hand, although the overall reduction in parameter count reduces memory demands, it does not decrease the computational load required for processing long texts, and the issue of prolonged model training times persists. To address the issues, this manuscript proposes a high-performance, lightweight model named GoalBERT. This model utilizes government-related datasets for pre-training on unannotated text, thereby enhancing its ability to understand text within the public administration domain. Furthermore, it employs a BiGRU model to improve contextual comprehension while reducing computational load by one-third compared to BiLSTM. The BiGRU layers are integrated into the Transformer architecture, partially replacingthe self-attention mechanism to further decrease computational demands. Additionally, BiGRU processes sequential information step-by-step, which is more efficient than the global computations required by self-attention, thereby accelerating overall model training speed. For model training, we have incorporated residual connections to enhance fitting efficiency and prevent issues such as gradient explosion and vanishing gradients. We also implemented a weight-sharing strategy to reduce the data requirements, thereby lowering the barriers to model training. These features enable our model to be trained and deployed with lower computational load and memory usage while maintaining a high level of accuracy. Through experiments, it is shown that the GoalBERT model proposed in the manuscript can complete model training and deployment with relatively small performance and memory costs and can be deployed on small computing terminals or edge computing power. The rest of the manuscript is organized as follows: Section 2 reviews recent advances in NER task models. Section 3 provides a detailed description of our GoalBERT model.