# An ELECTRA-Based Model for Power Safety Named Entity Recognition
## 2 Materials and Methods
### Rs-Electra  
To enhance the performance of the ELECTRA model, this paper has improved the normalization methods and activation functions of the transformer encoder's feedforward network(FFN) layer. The original ELECTRA model uses GELU as the activation function and LayerNorm as the normalization method. GELU is a smoothed approximation of ReLU, which allows for a non-zero gradient for negative inputs. LayerNorm is a normalization technique that re-centers and re-scales the inputs based on mean and standard deviation in each layer. Although GELU and LayerNorm are widely used in Transformer modeling, they are not necessarily the best choice for ELECTRA. In fact, recent research [26] has proposed a number of alternatives that may improve the performance and efficiency of ELECTRA. Two of them are RMSNorm and SwiGLu. RMSNorm and SwiGLu are two methods used to improve the performance of Transformer models by normalizing and gating the input and weight matrices, respectively. In this paper, we incorporate the two methods, RMSNorm and SwiGLu, into the ELECTRA model, and propose a new ELECTRA model, RS-ELECTRA. RS-ELECTRA uses RMSNormand SwiGLu in the Transformer encoder, which decreases computational complexity and improves the model's generalization and adaptability. RMSNorm is a normalization method that regularizes the summation inputs of a neuron layer based on the root mean square (RMS) of the inputs, endowing the model with rescaling invariance and implicit learning rate adaptivity. RMSNorm is computationally simpler and more efficient than LayerNorm because it does not need to compute the mean and standard deviation of the inputs. As shown in Formula (6):  
\[\bar{a_{i}}=\frac{a_{i}}{RMS(a)}g_{i},\ where\ RMS(a)=\sqrt{\frac{1}{n}}\sum_{i=1 }^{n}a_{i}^{2} \tag{6}\]  
In this formula, \(a\) represents the input vector, \(\bar{a}\) represents the output vector, \(n\) denotes the dimension of the input vector, \(a_{i}\) is the \(i\)-th element of the input vector, and \(g_{i}\) is a learnable parameter. The SwiGLU is a novel activation function that combines the advantages of the Swish [29] activation function and Gated Linear Units (GLU) [30]. The Swish function is defined as shown in Formula (7):  
\[Swish(x)=x\cdot sigmoid(\beta x) \tag{7}\]  
where \(\beta\) is a learnable parameter. Compared to the rectified-linear (ReLU) [31] activation function, Swish provides a smoother transition near zero, which can lead to better optimization and faster convergence. Swish has been proven to perform better than ReLU in many applications, especially in deep networks. GLU is a neural network layer composed of a linear transformation and a gating mechanism. It is defined as shown in Formula (8):  
\[GLU(x)=sigmoid(W_{1}x+b)\otimes(Vx+c) \tag{8}\]  
GLU is similar to Swish because it combines a linear function with a nonlinear function. In GLU, the linear function is controlled by the sigmoid activation function. In SwiGLU, the Swish function is used to gate the linear function of GLU. This allows SwiGLU to simultaneously gain the advantages of Swish and GLU while overcoming their respective shortcomings. The principle of SwiGLU is shown in Formula (9). \[\text{SwiGLU}(x)=\text{Swish}(W_{1}x+b)\otimes(Vx+c) \tag{9}\]  
where \(x\) is the input vector. \(W_{1}\) and \(V\) are weight matrices. \(b\) and \(c\) are bias vectors. \(\otimes\) represents element-wise multiplication. The Swish function is defined as \(\text{Swish}(x)=x\cdot\sigma(\beta x)\), where \(\sigma\) is the sigmoid function, and \(\beta\) is a learnable parameter, typically set to 1. SwiGLU maintains certain linear properties while providing nonlinear expressive capabilities, thereby enhancing the representational power and learning ability of neural networks. In large language models, SwiGLU is particularly suitable for handling complex semantic relationships and long-distance dependencies due to its advantages in nonlinearity, gating characteristics, gradient stability, and learnable parameters.