## 4 The Proposed Method
### Weight Initialization and Regularization Module  
**Weight initialization of LotHps:** To reduce the uncertainty of weight random initialization for the LotHps activation layer, we propose a novel weight initialization method that can make the error in the initial stage of model training smaller. Specifically, we use the weight coefficients obtained by approximating the ReLU function as the initial weights of the LotHps activation layer, which provide a good starting point for it. The approximate method we propose is as follows:  
Assume that \(\varphi_{0}(x)\), \(\varphi_{1}(x)\), \(\cdots\), \(\varphi_{n}(x)\) is a family of functions with weight orthogonal about the point set \(\{x_{i}\}(i=0,1,\cdots,m)\). In this case, we use a family of Hermite orthogonal functions where \(\varphi(x)\) refers specifically to \(h(x)\). Specify that the approximation function consisting of this family of orthogonal functions takes the form:  
\[S(x)=a_{0}\varphi_{0}(x)+a_{1}\varphi_{1}(x)+\cdots+a_{n}\varphi_{n}(x) \tag{6}\]  
The conventional approximation method is to minimize the sum of squared errors, as shown in the following equation:  
\[\|\delta\|_{2}^{2}=\sum_{i=0}^{m}\Bigl{[}S^{*}(x_{i})-y_{i}\Bigr{]}^{2}=\min_ {S(x)\in\varphi_{i=0}^{m}}\sum_{i=0}^{m}[S(x_{i})-y_{i}]^{2} \tag{7}\]  
where \(S^{*}(x)\) represents the best approximation polynomial, and \((x_{i},y_{i})\) represents the sample points, in this case specifically the points on the ReLU function. The conventional method only provides the best fit for the original function, which is effective for the forward propagation process of the neural network model. However, the gradient of its approximation function may have a large difference with the gradient of the original function, leading to a large error in the backward propagation process of the model. To address this, we consider adding the error of both derivative functions to the objective function. Additionally, since the output values of the batch normalization layer follow a normal distribution, the values are mostly concentrated around 0. We use a weight function \(\sqrt{1-\left(x/l\right)^{2}}\) to better approximate the function values and derivative values around 0, resulting in the final approximation objective we use:  
\[\underset{S(x)\in\varphi_{i=0}}{\underset{i=0}{\sum}}\sqrt{1-\left(x/l\right)^{ 2}}\cdot\left[S(x_{i})-f(x_{i})\right]^{2}+\left[S^{\prime}(x_{i})-f^{\prime}( x_{i})\right]^{2} \tag{8}\]  
where \(S^{\prime}(x)\) represents the derivative of the approximating function, \(f^{\prime}(x)\) represents the derivative of the approximated function, and \([-l,l]\) represents the approximation interval. **Weight regularization of LotHps:** During DNNs model training, we found that significant changes in the values of the weights of the LotHps activation layer or changes in sign caused large fluctuations in the model loss. To prevent the instability of weights during training, we combined the aforementioned weight initialization techniques and proposed a novel weight regularization module that can also improve the generalization ability of the model. Because \(w_{i}\) corresponds to the Hermite polynomial \(h_{i}(x)\), and \(h_{i}(x)\) itself is not in the same dimension, so we first use \(w_{i}=\frac{1}{\sqrt{2^{i}i}}w_{i}\) to get dimensionless \(w_{i}\). Then, let \(W_{t}=\left(w_{0}^{t},w_{1}^{t},\cdots,w_{2}^{t}\right)\) represent the weight of the LotHps activation layer during training, and \(W_{s}=\left(w_{0}^{s},w_{1}^{s},\cdots,w_{2}^{s}\right)\) represent its initial weight. We calculate the relative Euclidean distance between \(W_{t}\) and \(W_{s}\) as a regularization term to constrain \(W_{t}\).