## 4 The Proposed Method
### Variable-Weighted Difference Training (VDT) Strategy  
The proposed LotHps activation layer shows good performance on shallower DNNs, but its accuracy degrades slightly when applied to deeper networks. Inspired by knowledge distillation techniques, we propose a variable-weighted difference training (VDT) strategy designed to reduce the difference between DNNs using LotHps functions (called LotHps-based models) and DNNs using ReLU functions (called ReLU-based models), thereby improving the accuracy of LotHps-based models. In Figure 3, we demonstrate our novel VDT strategy, which leverages the original ReLU-based model as a teacher to supervise the training of the LotHps-based model. The proposed VDT strategy consists of two loss terms: the first loss term corresponds to the activation loss between the LotHps-based model and the ReLU-based model, while the second loss term comprises the output loss of both models and the cross-entropy between the LotHps-based model output and the true label. Notably, we utilize a weight function denoted by \(\rho(x)\) in Equation (16) to achieve a smooth transition between the two aforementioned loss terms. **Activation Loss:** To quantify the activation loss for the two models, we utilize Kullback-Leibler (KL) divergence. Specifically, we assume that activation layer output distributions of the LotHps and ReLU-based models are represented as \(q(x)\) and \(p(x)\) respectively. The KL divergence of \(q(x)\) and \(p(x)\) can be expressed as:  
\[D_{KL}(p\parallel q)=\sum_{j=1}^{M}\bigl{[}p\bigl{(}x_{j}\bigr{)}\text{log}p \bigl{(}x_{j}\bigr{)}-p\bigl{(}x_{j}\bigr{)}\text{log}q\bigl{(}x_{j}\bigr{)} \bigr{]} \tag{11}\]  
Let \(N\) be the number of activation layers, and \(D_{i}(p_{i}\parallel q_{i})\) be the KL divergence of the \(i\)-th activation layer. Then, the first loss term is:  
\[loss\_act=\sum_{i=1}^{N}D_{i}(p_{i}\parallel q_{i})=\sum_{i=1}^{N}\sum_{j=1}^ {M}p_{i}(x_{j})\left[\text{log}p_{i}\bigl{(}x_{j}\bigr{)}-\text{log}q_{i} \bigl{(}x_{j}\bigr{)}\right] \tag{12}\]  
where \(p_{i}(x)\) and \(q_{i}(x)\) are the \(i\)-th activation layer output distributions of the LotHps-based model and the ReLU-based model, respectively. **Output Loss:** For the loss terms of the final output distribution of the two models, we utilize the approach suggested in [28], which uses a response-based knowledge distillation method with a soft target technique:  
\[Q^{T}[i]=\frac{\exp(z_{i}/T)}{\sum\limits_{j}\exp\left(z_{j}/T\right)} \tag{13}\]  
where \(Q^{T}[i]\) is the soft target version of the class \(i\) sample prediction, \(T\) represents the temperature, and \(z_{i}\) is the logarithm of the original prediction. The second loss term is obtained by combining the output distribution differences and hard label loss as follows:  
\[loss\_out=\alpha_{1}T^{2}\cdot CE\left(Q_{H}^{T},Q_{R}^{T}\right)+\alpha_{2} \cdot CE\left(Q_{H}^{1},y_{\text{true}}\right) \tag{14}\]  
where \(Q_{H}^{T}\) and \(Q_{R}^{T}\) represent the soft target outputs of the LotHps and ReLU-based models, respectively, \(y_{true}\) represents the true label, and \(CE\) represents the cross-entropy function. \(\alpha_{1}\) and \(\alpha_{2}\) are two hyperparameters that control the relative magnitude of the two cross-entropy losses. **Loss Smooth Transition:** We did not simply weigh the two aforementioned loss terms arithmetically. Instead, we used a smooth transition function \(\rho(x)\) as a dynamic weighting function to achieve a smooth transition from \(loss\_act\) to \(loss\_out\). In this way, during the initial training, the main goal of the LotHps-based model is to reduce the difference between it and the activation output distribution of the ReLU-based model. In the later stages of  
Figure 3: Variable-weighted difference training (VDT) schematic. training, the LotHps-based model adjusts the model according to the final learning goals. The specific expression of the smooth transition function \(\rho(x)\) is as follows:  
\[\rho(x)=\left\{