## 6 Evaluation
### Plaintext Training
#### 6.1.1 MNIST Dataset  
For the MNIST dataset, we evaluated CNN-6 using our proposed methods. We set the batchsize to 64 and used the Adam optimizer. The initial learning rate for both the ReLU and LotHps-based models is set to 0.001. We adopt a dynamic adjustment learning rate strategy, where the learning rate decays by half every 10 epochs of training. The final accuracy of the original ReLU-based model is 99.57%, and the final accuracy of the LotHps-based model is 99.62%. We also tried to increase the number of Hermite orthogonal bases in the activation layer, but found that the accuracy actually decreased and the training process was prone to gradient explosion due to the runge phenomenon of higher degree polynomials. Since our LotHps-based model is more accurate than the original ReLU-based model, we do not need to use the variable-weighted difference training strategy to further improve its performance. This avoids adding additional computational overhead.  
We compared the accuracy and errors of the methods proposed in other literature on the MNIST dataset, and the models used by these methods are shown in Table 4. CryptoNets [9] used \(x^{2}\) to replace the original ReLU activation function, resulting in an absolute accuracy reduction of 0.32%. PPCN [10] used the Taylor approximation of the softplus function combined with the BN layer, which results in an accuracy reduction of 0.29%. CryptoDL [12] used a 2-degree polynomial approximation of the derivative function of ReLU and integrates it to obtain a 3-degree polynomial to replace the ReLU function. The original model has an accuracy of 99.67%, but this drops to 99.52% after the replacement. [PERSON] [35] used the idea of dynamic programming to train the polynomial as the model of activation function, and its accuracy was 99.26%. We used the approximate polynomial from Approx-ReLU [21] as the activation function, which resulted in a 0.03% reduction in model accuracy. However, the model test accuracy curve of this method fluctuates widely, as shown in Figure 4.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline
**Model** & **Batchsize** & **Tile Shape** & **Packing Mode** \\ \hline CNN-6 & 64 & \(8\times 8\times 1\times 4\times 64\) & F-W-H-C-B \\ AlexNet & 8 & \(8\times 4\times 4\times 16\times 8\) & C-W-H-F-B \\ ResNet-20 & 16 & \(4\times 16\times 16\times 2\times 16\) & C-W-H-F-B \\ \hline \hline \end{tabular}  
\(C\) represents the number of channels, W represents the width of the image, H represents the height of the image, F represents the number of convolution kernels, B represents the batchsize.  
\end{table}
Table 3: Parameter setting of inference optimization method.  
Figure 4 shows the training loss and test accuracy curves of the three activation functions: ReLU, LotHps, and Approx-ReLU [21]. From the figure, we can see that the model training losses of our proposed LotHps and Approx-ReLU are very close after 30 epochs. However, the model losses of our method are smaller in the early training period, which indicates the effectiveness of our proposed weight initialization method. Observing the test accuracy curves, we can see that the model test accuracy curves corresponding to the Approx-ReLU fluctuate more, while the curves of the LotHps are more stable and have higher test accuracy. This reflects that our weight regularization module can ensure the stability of model generalization ability in the process of model training and can effectively improve the model's generalization ability.