## 4 The Proposed Method
### Low-Order Trainable Hermite Polynomials (LotHps) Activation Layer  
As previously mentioned, the ciphertext produced by homomorphic encryption only supports addition and multiplication operations. Therefore, the standard ReLU activation function used in deep neural networks does not work properly in this context. To address this issue, we need to use a homomorphic-friendly polynomial as our activation function. Furthermore, performing a single multiplication on the ciphertext produced by homomorphic encryption is computationally expensive, so we want to minimize the number of multiplication operations. To achieve this, we need to design a low-degree polynomial activation function. In this section, we will discuss the important properties of Hermite orthogonal polynomials and how to use them as our activation layer.  
**Hermite polynomials:** The Hermite orthogonal polynomials are defined as \(H_{n}(x)=(-1)^{n}e^{x^{2}}\frac{d^{n}}{dx^{n}}e^{-x^{2}}\). They have been widely used in various fields due to their many excellent properties [26]. Here, we only introduce the orthogonality that we use. Specifically, for any two distinct non-negative integers \(n\) and \(m\), the Hermite polynomial \(H_{n}(x)\) and \(H_{m}(x)\) are orthogonal under the weight function \(e^{(-x^{2})}\), i.e.:  
\[\int_{-\infty}^{\infty}e^{-x^{2}}H_{n}(x)H_{m}(x)dx=\sqrt{\pi}2^{n}n!\delta_{ nm} \tag{2}\]  
where \(\delta_{nm}=1\) when \(n=m\), otherwise \(\delta_{nm}=0\). Additionally, the Hermite polynomials satisfy a three-term recurrence relation:  
\[\left\{\begin{array}{c}h_{0}(x)=1,h_{1}(x)=2x\\ h_{n+1}(x)=2 xh_{n}(x)-2 nh_{n-1}(x),n\geq 2\end{array}\right. \tag{3}\]  
**LotHps based on Hermite polynomials:** Based on the orthogonality of Hermite polynomials, we constructed a low-degree trainable Hermite polynomials (called LotHps) activation function. In order to maintain low depth of multiplication, we only use the lower degree terms \(h_{0}(x)\), \(h_{1}(x)\), and \(h_{2}(x)\) of the Hermite polynomials. The LotHps function proposed by us can be expressed as:  
\[LotHps(x)=w_{0}h_{0}(x)+w_{1}h_{1}(x)+w_{2}h_{2}(x) \tag{4}\]  
Figure 2: Low-degree Hermite neural network (LHDNN) framework.  
where \(w_{0}\), \(w_{1}\), and \(w_{2}\) are learnable parameters whose values are adjusted adaptively during neural network training. Specifically, during the backward propagation process in the model, the gradient of the weights in the LotHps activation layer can be derived using the chain rule. Assuming that \(\theta\) represents the objective function, the gradient of parameters in the LotHps activation layer are:  
\[\frac{\partial\theta}{\partial w_{i}}=\sum_{c}\frac{\partial\theta}{\partial H (x_{c})}\frac{\partial H(x_{c})}{\partial w_{i}}=\sum_{c}\frac{\partial\theta }{\partial H(x_{c})}h_{i}(x_{c})\text{, }i=0,1,2 \tag{5}\]  
where \(c\) represents the number of specific input channels, \(x_{c}\) represents the input value of the \(c\)-th channel of the Hermite activation layer, and \(\frac{\partial\theta}{\partial H(x_{c})}\) represents the gradient back-propagated from a deeper layer. With the gradient, we can update the values of \(w_{0}\), \(w_{1}\), and \(w_{2}\) through optimization algorithms such as stochastic gradient descent [27] to minimize the loss function.  
As to why we use the Hermite polynomials instead of the similar Legendre, Chebyshev and Laguerre polynomials, etc., this is because only the orthogonal interval of Hermite polynomials is \([-\infty,+\infty]\). This means that no matter how large the output value of the batch normalization (BN) layer in a DNN is, it is always in the orthogonal interval of the Hermite polynomial. Although we can satisfy the orthogonality condition for other orthogonal polynomials by scaling the input values, we will undoubtedly introduce more weights to train [26].