# Secure Convolution Neural Network Inference Based on Homomorphic Encryption
## 4 Our Scheme
### Batch Processing in Inference Stage
Then we need to perform the same filling operation for the convolution kernel and the convolution deviation, so that the convolution kernel is aligned with the image window. After the convolution kernel and window dot product have passed, the convolution operation of the ciphertext is completed by rotating the accumulation, and finally the bias is added, and the ciphertext feature map matrix is output, and each column represents the characteristics of different channels. Algorithm 3 shows the procedure. ```
0: A encrypted matrix(\(c\times f\)) of convolution windows \(S\), padding convlution bias vector(plaintext) \(B\), channel \(c\), padding convlution weights vector(plaintext) \(P\), each row represents a channel, kernel size \(k^{2}\)
0: Encrypted feature map of convlution layer output \(M\)
1:fork := 0 to c do
2:fori := 0 to f do
3:\(ct.temp_{i}\leftarrow MultP(ct.S_{i},\,p.P_{k})\)
4:\(ct.temp_{i}\leftarrow\text{Rescale}()\)
5:endfor
6:\(T_{k}\leftarrow\text{ct.temp}\)
7:endfor
8:fort := 0 to c do
9:fori := 0 to f do
10:forj := 0 to \(\lceil\log 2(k^{2})\rceil\)do
11:temp \(\leftarrow\) Rot(\(T_{[t][i]},\exp 2(\text{j})\))
12:\(T_{[t][i]}\leftarrow\) Add(ct.\(T_{[t][i]}\), temp)
13:endfor
14:\(T_{[t][i]}\leftarrow\) AddPlain(ct.\(T_{[t][i]}\), ct.\(B_{t}\))
15:endfor
16:endfor
17:fori := 0 to f do
18:forj := 0 to c do
19:\(M_{[i][j]}\gets T_{[i][i]}\)
20:endfor
21:endfor
22:return M
```

**Algorithm 3** Homomorphic Convolution

Next, we will pass the eigenvalue ciphertext output by the convolutional layer through the activation function, here we use the linear polynomial function \(poly_{s}wish(M_{k})=0.03347+0.5x=0.19566x^{2}-0.005075x^{4}\) proposed by [PERSON] et al. [12] to approximate the Swish activation function. This step requires the convolutional layer to output the values of each neuron through the activation function: \(Ct.A\leftarrow poly_{s}wish(M_{k}),0\leq k\leq channel\times feature _{*}\). Since the highest order of the approximate polynomial is order 4, the highest-order input is squared twice and then multiplied by the corresponding coefficient, thus consuming three levels of the homomorphic encryption scheme. The activated neuron will be fully connected through two dense layers, we expand the activation layer output into a ciphertext vector with the size channel * feature map size, and then the \(outputsize*Inputsize\) of the first dense layer is batch expanded similarly to a convolution window, aligned with the activation layer output. After encryption, there is an input size ciphertext. In the ciphertext output of the previous layer, only the first slot per position of each image is the correct result of accumulation, in order to perform the next matrix multiplication, we will activate the output tensor of the layer to expand horizontally, replacing the slots of the output dimension size of the subsequent dense layer with the value of the first slot. To achieve this, we choose to consume one more level, multiply each line of the output ciphertext with the \([1,0,0,\ldots,1,0,0]\) vector, which corresponds to the correct slot and the 0 value empties the other slots. Then, we need to rotate \(log_{2}^{do}\) times to expand the value of the first slot to the same scale as the weight matrix, so it is the output dimension of dense layer 1. The same area of the feature map of different images in each ciphertext is spaced at the same interval, so the rotation operation can be synchronized to all images in the batch. Figure 4 shows the simplified process of ciphertext slot transformation. When the modulus is \(N=32\),678, the number of plaintext slots is N/2 = 16,384, 141 pictures can be batched at the same time, if the N value is larger, more plaintext slots can be generated for batch processing, but the SEAL library limits each plaintext to \(2^{60}\) bits. Algorithm 4 provides a detailed description of the specific implementation process. Figure 5 shows the flowchart of Algorithm 4. Figure 4: Simplified batch processes. Figure 5: The flowchart of Algorithm 4.