DEIM Forum 2023 5b-9-1

Latency Improvement of
Homomorphic Encrypted Deep Neural Network Training
Tiago Monteiro†

Takuya Suzuki‡†‡Graduate School of Fundamental Science and Engineering

andHayato Yamana§

3-4-1 Okubo, Shinjuku-ku, Tokyo, 169-8555 Japan§ Faculty of Science and Engineering, Waseda University 3-4-1 Okubo, Shinjuku-ku, Tokyo, 169-8555 Japan
E-mail:†§ {tiago, t-suzuki, yamana} @yama.info.waseda.ac.jp

Abstract Machine learning as a service (MLaaS) has become increasingly popular. However, MLaaS has privacy concerns
attached. Privacy-preserving machine learning (PPML) solves privacy concerns through cryptographic techniques such ashomomorphic encryption (HE). HE enables computations over ciphertexts without decryption. However, HE requires highcomputational costs and has limited operations. Consequently, many optimization algorithms, key components in PPML training
over HE, are hard to be adopted. Therefore, adopting a suitable optimization algorithm for HE is important. This paper aims todecrease the training latency of a homomorphic encrypted deep neural network. We take advantage of Nesterov’s accelerated
gradient descent and the SIMD computation capabilities of HE.
Keywords: Privacy-preserving machine learning, Homomorphic encryption, Deep neural network training, Nesterov’s
accelerated gradient, SIMD computation
this front.

1. Introduction
Deep learning (DL) has become a valuable method for

In 2019, Nandakumar et al. [7] was the first to achievesolving multifaceted problems. Finance, speech/visual

private training of neural networks non -interactively. They

recognition, and health sciences are examples of fields

demonstrated a neural network that can train a mini-batch

influenced by this technology. However, to fully takeof 60 samples in 40 minutes meanwhile achieving an

advantage of DL, two prerequisites exist: 1) users must be

accuracy of 97.8% on the MNIST dataset. However, theycapableused table-lookups to calculate the non -linear functions,ofselectingadequate

DL

model

and

hyperparameters for a specific task ; and 2) users must haveseverely hindering their work's latency.

sufficient computational resources to execute a selectedmodel. Machine Learning as a Service (MLaaS ), a cloud-In 2020, Lou et al. [ 8] improved the solutions proposed

based service, is a solution that provides a layer ofby Boemer et al. [5] and achieved 98.8% accuracy on the

abstraction around the two prerequisites needed to take

MNIST dataset in 5 epochs, which took 8 days to executeadvantage of DL models successfully.

over the prior state -of-the-art system [7]. They utilized

Despite the advantages provided by MLaaS platforms,

CHIMERA [9], a key-switching scheme to solve the issues

users with sensitive information such as medical orwith BGV-based lookup tables. Although their results

financial records are concerned by potential data leakages,

improved, they did not present the latency cost ofwhich are becoming more common. In response to thisswitching between schemes in their results.

concern, privacy-preserving machine learning (PPML)

The above works provided breakthroughs in non-ensures that MLaaS parties’ resources , such as the user’s

interactive DL training with HE, but there is still space fordata or the service provider ’s neural network model, arelatency improvements.

secure. With homomorphic encryption (HE) [1,2], we canassure data privacy and usability in a MLaaS pipeline.
Predictions with PPML over HE have achieved goodresults [3, 4, 5, 6]. Conversely, PPML over HE still
struggles to train a DL model with encrypted data, having
not

yet

achievedsignificant

breakthroughs.

The

computational complexity of the operations involved in the
DL’s tasks and the high computational cost of HE schemes
are the main reasons behind the lack of research effort on

In this work, we further reduce the training latency of ahomomorphically encrypted neural network model . Our
main contributions are the following:•

We introduce the usage of Nesterov’s accelerated
descent (NAG) instead of gradient descent (GD) . NAG
is a promising

optimization method because it

accelerates training convergence, i.e., decreases thenumber of training iterations to achieve maximum•accuracy compared to GD.

directed graphs where input nodes are connected to output

We take advantage of the SIMD capabilities presented

nodes through several hidden layers. At the core of neuralin HE schemes to shorten the homomorphically

networks’ operations, matrix multiplications and dot

encrypted DL training latency.

products are performed. In deep neural network training,

The remainder of this paper is organized as follows:forward and backpropagation stages exist. The training

Section 2 presents the required background on neuralphase aims to learn parameters that minimize the values

networks and HE to build our solution. Section 3 shows afrom a pre-defined loss function.

brief survey on PPML with HE solutions, highlighting the

𝐻𝑙 = 𝜙(𝑊 𝑙 ∙ 𝑍 𝑙−1 )( 2.1 )

key achievements by works similar to ours . Section 4
describes our proposed solution; namely, we show how we

A network with 𝐿 layers is traversed sequentially fromsuccessfully apply NAG . Section 5 provides experimental

the input to the output layer. Equation (2.1) describe s theresults and discussion about the obtained results. Finally, aforward propagation at the 𝑙-th layer. The output of the 𝑙-conclusion is provided in Section 6.

th layer 𝑍 𝑙 is obtained by applying a non-linear function
𝜙 to the matrix product of the weights from the current
layer, 𝑊 𝑙 , with the output from the (𝑙 − 1)-th layer, 𝑍 𝑙−1.

2. Preliminaries
2.1 Fully homomorphic encryption (FHE)
Homomorphic encryption (HE)

𝑊𝑡+1 ≔ 𝑊𝑡+1 − 𝛼is a cryptographic

𝜕ℒ
𝜕𝑊𝑡+1

( 2.2 )

technique that allows computations over ciphertexts

Equation (2.2) describes the gradient descent algorithm

without leaking its contents, where the results from theat 𝑡 -ath iteration/step with learning rate 𝛼 and gradientcomputations over ciphertexts are equivalent to the results

𝜕ℒ, of the loss function ℒ . GD is an optimizationfrom the computations over plaintexts. HE allows two𝜕𝑊𝑡+1

operations to be performed on ciphertexts: addition andalgorithm aiming to find the local minimum of ℒ. Duringmultiplication.

thebackpropagation

stage,

anetworkistraversed

Presently, fully homomorphic encryption (FHE) is asequentially, in reverse order, from the output to the input

variant of HE that allows an arbitrary number of additions

layer. The loss function result, calculat ed at the end of theand multiplications. In an FHE scheme, when a plaintext is

forward propagation, is passed backward so that theencrypted to a ciphertext, a random element, called noise,

weights from each layer are updated according to theiris added to the ciphertext so that it is harder to reveal therespective influence in the error result .

original plaintext. Moreover, the noise in a ciphertext
grows with every homomorphic operation. After a number

2.3 Nesterov’s Accelerated Gradient (NAG)

of operations, the noise growth surpasses the ciphertext

The training phase is utilized to find the weights of thethreshold, which results in rendering the ciphertext

hidden and output layers that minimize the values of a loss

unusable.

function. GD follows the negative gradient of an objective

To overcome the noise accumulation problem, a special

function to locate the local minimum of a loss function.

operation, called bootstrapping, is used. Bootstrapping

However, a GD’s limitation arises when the negative

comprises a ciphertext re -encryption using a different key;

gradient direction rapidly oscillates: the zig-zagging

therefore, a new ciphertext with smaller noise than the

problem. Zig-zagging slows down the loss function

original ciphertext i s obtained. Bootstrapping allows any

minimization, i.e., unnecessary iterations are made to

ciphertext

minimize a loss function adequately.

tobeevaluatedcorrectly;

however,bootstrapping is a time and resource consuming operation.

A solution to the zig-zagging problem is the momentum

In this study, we adopt the CKKS scheme [2] , which canmethod [10]. Yurii Nesterov proposed one of the first GD

perform approximate computations on fixed -point real

schemes using momentum in 1983 [11]. In 2013, Sutskever

numbers, bootstrapping, and single -instruction multiple -et al. [12] popularized its application in neural network

data (SIMD) computations on the ciphertext.

training. The update rules take the following for m shownin Equation (2.3) and Equation (2.4). Equation (2.3)

2.2 Deep learning model

describes how the next step in the algorithm is taken, andNeural networks are multi -layered, weighted, andEquation (2.4) describes the update of the weight matrix

approximation across a wider range.

𝑊.
In Equation (2.3), starting with 𝑉0 = 0 , the algorithm
iterates for 𝑡 = 1,2, ⋯:𝑉𝑡+1 = 𝜇𝑉𝑡 − 𝛼∇ℒ(𝑊𝑡 + 𝜇𝑉𝑡 )( 2.3 )

𝑊𝑡+1 = 𝑊𝑡 + 𝑉𝑡+1

( 2.4 )

, where 𝛼 is the learning rate; ∇ℒ is the direction of thedescent; 𝜇 is the momentum coefficient; and 𝑉 is thevelocity matrix, accumulating previous gradient values.
The momentum term 𝜇𝑉𝑡 scales the value of previous
gradients, determining how much the previous gradients
influence the current value. 𝜇𝑉𝑡 makes the algorithm movefaster, i.e., take bigger steps, when updates are consistently
small and in the same direction, and move slow, i.e., takesmaller steps, when the gradient direction is significantly

Figure 1 – Original sigmoid (blue), least squares fitting

oscillating. The gradient term ∇ℒ(𝑊𝑡 + 𝜇𝑉𝑡 ) determines

approximation (green), and Taylor approximation (red)the descent direction. Compared with Equation (2.2) thegradient term from Equation (2.3) includes 𝜇𝑉𝑡 , which

3. Related Work
This section describes recent works that used non-corrects the step taken at 𝜇𝑉𝑡 if the update is poor, thus
giving more stability to NAG.

interactive HE neural networks. We selected the non interactive HE approach for our neural network because it

2.4 Least squares approximation of the sigmoid
function
Although NAG performs better than GD, other technicalresembles the best MLaaS platforms. M LaaS platforms do
not need the user to be online meanwhile the platform
performs computations. Figure 2 outlines the common non-problems remain in implementing our solution. In Equationinteractive

training

process. An

advantage

of

non -

(2.1), the non-linear function 𝜙 is the biggest evaluationinteractive training is that a user can go offline after the

obstacle, since HE schemes only allow the evaluation ofuser sends the data to the service provider. We explain the

additions and multiplications.

flow of the non-interactive training as follows :To make use of the sigmoid function as our non -linear

1.

A user encrypts data with the user’s public key.

function, we can adopt one of two options : lookup tables

2.

The user shares the encrypted data and the public key
to the service provider.

or function approximation. Lookup tables have been used
to train DNNs in [7, 8]. One of the disadvantages of lookup

3.

encrypted data.

tables is the low resolution of its values. The low value
resolution is caused by the limited number of entries in the

The service provider initiates training with the4.

After the training, the service provider obtains the

lookup table. Moreover, the time preparing the table lookup

encrypted DL model parameters that are ideal for

is long. Another solution is approximating the non -linear

performing predictions. Note that the service provider

function [14]. Taylor polynomials have high accuracy

can train the model without learning anything about

values over small ranges , however, the error grows rapidly

the user’s data or the resulting model parameters thatoutside the specified range.

have been learned.

Therefore, we use the least squares fitting polynomial to5.

parameters to the user.

approximate the sigmoid function. Least squares provide sa sufficient approximation with a given interval. Figure 1

The service provider sends the encrypted DL model

6.

The user decrypts the encrypted DL model parameters

plots the original sigmoid function, its least squares fitting

and obtains the unencrypted DL model parameters.

polynomial (with degree 3) , and its Taylor approximation

Note that only the user can decrypt the encrypted data(of degree 3) within the interval [−8, 8]. It can be observed

since only the user has access to the private key.

that the Taylor approximation provides an accurate
approximation only around a small range, while the leastsquares fitting polynomial provides a more accurate

NAG is executed. First, the service provider receives two
sets of ciphertexts, |𝑋| and |𝑌| , corresponding to theencryption of the training set and its labels , respectively.
The service provider initializes the encrypted weights |𝑊|
randomly, and sets the encrypted momentum values |𝑉| tozero. For each layer 𝑙 , the output of the layer, |𝐻𝑙 | , iscalculated by the polynomial approximation of the sigmoid
function 𝜙 ′ . The input of 𝜙 ′ is the result from the matrix
multiplication between the encrypted weights at layer 𝑙 ,|𝑊 𝑙 |, and the output from the previous layer |𝐻𝑙−1|. Note
Figure 2 – Neural network training system modelthat the values of|𝐻0 | are assgined to|𝑋| . Aftercompleting the traversal of the neural network from theTable 1 – Recent works on training DNNs over HE
Paper

SchemeDatasetAcc(%)

input to the output layer, the error result 𝒥 is calculated

#Epo
ch

Tim
e

by the mean squared error between |𝐻𝐿 | and |𝑌|.

each layer sequentially in reverse order, updating each
updates at each layer, the encrypted momentum values are

Nandakumar
et al. [7]

BGV [13]

MNIST

97.8

50

13.4
years

Low et al.
[8]

CHIMERA
[9]

MNIST

98.8

5

8
days

With the obtained error values , the algorithm traverses
layer ’s weights with regard to the error value. During the
calculated using Equation (2.3) before updating theweights using Equation (2.4).

Table 1 summarizes recent research on DL training with
FHE. The problem with existing studies is that they do not

It is expected for NAG to achieve maximum accuracy inachieve optimal latency results for training. Low et al. [8]fewer steps compared to GD. Concretely, after 𝑡 iterations,improve previous state-of-the-art results [7] by achieving
higher accuracy within fewer training iterations , but (1)NAG will have a rate of convergence of 𝑂(1⁄𝑡 2) comparedto 𝑂(1⁄𝑡) in GD. However, NAG has higher computational

they use the same optimization algorithm as [ 7] to perform

complexity in each hidden layer compared to GD. The

training, and (2) they do not focus on packing the input

increased complexity from NAG is caused by additional

data effectively to decrease latency. Thus, a new method is

calculations. Specifically the scaled momentum values,

needed to obtain enough accuracy in a shorter time.

4. Proposed method
In this work, we propose to reduce the latency of ahomomorphically encrypted deep neural network with HE,
using the non-interactive HE approach. Our methodology
combines the optimization algorithm NAG to decrease thenumber of training iterations needed to achieve maximum
accuracy of a model and an efficient ciphertext packing
method that takes advantage of CKKS’s SIMD capabilities.
This section is organized as follows: we explain how weadapt NAG to our deep neural network in Section 4.1, andin Section 4.2 we describe the ciphertext packing method
utilized in our experiment.

4.1 Evaluation of NAG
Algorithm 1 shows how theencrypted training withand the gradients of the weight values with the respective
scaled momentum values.
Algorithm 1: Encrypted NAG training
1: Input: encrypted training samples |𝑋|, encrypted
training labels |𝑌| , learning rate 𝛼 , momentum
parameter 𝜇, the number of iterations 𝑇, numberof layers 𝐿, and a polynomial approximation of
sigmoid function 𝜙 ′2: Output: encrypted weights |𝑊1 |, |𝑊 2 |, ⋯ , |𝑊 𝑙 |3: |𝑊1 |, |𝑊 2 |, ⋯ , |𝑊 𝑙 | ← Random initialization
4: |𝑉 1 |, |𝑉 2|, ⋯ , |𝑉 𝑙 | ← Initialize to zero
5: |𝐻0 | ≔ |𝑋|
6: for 𝑡 ← 1 to 𝑇 do
7:
for 𝑙 ← 1 to 𝐿 do
8:
|𝐻𝑙 | = 𝜙 ′ (|𝐻l−1 ∙ 𝑊 𝑙 |)9:
end for
10:
𝒥 ← 𝐿𝑜𝑠𝑠11:
for 𝑙 ← 𝐿 to 1 do𝑙 | ← |𝜇𝑉 𝑙 | − |𝛼𝛻𝒥(|𝑊 𝑙 + 𝜇𝑉 𝑙 |)|
12:
|𝑉𝑡+1
𝑡
𝑡
𝑡
𝑙 | ≔ |𝑊 𝑙 + 𝑉 𝑙 |
13:
|𝑊𝑡+1
𝑡
𝑡+1
14:
end for
15: end for
16: return |𝑊1 |, |𝑊 2 |, ⋯ , |𝑊 𝑙 |pack and encrypt a matrix 𝑀 ∈ ℝ3×6 and a vector 𝑣 ∈ ℝ1×6.

4.2 Ciphertext packing
As mentioned in Section 2.1, we utilize the SIMD

Note that the number of features in M must match thecapabilities of HE. Specifically, we take advantage of HE’s

number of columns in 𝑣. The values in 𝑣 are replicatedpacking capabilities.

(dark blue squares) to match the dimensions from 𝑀.

Ciphertext packing

consists ofencrypting a vector of 𝑛 plaintexts into a single ciphertext.

Once the data are packed, we compute element -wise

Each plaintext element is stored separately in a ciphertext

multiplication over the corresponding ciphertexts. The

slot,

results (yellow squares) are obtained by repeating theandciphertext

operations(additions

andmultiplications) are performed slot-wise, i.e., element -rotation and additionwise.

the values marked with “*” (light grey squares) are

In this work, we adapt the packing algorithm ideas of
Han et al. [15] and Aharoni et al. [16]. Han et al.’s

operations log 2 (# 𝑠𝑙𝑜𝑡𝑠). Note that

unknown/garbage values. The unknown values can becleaned by multiplication with zero vectors.

algorithm [15] allows us to partition a matrix of data into
smaller sub-matrixes and then encrypt the sub-matrices
with multiple ciphertexts. Aharoni et al.’s algorithms [16]extend the matrix vector multiplication algorithm from[15] to include matrix multiplications.

Figure 4 – Matrix vector multiplication

Given that the input data is split into batches, a batch
can be considered to have 𝑛 samples, and each sample to

5. Experimental evaluation

have 𝑚 features. Thus, a batch 𝑋 can be represented as a

We conducted experiments to evaluate NAG against GD

𝑛 × 𝑚 matrix. Then, 𝑋 is divided into multiple 𝑓 × 𝑔

utilizing the MNIST [17] dataset. We compared NAG and

sub-matrices 𝑋𝑖,𝑗 for 0 ≤ 𝑖 < ⌈𝑛⁄𝑓 ⌉ and 0 ≤ 𝑗 < ⌈𝑚⁄𝑔⌉ .

GD using two packing algorithms: the packing algorithm

The sub-matrices are supposed to be packed into a single

from Nandakumar et al. and the packing algorithm from

ciphertext; therefore, 𝑓 and 𝑔 are set to utilize theHan et al. and Aharoni et al.

maximum number of 𝑠𝑙𝑜𝑡𝑠 in a ciphertext.
Figure 3 shows an example of a training dataset

5.1 Dataset

containing 3 samples with 6 features. The dataset is

The MNIST dataset for handwritten digit recognition

partitioned into 4 sub-matrices and then encrypted byconsists of 60,000 training samples and 10,000 testing

ciphertexts containing 8 slots each. Note that the slots thatsamples. Each image contains a handwritten number

are not utilized (light blue squares) are set to zero.

between 0 and 9. The images of the MNIST dataset have agray-scale 28x28 pixel resolution, with one handwritten
digit located at the center of the image. Each image islabeled with a class ranging from 0 to 9.
5.2 Network architecture
The neural network in our experiment followed the samearchitecture used by Nandakumar et al.’s NN2 [7], which

Figure 3 – Partition of a matrix (left) followed byconsists of a 3 layer fully-connected network with 64

encryption through multiple ciphertexts

neurons on the input layer, 32 neurons on the first hidden
layer, 16 layers on the second hidden layer, and 10 neurons

4.3 Ciphertext multiplicationon the output layer.

Matrix multiplications are some of the most timeconsuming operations in the HE. To efficiently perform
matrix multiplications, we utilized the packing scheme
mentioned in the previous section.
To

exemplify

theunderlying

5.3 Evaluation method
We implemented the homomorphically encrypted deepneural network over HE using HEAAN’s [ 18] CKKS

steps

in

matrixscheme.

multiplication, we utilize an example of how matrix -vector

When performing the experiments, two configurations

multiplication is performed on Figure 4. The following

were prepared for the target neural network : baseline andsteps can be generalized to matrix multiplications [16]. We

proposed. The baseline configuration utilizes the packing

and matrix multiplication algorithms used by Nandakumar

Table 5 shows the latency of matrix multiplications withet al., and the proposed configuration uses the partitionedone

packing and matrix multiplications algorithms described inperformed 9.8 times faster matrix multiplications on

Section 4.2. The configurations’ CKKS parameters are

hidden layer 1 compared to the baseline. This is because

shown in Table 2. For both configurations , we set thethe proposed packing algorithm requires 1 multiplication

rescaling factor to 23, and the modulus size to 600. For theandbaseline configuration, we set the number of ciphertext

multiplication s. In contrast, the baseline packing algorithm

slots to 64, and for the proposed configuration , we set therequires at most 𝑛 2 multiplications when performiongnumber of slots to 2,048.

matrixSimilar to Nandakumar et al., each image from themini-batch.

at

most

The

In

packingrotationslog 2 (# 𝑠𝑙𝑜𝑡𝑠)

multiplications.

rotations,

proposedduringcomparison

homomorphic

algorithm

tomatrixciphertext

multiplicationsare

MNIST dataset was compressed by cropping the centralcomputationally more expensive rotations , thus, the lesser

24x24 pixels and rescaling by a factor of 1⁄3 using

multiplications, the lower the latency.

bicubic interpolation , thus obtaining an 8x8 pixel image
representation.
We evaluated the execution time of the training
algorithm, the generation of weights, and the matrix
multiplication operation. Note that the execution time wasaveraged by three continuous executions after the first
execution.

Table 3 – Execution time of 1 training iteration with 1
mini-batch (containing 64 samples)
Baseline
Proposed
Optimization
algorithm
Training latency (min)
GD
34.07
7.50
NAG
46.15
8.35

We used a server which had a Xeon Platinum 8280 (2.7GHz) with 56 cores, and 1.5TB of main memory fortraining over HE.
Table 2 – CKKS parameters for the experiment
Packing
Rescaling
ModulusLevel# slotsalgorithm
factorsize
Baseline
23
34
600
64
Proposed
23
34
600
2,048
5.4 Evaluation results
Table 3 shows the training execution time results over
the MNIST dataset. The measurements were performed
using 56 threads with NTL for one training iteration. One
training iteration is enough to estimate the overall
execution time of the training algorithm.
When using NAG, 1 training iteration took 8.35 minutes,
5.52 times faster than the baseline method using NAG.
When using GD, 1 training iteration took 7.50 minutes,
4.54 times faster than the baseline method.
Table 4 shows the execution time encrypting thegenerated weights. The proposed algorithm is has lower
execution time than the baseline algorithm because theproposed method’s weight encryption algorithm is the sameas the encryption algorithm of the inputs. In contrast, thebaseline method had each element on a weight matrix
encrypted individually, and the input data is encrypted byrow. Moreover, the proposed packing algorithm has smaller
latencywhen

different layers.

performing

matrixmultiplications

in

Table 4 – Execution time of weight encryption
HiddenHiddenPacking
Outputlayer 1layer 2algorithm
layer (sec)
(sec)
(sec)
Baseline
96.34
22.02
10.86
Proposed
23.30
4.72
4.68
Table 5 – Execution time of matrix multiplications
HiddenHiddenPacking
Outputlayer 1layer 2algorithm
layer (sec)
(sec)
(sec)
Baseline
392.85
68.69
38.57
Proposed
40.05
26.72
20.29

6. Conclusion
This paper proposed a set of methods to decrease thetraining latency of a homomorphically encrypted deepneural network. Nesterov’s accelerated gradient (NAG)
descent was used instead of gradient descent (GD) todecrease the number of iterations and to achieve maximum
accuracy in the training phase ; and the packing algorithms
from Han et al. and Aharoni et al . were adapted into oursolution.
The experimental evaluation

was conducted on acompressed MNIST dataset, confirming that the training
latency was lower than a neural network packing the databy rows. Compared to the baseline packing algorithm, theproposed packing algorithm can decrease the neuralnetwork’s

latencyinweightgeneration

andmatrixmultiplication .
Future work includes 1) conducting the accuracy

comparison between GD and NAG after the homomorphic
training, and 2) evaluating the training latency of NAG
with the proposed packing solution on the MNIST dataset
without compression

References[1] Gentry,
C.
(2009).
Fully
homomorphic
encryption using ideal lattices. Proceedings of the
41st annual ACM symposium on Theory of computing,pp. 169–178.
[2] Cheon, J. H., Han, K., Kim, A., Kim, M., andSong, Y. (2018). Bootstrapping for approximate
homomorphic encryption. Proceedings of the Annual
International Conference on the Theory andApplications of Cryptographic Techniques , pp. 360 –384. Springer, Cham.
[3] Gilad-Bachrach, R., Dowlin, N., Laine, K.,
Lauter, K., Naehrig, M., and Wernsing, J. (2016).
Cryptonets: Applying neural networks to encrypted
data with high throughput and accuracy. International
Conference on Machine Learning, pp. 201 –210.
[4] Al Badawi, A., Jin , C., Lin, J., Mun, C. F., Jie, S.
J., Tan, B. H. M., Nan, X., Khin A. M. M.,and
Chandrasekhar, V. R. (2020). Towards the alexnet
moment for homomorphic encryption: Hcnn, the firsthomomorphic cnn on encrypted data with gpus. IEEE
Transactions on Emerging Topics in Computing, 9(3),pp. 1330–1343.
[5] Boemer, F., Lao, Y., Cammarota, R., andWierzynski, C. (2019). nGraph-HE: a graph compiler
for deep learning on homomorphically encrypted data.
Proceedings of the 16th ACM International
Conference on Computing Fron tiers, pp. 3–13.
[6] Dathathri, R., Saarikivi, O., Chen, H., Laine, K.,
Lauter, K., Maleki, S., Musuvathi, M., and Mytkowicz,
T. (2019). CHET: an optimizing compiler for fully homomorphic
neural -network
inferencing.
Proceedings of the 40th ACM SIGPLAN Conferen ceonProgramming
Language
Designand
Implementation, pp. 142 –156.
[7] Nandakumar, K., Ratha, N., Pankanti, S., andHalevi, S. (2019), Towards Deep Neural Network
Training on Encrypted Data. IEEE/CVF Conference
on Computer Vision and Pattern Recognition
Workshops, 2019, pp. 40–48[8] Lou, Q., Feng, B., Charles Fox, G., and Jiang, L.
(2020). Glyph: Fast and accurately training deepneural networks on encrypted data. Advances in
Neural Information Processing Systems, 33, pp.
9193–9202.
[9] Boura, C., Gama, N., Georgieva, M., and Jetchev,
D. (2020). Chimera: Combining ring-lwe-based fully
homomorphic encryption schemes. Journal of
Mathematical Cryptology, vol. 14, no. 1, pp.316 -338.
[10] Kim, A., Song, Y., Kim, M., Lee, K., and Cheon,
J. H. (2018). Logistic regression model tra ining basedon the approximate homomorphic encryption. BMC
medical genomics, 11(4), pp. 23–31.
[11] Nesterov, Y. E. (1983). A method for solving theconvex programming problem with convergence rate
O(1/k^2 ). Doklady Akademii Nauk SSSR , 269(3), pp.
543–547.
[12] Sutskever, I., Martens, J., Dahl, G., and Hinton,

G. (2013). On the importance of initialization andmomentum in deep learning. International conferenceon machine learning, pp. 1139 –1147. PMLR.
[13] Brakerski, Z., Gentry, C., and Vaikuntanathan, V.
(2014). (Leveled) fully homomorphic encryption
without bootstrapping. ACM TOCT, vol. 6, no. 3, pp.
1–36.
[14] Baruch, M., Drucker, N., Greenberg, L., andMoshkowich, G. (2022). A Methodology for Training
Homomorphic Encryption Friendly Neural Networks.
Applied Cryptograph y and Network Security
Workshops. Lecture Notes in Computer Science, vol13285. Springer, Cham.
[15] Han, K., Hong, S., Cheon, J. H., and Park, D.
(2018). Efficient logistic regression on large
encrypted data. Cryptology ePrint Archive.
[16] Aharoni, E., Adir, A., Baruch, M., Drucker, N.,Ezov, G., Farkash, A., Greenbder, L., Masalha, R.,
Moshkowich, G., Murik, D., Shaul, H., Soceanu, andSoceanu, O. (2020). HeLayers: A Tile Tensors
Framework for Large Neural Networks on Encrypted
Data. arXiv e-prints, arXiv-2011.
[17] LeCun, Y., Burges, C. J., and Cortes, C. (1998).
The mnist database. MNIST handwritten digit
database. Retrieved February 14, 2023, fromhttp://yann.lecun.com/exdb/mnist/[18] Snucrypto. (n.d.). SNUCRYPTO/Heaan. GitHub.
Retrieved
February
15,
2023,
from
https://github.com/snucrypto/HEAAN