Abstract

Introduction
Deep neural networks have proven to be a promising tech- nology due to their ability to achieve competitive machine learning performance in diverse domains including com- puter vision. Ever since AlexNet achieved a top-5 error rate of 16.4% using Convolutional Neural Networks (CNN) for the image classification task on the ImageNet dataset Figure 1: In a conventional Machine Learning as a Ser- vice (MLaaS) scenario, both the data and model parameters are unencrypted. Existing works on secure inference assume only the data is encrypted. Our proposed approach encrypts both the data and model parameters. of 1.2 million high-resolution images as part of ILVSRC 2012 (Krizhevsky, Sutskever, and Hinton 2012) competi- tion, various CNN architectures (Simonyan and Zisserman 2014)(Szegedy et al. 2015)(Zeiler and Fergus 2013) have been developed to improve accuracy and other performance metrics. Though the continuous improvement in the per- formance of CNNs has hastened their adoption in broader computer vision applications such as mobile and embedded devices (Howard et al. 2017)(Phan et al. 2020), the large computing/memory resource requirements of CNNs present practical challenges in applying them in many applications. This is often overcome by outsourcing the network train- ing and/or inference computations to the Cloud, which is commonly referred to as Machine Learning as a Service (MLaaS). In MLaaS, there are three parties: (i) the end-user, who requires predictions on sensitive data, (ii) the cloud service provider, who has access to large computational resources, and (iii) the model owner, who has developed the inference model, trained using machine learning algorithms that may be proprietary. In conventional MLaaS scenarios, the cloud service providers typically have access to the inference mod- els as well as the content of the query data. In many appli- cations, such access to data and the model is undesirable because the query data/models could reveal sensitive infor- mation. Moreover, sharing of such data may be prohibited under emerging privacy regulations such as GDPR. This has led to the development of a variety of privacy preserving machine learning algorithms. In this paper, we focus on a specific class of privacy preserving machine learning algo- rithms that rely on Homomorphic Encryption (HE). Most of the approaches presented in the literature for en- crypted inference encrypt only the input data and do not en- crypt the model parameters. This is based on the assumption that the same entity owns the model and runs the cloud ser- vice. However, access to the model parameters could lead to competitive disadvantage for the model owner, when the model owner and cloud service provider are separate entities (e.g., a start-up providing the MLaaS service on a Amazon or Google cloud). Hence, it becomes imperative to protect both the input data and model parameters in such scenarios (see Figure 1). (Bost et al. 2015) discussed about encrypted models in the context of Hyperplane Decision, Navie Bayes and Decision Tree algorithm. In this work, we focus on the scenario where the model owner and cloud service provider are separate entities and the cloud provider is not trusted. Therefore, we encrypt the model parameters 1 in addition to the input data using the end-user's public key (as shown on Figure 1). The advantage of this approach is that the cloud service provider cannot derive any benefit from the model directly. The disadvantage of model parameter re-encryption using the specific client's public key is the increased compu- tation and communication burden of the model owner.
Contributions
The objective of this work is to develop the component building blocks for enabling generic CNN inference on encrypted data. We use the implementation of the CKKS scheme available in the HElib library (Halevi and Shoup 2020) for all our analysis. However, we note that the lack of a bootstrapping implementation for CKKS makes it impos- sible to perform inferencing on any arbitrary CNN. Hence, we design depth-constrained CNNs that operate within the multiplicative depth limitations of the underlying leveled HE scheme. Our contributions are four-fold: 1. MLaaS scenario for convolution neural networks is pro- posed where both the input data and model parameters are encrypted, thereby enabling a model owner to provide the MLaaS service on an untrusted cloud. 2. CNN inference on encrypted data using the CKKS scheme is presented, which eliminates the need for careful model parameter quantization. 3. Implementation and performance analysis of illustrative linear operators such as convolution and matrix multipli- cation as well as non-linear operations such as ReLU and maxpooling in the encrypted domain is provided. 4. Various multithreading strategies were explored and the analysis revealed that the optimal strategy involves a com- plex interplay between system resources and the compu- tations involved.
Background and Related Work
Homomorphic Encryption
Homomorphic encryption schemes enable a class of com- putations on the ciphertexts (i.e., encrypted data) with- out decrypting them (Acar et al. 2018). Let [ x ] denote the encryption of value x using a public key pk . A cryp- tosystem is said to be fully homomorphic (FHE) (Gentry 2009) if it enables computation of any arbitrary function f ( x 1 , x 2 , · · · , x d ) in the encrypted domain. In other words, there exists a function g such that f ( x 1 , x 2 , . . . , x d ) = D (g([x 1], [x 2], · · · [x d]), sk), where D represents the cor- responding decryption mechanism of the cryptosystem us- ing private/secret key sk. Recently, a number of FHE schemes have been proposed based on the hardness of the Ring Learning With Errors (RWLE) problem. The most well-known examples of such schemes are BFV (Brakerski 2012)(Fan and Vercauteren 2012), BGV (Brakerski, Gentry, and Vaikuntanathan 2014), TFHE (Chillotti et al. 2020), and CKKS (Cheon et al. 2017) cryptosystems, all of which support both additive and mul- tiplicative homomorphism. Among the well-known FHE schemes, only CKKS natively supports floating-point oper- ations, albeit with approximate (limited precision) computa- tions. Most of the previous attempts at encrypted inference coarsely quantize the weights of a neural network in order to satisfy the requirements of the underlying HE schemes (Gilad-Bachrach et al. 2016; Lou and Jiang 2019). In this paper, we explore the use of the CKKS scheme, which elim- inates the need for transformations of operands as integers. FHE is typically obtained by combining a leveled HE scheme with a bootstrapping operation. Since each HE com- putation increases the "noise" level of a ciphertext, a lev- eled HE scheme has a (parameter-dependent) limit on the computational (multiplicative) depth of the arithmetic circuit that can be computed, while still allowing for meaningful decryption. This limitation is overcome by using Gentry's bootstrapping technique (Gentry 2009), which "refreshes" the ciphertext and reduces its noise level (at the cost of re- lying on circular security). While bootstrapping can enable computation of arithmetic circuits with arbitrary multiplica- tive depth, it comes at a prohibitively high computational cost. For some FHE constructs such as CKKS, there is no existing implementation of true "Gentry-style" bootstrap- ping. This leads to an intractable conundrum. While mod- ern machine (deep) learning algorithms rely on their depth to achieve good generalization performance (Poggio et al. 2017), deeper networks are inherently more difficult to com- pute in the encrypted domain without bootstrapping. There- fore, optimizing the computational depth to achieve an ac- ceptable trade-off between accuracy and practical compu- tational feasibility is one of the fundamental challenges in encrypted deep learning.
Secure Machine Learning Using FHE
Every supervised machine learning algorithm involves two steps: (i) learning the model from training data; (ii) infer- ence on new unknown samples. We presume that an efficient model has already been trained from data available in clear. Our focus is on how to carry out encrypted inferencing with minimal leakage of information for all the parties in MLaaS while providing guaranteed privacy. Privacy preserving machine learning can have different challenges and threats based on what is required to be pri- vate. In (Al-Rubaie and Chang 2019), various types of at- tacks ( e.g. , reconstruction attacks, model inversion attacks, membership inference attacks) that could potentially reveal private/proprietary information from data/model have been discussed. The work also presents various cryptographic and non-cryptographic approaches to thwart these attacks. Key techniques that have been explored to achieve privacy pre- serving machine learning include differential privacy, secure multiparty computation (SMPC), and homomorphic encryp- tion. Homomorphic encryption has been considered more suitable for the cloud-based applications, but its practical feasibility is still in question. SMPC-based methods have been generally deemed more practical, but it requires inter- action between the client and server resulting in large com- munication cost. Differential privacy techniques lack the for- mal guarantees of security under assumptions of more con- servative attack models. Most of the recent work in privacy-preserving ma- chine learning (Gilad-Bachrach et al. 2016)(Juvekar, Vaikuntanathan, and Chandrakasan 2018) (Chou et al. 2018)(Dathathri et al. 2019) has primarily focused on se- cure inference where the model has already been trained in the clear. Once the model has been trained, these methods use either SMPC or homomorphic encryption techniques to enable inference. While CryptoNets (Gilad-Bachrach et al. 2016) rely only on homomorphic encryption to enable in- ferencing over private input data, Gazelle (Juvekar, Vaikun- tanathan, and Chandrakasan 2018) uses a combination of homomorphic encryption and garbled circuits to achieve two orders of magnitude speed improvement. Faster Cryp- toNets (Chou et al. 2018) have demonstrated improved per- formance over that of CryptoNets by leveraging sparsity properties and reduced the amortized inference time from minutes to seconds. CHET (Dathathri et al. 2019) provides an optimizing compiler for Fully Homomorphic Neural Net- work Inference. It basically provides high level user frame- work to automate the tuning of parameters for security as well as performance without degrading the accuracy. The framework provides high level and low level intermediate representations that can be linked with different homomor- phic libraries. Their results showed that automated homo- morphic circuits from the compiler outperform the circuits that were manually tuned. An encrypted inference service built on top of TensorFlow using secure multiparty computa- tion is discussed in (Dahl et al. 2018). Encrypted model was considered in (Bost et al. 2015) for Hyperplane Decision, Naive Bayes, Decision Tree algorithms, but not convolution neural network.
Convolutional Neural Network (CNN)
A CNN is a multi-layered neural network that is usually ap- plied to image/video data. Let X( ℓ ) and Z( ℓ ) denote the input and output of layer ℓ (ℓ = 1, 2,         , D) in the network. Each layer in the network performs specific arithmetic op- erations on the outputs of the previous layer and passes the result to the subsequent layer, i.e., X(l) = Z(l − 1) and Z(0) is the raw input data. This structure allows higher-level abstract features to be computed as non-linear functions of lower-level features (starting with the raw data). The first few layers of a CNN generally consists of the three types of operations. 1. Convolution: This is a linear operation, where the input to the layer 2 (X) is convolved with a kernel (W) to generate a filtered output Y . Let A i,j denote the element in the i- th row and j -th column of a matrix A . Suppose that the size of input X is ( M × N ) and the size of kernel W is ( P × Q ). Typically, one can pad the input with zeros to compute the filter responses at the edges. Let X be the zero-padded version of the input X .
where ⟨ a, b ⟩ is the inner product of two vectors α and b, W is the vectorized (flattened) version of the kernel W , and X m,n is the vectorized version of the input window (a window of size P × Q with X m,n as the top-left el- ement) selected from the padded image. For the sake of simplicity, the stride length is assumed to be 1 in equation (1). 2. Activation: In this step, a point-wise non-linear function is applied to the filter responses. The commonly used acti- vation functions are rectified linear unit (ReLU), sigmoid, and hyperbolic tangent (tanh). In this paper, we consider only the ReLU activation function, which is defined as follows:
3. Pooling: This is typically used for dimensionality reduc- tion, where multiple responses within a neighborhood are pooled together. While max pooling is often used for se- lecting the most dominant response in each neighborhood and is a non-linear operation, mean pooling computes the average response in each neighborhood and is a linear op- eration. The final output ( Z ) of the activation and pool- ing layers can be modeled as a function f of the filter responses, i.e., Z = f(Y), where f is non-linear. After the initial layers, the data is flattened into a vec- tor and a few fully connected (FC) layers are added to the network. The output of each node in the FC layer is often computed by applying a non-linear activation function to the weighted average of its inputs, which includes a bias term that always emits value 1. This can be mathematically rep- resented as Z = f(WX), where W is the weight matrix of the FC layer and f is the activation function. The final layer in a CNN is usually a softmax layer that provides an approximate probability distribution over the class labels.
Proposed Approach
In this work, we employ a typical CNN architecture as an illustrative example. Though FHE schemes allow arbitrary computations on the encrypted data, two key challenges need to be circumvented before applying a CNN on en- crypted inputs. The first challenge is the appropriate packing of input data to efficiently make use of the Single Instruction Multiple Data (SIMD) operations. The next challenge stems from the inherent inability of most FHE schemes to directly compute non-linear functions (with the exception of polyno- mials). Often, the non-linear function is approximated by an iterative algorithm or a polynomial function. Since the ap- proximation error is generally inversely proportional to the computational depth (using more iterations or higher-degree polynomials leads to less approximation error), the trade-off between computational depth and accuracy must be care- fully managed. Convolution  Operator: We employ the CKKS scheme for encrypted operations. The native plaintext in the CKKS scheme is a polynomial in the cyclotomic ring, which en- ables packing multiple plaintext values into different "slots" in the ciphertext. This ciphertext packing enables paral- lelization of addition and multiplication operations through SIMD operations. However, it must be emphasized that it is not possible to randomly access values in the individual slots of the ciphertext after packing. Since only a limited set of operations (e.g., rotation of values in the slots) are pos- sible within a ciphertext, the benefits of SIMD operations can be fully realized only when there is minimal interaction between the slots. Therefore, we follow a more conserva- tive and straightforward approach to ciphertext packing that amortizes the computational time by processing a batch of images in parallel. Let K be the number of slots available in the cipher- text. Given a batch of K images X (1) , X (2) , . . . , X (k) where each image is of size M × N , we represent the en- crypted image as a ciphertext matrix [ X ] of size M × N as shown in Figure 2(a). The ciphertext [ X ] m,n contains the encrypted values of pixel ( m,n ) from all the K im- ages, i.e. [X]m,n = [[x(1),[[x(2),[[x(x(x)]]. Sim- ilarly, the encrypted convolution kernel is also represented as a ciphertext matrix [W] of size P × Q , where P × Q is the kernel size 3 . However, as shown in Figure 2(b), the en- crypted weight value [ W p,q ] is repeated across all the slots of the ciphertext [ W ] p,q While the above simple packing approach greatly in- creases the memory requirement, it provides great simplicity in terms of implementing the convolution operator. One can directly treat the encrypted image matrix [X] and the en- crypted kernel [ W ]] as equivalent of the plaintext matrices X and W , respectively, and compute the convolution re- sults as the inner product between the vectorized versions of the selected image window and the kernel. Thus, equation (1) can be modified as follows:
When there is no padding involved, the above approach requires ( M − P + 1)( N − Q + 1) PQ ciphertext multipli- cations (henceforth denoted as CT-CT mult) and ciphertext additions (henceforth denoted as CT-CT add) and consumes a multiplicative depth of 1. Activation and Pooling Operators: In this paper, we im- plement a ReLU function using a polynomial approxima- tion. In (Gottemukkula 2020), a number of polynomial ap- proximations have been proposed for the ReLU function. Since one of our objectives is to minimize the computational depth, we choose the following polynomial of degree 2 for our approximation.
A linear transformation may be required to limit u within the appropriate range before the above polynomial approxi- mation is applied. The polynomial approximation of ReLU consumes only a multiplicative depth of 1 and requires 1 CT- CT mult, 2 CT-PT mults (multiplication a ciphertext with a plaintext operand), 2 CT-CT adds, and 1 CT-PT add (addi- tion of a plaintext operand to a ciphertext) operations. In terms of pooling, we consider mean pooling operator in a 2 × 2 neighborhood, which is a linear operation. While mean pooling does not involve any ciphertext multiplication (CT-CT mults), it does require 4 CT-CT adds and 1 CT-PT mult. Our activation and pooling operations can be summa- rized as follows:
where

Experimental Results
Environment
For FHE, we chose IBM's open-source library named HE- lib (Halevi and Shoup 2020) version 1.0.1. Since FHE based experiments require signficant compute capability and effi- ciency, we chose the natural API interface of HElib library through C++. The configuration details of the computing en- vironment are listed below: • System used: Virtual Machine ppc64le POWER9, • CPUs: 112 CPUs with 14 sockets, 1 core/socket, 8 threads per core •  Memory: RAM - 511 GB, (a) (b)
Impact of FHE parameters
Three key parameters define the operational characteristics of the FHE scheme and determine the security level ( λ ). The plaintext space of CKKS is the set of polynomials in the cyclotomic ring Z [ X ]/(Φ m ( X )), where Φ m ( X ) is the m th cyclotomic polynomial with degree given by Euler's totient φ(m). The number of slots (K) in the ciphertext is given by K = φ(m)/2. The second key parameter is L , which is the bitsize of the modulus of a freshly encrypted cipher- text. Since we are using a leveled HE scheme, L determines the depth of the circuit that can be evaluated without boot- strapping. Increasing L allows more computations (multipli- cations) to be performed before hitting the noise threshold. Finally, the parameter r determines the computational preci- sion in the encrypted domain. While larger values of m (and hence φ(m)) increase the security level λ, larger values of L and r generally decrease the security. In our experiments, we choose m = 2 16 (corresponding to φ ( m ) = 32768 and K = 16384), r = 35, and vary L depending on the compu- tational depth required. We first evaluate the impact of the depth parameter L on the computational time, security level, and ciphertext size. For measuring the computational time, we consider the aver- age time required for the adding/multiplying two ciphertexts (filled with constant values in all the slots) over 1000 trials. As shown in Figure 3, increasing L increases the computa- tional time and ciphertext size in a linear fashion. Moreover, the rate of increase is much higher for multiplication of two ciphertexts (CT-CT mult) than for the addition of two ci- phertexts (CT-CT add). Our analysis also indicates that for the first multiplication to be successful, a minimum value of L = 200 bits is required and each additional multiplication in the circuit (increase in the multiplicative depth by 1) re- quires an increase in the value of L by approximately 100 bits. It must be highlighted that the impact of L on the secu- rity estimate (λ) is non-linear. Note that evaluation of deeper (multiplicative depth) circuits would require large values of L . This in turn would require a large increase in the value of m to maintain the same level of security, consequently slowing down the computations significantly as well as in- creasing the memory requirements (due to larger ciphertext sizes). Therefore, it is critical to constrain the depth parame- ter L to a reasonable value and design the machine learning model to fit the depth constraint.
Dataset and Inference Results
We use the MNIST dataset for our experiments. This dataset consists of 6000 28 × 28 grayscale images of the 10 digits (0-9), along with a test set of 10000 images. We designed a simple CNN model that consists of a single convolutional layer with 28 filters (each having a kernel size of 3 × 3) without any padding, followed by ActPool1 layer (polyno- mial approximation of ReLU and mean pooling), a flatten- ing layer (whose output dimension is 13 × 13 × 28 = 4732), a single fully connected layer of size 4732 × 10, and a fi- nal softmax layer. The CNN is trained using the Keras li- brary for 10 epochs using a sparse categorical crossentropy loss function and ADAM optimizer with a learning rate of 0.01. The test accuracy of this simple CNN model on plaintext images was found to be 97.86%, which is only marginally lower than that of a CNN where the approximate ReLU function is replaced with the standard ReLU function (98.39%). Next, we focus on the problem of encrypted inference where both the inputs to the CNN and weights of the net- work are encrypted. To implement inference based on the above simple CNN model, the depth parameter L was set to 600 and the values of m = 2 1 6 and r = 35 were re- tained. The conservative security estimate based on these pa- rameters is 128 bits, which is obtained by using an in-built library call in HElib. Note that the inference process does not require the computation of the softmax function, which is monotonic in nature. Thus, the user can decrypt the out- put of the fully connected layer to infer the predicted class, which is the class with the maximum output at the FC layer. Based on the above HE parameters, the number of slots ( K ) available is more than the number of test images (10000). Thus, the complete inference process could be executed in the encrypted domain in one batch without any loss of test accuracy (97.86%). The time required for the execution of each layer in the CNN during encrypted inference is shown in Table 1. It must be emphasized that the reported timings in Table 1 are for evaluating a single filter on a single thread. The results for the complete network (all 28 filters) are shown in the next sub-section on multi-threading. As one would expect, the convolution layer is primary bottleneck in terms of num- ber of computations. On the other hand, the fully connected (FC) layer presents a significant challenge in terms of mem- ory bottleneck. Even for a single filter, the FC layer requires holding (( M − P + 1)( N − Q + 1)( R + 1)/4) ciphertexts in memory, where R is the number of classes (( R = 10 for MNIST). The number of ciphertexts gets inflated by a fac- tor of 28 when all the filters are considered, which makes parallelization more difficult.
Parallelization using multi-threading
HElib supports thread level parallelism (inherited from the underlying NTL library), which can speed up the execution time. Theoretically, the design of the proposed simple CNN allows us to compute the response of each convolution filter as well as the subsequent activation, pooling, and FC lay- ers independently. However, in practice this was not feasible due to memory constraints. Moreover, our computing envi- ronment has 112 CPUs, but the model has only 28 filters. Therefore, to make the best use of the available resources we had to implement a nested multi-threading strategy. In the nested multi-threading strategy, we spawn F (we choose F = 14 or F = 28) threads for the filters, and each filter thread in turn spawns C convolution threads (we choose C = 1, 3, 5 or 7). Each convolutional thread oper- ates on a horizontal partition of the full image. For example, when C = 7, the 28 × 28 image is partitioned into 7 horizon- tal sub-images with appropriate overlap (to avoid boundary issues). Since the approximate RELU is a pixel-wise opera- tion, it can be applied to each horizontal partition indepen- dently. But since the mean pool operation could span across partitions, we have to wait for all the convolution threads to complete before meanpool is applied. Finally, due to mem- ory constraints, we could not proceed with the FC layer di- rectly. Instead, we waited for all the filter threads to com- plete and then spawn H new threads, one for each of the 10 classes (columns of the FC layer matrix). As earlier, these H class threads further spawn J channel threads to parallelize the matrix multiplication task along rows of the FC layer matrix. Figure 4 shows the execution times for the above nested
Table 1: Execution time for different layers of the proposed simple CNN model. multi-threading strategy. For this experiment, we fix H = 10 and J = 1 and vary F and C . From this figure, we can ob- serve that having more filter threads generally leads to faster execution. Since the maximum number of available threads is 112, we observe that utilizing only 70-80 threads provides the optimal results because it achieves the best compromise between the number of threads and thread efficiency. In par- ticular, when F = 28 and C = 3, 84 threads are utilized in total and it results in the lowest execution time of 864 sec- onds. Similarly, when the number of channel threads J is increased, the best results were obtained when J = 7 lead- ing to a total execution time of 561 seconds. Thus, the proposed multi-threading strategy leads to ap- proximately 40 times improvement over the time required for computing all the 28 filters on a single thread (28 × 812[Table1] = 22736 seconds). This is achieved by utiliz- ing 70 – 80 threads on average. If the memory constraints can be surmounted, there is scope for further refinement in the proposed multi-threaded strategy. It must be emphasized that the values reported in the above paragraphs are the actual execution times and not the amortized time. Since K images are processed in parallel exploiting the SIMD mechanism, the amortized time per im- age could be orders of magnitude lower. Since K  = 16384 in our experiments, the amortized execution time for infer- ence on a single image can be estimated as 34 milliseconds. Note that most of the reported literature on inference based on encrypted MNIST data (e.g., (Gilad-Bachrach et al. 2016; Chou et al. 2018)) use the BFV encryption scheme available in the Microsoft SEAL library implemented on a x86 plat- form. Hence, it is not possible to directly compare the exe- cution times reported in this work with those reported in the literature. However, we do note that the wall-clock run time reported in (Gilad-Bachrach et al. 2016) is 250 seconds, with only 10 5 × 5 filters and a stride length of 2, which is com- parable to our run time of 561 seconds (with 28 filters and a stride length of 1). Moreover, unlike (Gilad-Bachrach et al. 2016), the weights of the convolution and fully-connected layers are encrypted in our proposed implementation. It must be highlighted that the ciphertext packing strat- egy used in this work has high latency for a single inference. This could be addressed using different packing strategies as discussed in (Brutzkus, Gilad-Bachrach, and Elisha 2019), which can reduce the latency at the cost of decreasing the throughput. For example, for single image inference with a different packing strategy (the entire image is encrypted within a single ciphertext), we were able to achieve a to- tal inference time of 8.8 seconds with encrypted model pa- rameters. When the model parameters are not encrypted, the latency can be further reduced to 2.5 seconds, which is com- parable to the latency of 2.2 seconds reported in (Brutzkus, Gilad-Bachrach, and Elisha 2019).
Conclusions and Future Work
In this paper, we attempted to address the computation com- plexity in a convolutional deep neural network for encrypted processing. The approach presented in the paper describes a secure way to compute convolution, non-linear activation and pooling layers using power of the CKKS FHE scheme. We exploited two features to tame the complexity: SIMD ciphertext packing and thread level parallelism. In addition, we empirically set the depth of the circuit to further mini- mize the execution time. Using a recent method to approx- imate ReLU, we were able to build a bootstrap-free ReLU function in FHE. Since the max pooling layer requires boot- strapping, we replaced it with a mean pooling layer. Overall, we show that it is possible to implement encrypted inference in reasonable time by investing in the right CNN design and parameter choices. In future, we will proceed to integrate other layers with the end-goal of support full inferencing and training on en- crypted data. We also observe that an overall latency of 561 seconds could be too high for some applications. The limi- tation of the ciphertext packing strategy used in this work is high latency for a single inference and high memory require- ment. This could be addressed using different packing strate- gies as discussed in (Brutzkus, Gilad-Bachrach, and Elisha 2019), which can reduce the latency while also decreasing the throughput. Since all the ideas contained in (Brutzkus, Gilad-Bachrach, and Elisha 2019) are practical HE imple- mentation nuances that do not require any change to the ma- chine learning model, they can be readily applied to the pro- posed approach. We leave this analysis for future work.
Acknowledgement
The work was initiated when authors Nandaku- mar/Ratha/Pankanti were part of IBM Research. We thank IBM, USA for providing us the computing facilities. Author Kumar acknowledges Infosys Foundation for the financial support to IIIT-Bangalore through the Infosys Foundation Career Development Chair Professor.