Ensemble Homomorphic Encrypted Data Classification
Authors: Dana R Alsagheer
Hadi Mansouifar
lei Xu
Qian Lou
Weidong Shi
lin chen
Venue: Submitted to ICLR 2023 (2023)
Keywords: Machine Learning Privacy
Homomorphic Encrypted Data Classification
Ensemble Learning

Abstract: Homomorphic encryption (HE) is encryption that permits users to perform computations on encrypted data without first decrypting it. HE can be used for privacy-preserving outsourced computation and analysis, allowing data to be encrypted and outsourced to commercial cloud environments for processing while encrypted or sensitive data. HE enables new services by removing privacy barriers inhibiting data sharing or increasing the security of existing services. A convolution neural network (CNN) with shallow architecture can be homomorphically evaluated using addition and multiplication by replacing the activation function, such as ReLU, with a low polynomial degree. To achieve the same performance as the ReLU activation function, we study the impact of applying the ensemble techniques to solve the accuracy problem. Our experimental results empirically show that the ensemble approach can reduce bias, and variance, increasing accuracy to achieve the same ReLU performance with parallel and sequential techniques. We demonstrate the effectiveness and robustness of our method using three data sets: MNIST, FMNIST, and CIFAR-10 

Under review as a conference paper at ICLR 2023
THE IMPACT OF ENSEMBLE ON HOMOMORPHIC ENCRYPTED DATA CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
Homomorphic encryption (HE) is encryption that permits users perform computations on encrypted data without first decrypting it. HE can be used for srvvacyprreeevingg outsourced computation and analysis, allowing data to be encrypted and outsourced to commercial cloud environments for processing while encrypted or sensitive data. HE enables new services by removing privacy barriers inhibiting data sharing or increasing the security of existing services. A convolution neural network (CNN) can be homomorphically evaluated using addition and multiplication by replacing the activation function, such as ReLU, with a low polynomial degree. To achieve the same performance as the ReLU activation function, we study the impact of applying the ensemble techniques to solve the accuracy problem. Our experimental results empirically show that the ensemble approach can reduce bias, and variance, increasing accuracy achieve the same ReLU performance with parallel and sequential techniques. We demonstrate the effectiveness and robustness of our method using three datasets: MNIST, FMNIST, and CIFAR-10.
INTRODUCTION
Homomorphic encryption (HE) is private artificial intelligence (AI) application that allows users to perform computations on encrypted data without decryption, and the result of calculations will be in encrypted form when decrypted. As result, HE is ideal for working with sensitive data to privacy-preserving outsourced storage and computation. In other words, HE allows data to be encrypted and outsourced commercial cloud environments for processing, all while encrypted. Deep learning on the cloud enables designing, developing, and training deep learning applications faster by leveraging distributed networks and HE and cloud computing, allowing large datasets to be easily ingested and managed to train algorithms. It will enable deep learning models to scale efficiently and lower costs. HE scheme adopting bit-wise encryption performs arbitrary operations with an extensive execution time but to shorten execution time, a method adopts HE scheme to encrypt integers or complex numbers. An HE scheme is usually defined only supports two finite field operations, addition, and multiplication, which can behave entirely differently than floating point numbers operations used in typical AI applications. Accordingly, functions commonly used in deep learning, such as ReLU, Sigmod, and max-pooling, are not compatible with HE OOba 2020). To address this issue, the polynomial activation function evaluate CNN to address since HE straightforwardly supports additions and multiplications. Due to the increased complexity in computing circuits with nested multiplications, it is desirable restrict the computation low-degree polynomials (Gilad-Bachrach
finite field, so
al. 2016). However, replacing ReLU with a low-degree polynomial, combined with other techniques such as batch normalization (BN) SSLoffeandC.Szeeeddy 2015), still suffers from high bias, variance, and low accuracy. Intuitively, the ensemble is a machine learning approach that consists of a set of individual weak learning models working sequential or parallel. The outputs are combined with a decision fusion strategy to produce a single and better performance than any single model (Huang et al.) 2009). That motivates us to introduce the ensemble approach to enhance accuracy by reducing bias and variance when an HE scheme. Ensemble learning has been recently known to be an essential reason for improving the performance of deep learning models because sub-models do not strongly depend each other, even though they are trained jointly. Moreover, they exhibit ensemble like behavior in that their performance smoothly correlates with the number of valid paths and enables them to work with the depth of HE network (Veit al. 2016). The success of ensembles is due

---

Under review as a conference paper at ICLR 2023
to the focus getting better-performed new model by reducing the bias variance of weak learners by combining several them to create a strong learner that achieves better performances. It can be differentiated as bagging will focus getting ensemble model with reduced variance, whereas boosting and stacking will produce strong models less biased than their components (Sagi & Rokach 2018). Generally, ensemble techniques are considered better performance due to lower error and overfitting than any individual method, leading to better performance the test If each learner might have a sort of bias, or variance, combining them can reduce this bias. Generally, we can say ensemble is one of the most approaches to better performance (Zhou et 2002).
of the best approaches to
In this paper, we propose an ensemble approach to improve the accuracy of HE-based privacy- preserving data inference with deep neural networks (DNN) for both sequential and parallel ensembles when replacing the ReLU activation function with polynomials. We applied customized sequential ensemble techniques that can be applied different numbers of CNN models, which will be involved in the multi-class prediction while using polynomials as activation functions in the hidden layer. We applied the bagging method for the parallel ensemble technique and studied the ensemble's 's impact bias and variance. Our results indicate that ensemble could significantly reduce variance and boost accuracy. To the best of our knowledge, this is the first work to investigate the ensemble approach in the context of HE-based privacy-preservie the accuracy problem caused by replacing activation function with polynomials. Most of the previous efforts were focused on choosing better single polynomial increase the accuracy. In contrast, our work focuses on improving the low accuracy classification model by combining weak models considering the requirements for encrypted data. Figure to increase the accuracy while using polynomials in the hidden layers in the convention network.
inference with DNN solve
X illustrate an ensemble approach
Ensemble Methods
Avenaging
Blending Boosting
Awenaging
Replace ReLU in Hidden layers polynomials.
Apply ensemble approach to enhance accuracy by comping models to get better predictions.
Figure I: An ensemble approach to increase the accuracy while using polynomials in the hidden layers in the convolution network.
In summary, our contributions to this paper include:
We have investigated the impact of using the sequential ensemble technique on the accuracy, and the result indicates significant improvement, reaching the same ReLU performance while using polynomials.
We have studied the impact of the parallel ensemble technique, especially bagging, and our result shows an improvement in the variance result without increasing bias.
We have demonstrated the effectiveness and robustness of our method using three datasets; MNIST (LeCun 1998), FMNIST (Xiao et al. 2017). and CIFAR-I0 (Krizhevsky et al. 2014).
The rest of the paper is organized as follows: Section 2 reviews related works and summarize the proposed approach's 's advantages. Section discussss the background. Section S presents the details of the proposed method, and demonsstates the effectiveness of the proposed approach with experimental results. Finally, we conclusion the paper in Section
2 RELATED WORKS
Replacing the ReLU activation function with polynomial: To study the impact of replacing the ReLU activation function with the polynomial activation function in neural networks, several works
2

---

Under review as a conference paper at ICLR 2023
Table 1: Accuracy performance in related research that have used the same dataset and the type of pooling.
Accuracy (6)
CIFAR-
Reserech
Pooling
BN
MNIST
MNIST
10
Baaad-Bachrachhecccoo al. 2016
Average
98.95
(Cabannne et al. 2017))
Average
Yes
99.30
Ciang al) 2018)
Scaled mean
98. 10
(ll Badawi 2018
Average
99.00
77.55
such (Jiang et al. 2018) and (Bourse et al. 2018) point out the challenging caused by replacing the ReLU function, and the major challenge is severely decreasing in the accuracy. 2018, (Al Badawi et al. 2018) tested the inference of convolutional neural network (CNN) using graphical processing and achieved an accuracy of 99% on the Modified National Institute of Standards and Technology (MNIST) dataset while achieving low classification accuracy (77.55% (CIFAR)-10 dataset. (Ali ct al.) 2020) studied a deep network with nine ReLU activation layers. CIFAR-100 datasets, using an improved ReLU and min-max approximation. However, they used very deep network not capable of HE. Additionally, (Onoufriou et al. (222)) investigated how FHE with deep learning can be used at scale toward accurate sequence prediction. They reached an accuracy of 87.6% on average; however, they used confidential data that cannot be made public. Some work such as
et al.) 2016), (Jiang et al. 2018). and (Chabanne et al. (2077) only studied the accessible gray-scale dataset MNIST because it is easy to classify with high accuracy. Table illustrates the accuracy of each research in the same dataset that we have used.
Ensemble techniques: Numerous works studied the impact of the ensemble approach in solving the problem of bias and variance. They point out that one of the main reasons behind the success of the ensemble methods is combining multiple ways, such averaging and voting, where the ensemble model performs better than any of the individual models. (Dietterich) 2000) reviewed ensemble methods and explained why ensembles could often perform better than any single model. The reasons for the success of ensemble learning include: statistical, computational, and representation learning. Bias-variance decomposition of the expected misclassify rate, the most commonly used loss function in supervised learning and frequency- biased and shows how to correct for this bias and study the decomposition of various algorithms and datasets (Kohavi et al. 1996). (Webb & Zhhnng 2004), investigated the different mechanism that allows for trade-offs the bias-variance decomposition of the error ensemble methods, such as the random forest and application of post-processing algorithms. (Breiman (001) showed heuristically that the variance the bagged predictor less than the original predictor and proposed that bagging is better in higher dimensional data. (Bühlmann & Yu 2002) explained how bagging gives smooth, slight variance and mean bias error, introducing sub bagging and half sub bagging. A high-bias model is caused by low degree of the polynomial will result in underfitting. In contrast, high variance caused when the model has many degrees of freedom, like high-degree polynomial model, and causes overfitting.
estimation of the decomposition terms, is
BACKGROUND
Machine learning aims to build model that performs well with the training and the testing Iata.It is crucial to understand prediction errors, bias, and variance, then a trade-off between a model's ability to reduce bias and variance (Geman al.) 1992), (Brown al. 2005). (Domingos) 2000), (Rocks)
& Mehtal (022). The proper calculation of these errors would increase the accuracy and avoid the mistake of overfitting and underfitting, both leading poor predictions (Bullinaria 2015). (Dong
et al. 2020).
Overfitting: A model performs well the training but does not perform well the test set; overlitting occurs if the model shows low bias but high variance. Training the model with so much data will learn from our dataset's noise and inaccurate data entries. As a result, the model does not categorize the data correctly.
Underfitting: It occurs when the machine learning algorithm cannot capture the data trend. Intuitively, underfitting happens when the model or algorithm does not sufficiently fit the data. Specifically, underfitting occurs if the model algorithm shows low variance but high bias.
3

---

Under review as a conference paper at ICLR 2023
The error consists of the essential three components: Bias and variance, and irreducible errors as follow:
Error(z)= Bais + V Variance + Irreducible.Error
To build good model, need to find good balance between bias and variance such that it optimizes the total error. Since we can't control Irreducible error, we can handle bias and variance by minimizing the sum of the bias and variance contributions to the generalization error. The definition of Bias Error and Variance Error is as follows:
Bias is the average difference between predicted and actual results. High bias means we are getting low-performance Under-fitingg
Bias Error:
Variance Error: Quantify the expected value difference the same way observation on that time model is over-fitting It is caused by a very complex model on simple data. Train a model that shows high variance near 100% accuracy on training data. However, checking the present data model fails to predict the correct result.
Managing bias and variance is crucial achieving a well-performing training model prediction. There will be a trade-off between minimizing the bias and minimizing the variance as given (Geman
al. 1992).
Brown et al. 2005), (Domingos 2000):
1
Elo d bios? M var + (1
covar
M
bias =
M var Σ F0 -()
M var Σ (0ox=11or]]]
var
Cocar M(M 1) ΣΣ Elo, EOo,ll0o, E|0,]l
jØ1
Where is each model's ' target O2 output, and M the ensemble size. Here, the bias term measures the average difference between the base learner and the model output, var indicates their average variance, and covar is the covariance term measuring the pairwise difference of different bases. According (Bylander & Tate 2006) and (Loughrey & Cunningham 2005), there is single solution to avoid overfitting. However, we can use techniques to eliminate overfitting, such as increasing the data size, cross-validation, early stopping, regularization, and ensemble technique.
3.1 ENSEMBLE TECHNIQUE
Neural network models are a nonlinear method that enables it to learn complex nonlinear relationships in the data. However, will make the models extremely sensitive initial conditions in terms of random weights and the statistical noise in the training dataset. Accordingly, every time the model is trained, will learn a different version from the mapping input and output. As result, the ensemble can successfully reduce the variance of neural network models by combining the perdition of multiple models instcad single model to ensure that the most stable and best possible prediction is considered. According to (Krizhevsky al. 2017), the performance of one model compared to ensemble predictions averaged over two, five, and seven models are distinct. Averaging the predictions of five similar CNN's gives error rate of 16.4% whereas averaging the predictions of two pre-trained CNNs with the same five CNN's gives an error rate of 15.3%. Indeed, an ensemble improves the quality of essential algorithms and increases the underlying algorithms' diversity. Also, it aims to make it sufficiently diverse, and the correct operation of other algorithms will compensate for the errors of individual algorithms. The essential idea of ensemble methods is to reduce the bias-variance of weak learners by combining several of them to create strong learner called the ensemble model that achieves better performances. The vulnerable learners or base models can be used building blocks for designing more complex models by combining several. The primary models' performance is feeble because they are of high bias due to a low degree of freedom models and high variance to be robustly caused by the high degree of freedom models (Roccal 2019). The number of models in the ensemble is essential to keep small due to computational expense in training models and the diminishing returns in performance from adding more ensemble members. We can use different approaches, such as various datasets, models, or combinations.

---

Under review as a conference paper at ICLR 2023
3.2 CATEGORIES OF ENSEMBLE METHODS
The ensemble strategies are broadly categorized into sequential ensemble and parallel ensemble techniques:
Sequential ensemble techniques:
Generate base learners in a sequence, creating dependency between each base Iearner. The model's performance is improved by assigning higher weights to previously misrepresented learners (Odegua 2019).
Parallel ensemble technique: The base learners are created in a parallel format where the methods take advantage of the similar generation of base learners to encourage independence, significantly reducing the error between the base learners. Most ensemble techniques apply single algorithm in base learning, resulting in homogeneity in all base learners. Other methods involve heterogeneous base learners, giving rise to heterogeneous ensembles, which means base learners are learners of distinct types. contrast, homogeneous base learners are base learners of the same kind with similar qualities.
Input
hevi!
hav
houp
Classifier
Classifier
Classifier
Output
Output
Output
Ensemble
Oebrit
Oxteitt
OMtan3
Ensomble
(A)
output
Figure 2: An ensemble approach to increase the accuracy while using polynomials in the hidden layers in convolution network (A)Sequential ensemble technique (B) Parallel ensemble technique
Generally speaking, existing ensemble learning methods can be grouped into three types:
Bagging: Homogeneous weak learners independently learn from each other in parallel and combine them following some deterministic averaging process (Breiman 1996). The model is called stable when is less sensitive to small fluctuations in the training date (Alelyani] [021). Bagging is more useful when the model is unstable, while bagging is not valuable for improving the performance in stable models.
Boosting: Homogeneous weak learners sequentially learn in an adaptive way where the base model depends on the previous model and combines them following a deterministic strategy (Freund 1995).
Stacking: Heterogeneous weak learners, learning in parallel and combining and training meta- model and the output prediction based the different weak models' predictions (Wolpert 1992..
But other variant methods are also possible based specific problem needs.
3.3 ADVANTAGE AND DISADVANTAGES OF ENSEMBLE LEARNING
Advantages of Model Ensembles
Ensemble methods have higher predictive accuracy compared to the individual models.
Ensemble methods beneficial when there is both linear and non-linear type in the dataset; by combining different models to handle this type of data.
With ensemble methods, bias and variance, can be reduced, which avoids underfitting and overfitting issues and makes it less noisy with more stability.
Ensembles are not always better because they cannot help unknown differences between the sample and the population. Ensembles should be used carefully due to their difficulty in interpreting.
5

---

Under review as a conference paper at ICLR 2023
Using the ensemble approach should consider the cost of creating and computing, increasing the complexity of the classification, deploying the ensemble carefully, and being hard to implement in real-time platform. Generally, ensembles provide higher predictive accuracy by creating lower variance and bias and providing deeper understanding of the data. However, selecting the model and the suitable ensemble methods crucial since wrongly choosing the technique will lead to lower predictive accuracy than an individual model.
PROPOSED METHOD
Our proposed method is to replace the ReLU activation Function that is not capable of HE in the hidden layers with polynomial function, then apply the ensemble technique as an extra step boost the accuracy of the classification model. We believe the ensemble will solve the accuracy problem caused by using the polynomial activation function hidden layers. We study the accuracy for both parallel and sequential.
4.1 SEQUENTIAL ENSEMBLE TECHNIQUES
We create an ensemble multiple CNN models. This ensemble is applied to the multi-class prediction of the three datasets. First, we define the individual CNN model, which is trained sequentially. Every individual model gives its prediction, then the final prediction of the ensemble model will be the most frequent prediction by all the individual CNN models. Only ReLU functions in the hidden layer are replaced:
of (2) mar (z; 0)
With the low-degree polynomial activation function:
ppoly(x) x= +
We choose d polynomials according (Ali et al.) 2020) will lead to better performance, and higher than degree-2 polynomials would result in overfitting
Architecture of convolution network: We train on MNIST, FMNIST, and CIFAR-10 with polynomial activation functions. The structure of the CNN model is as follows:
Two convolutional layers: first layer with 32 filters of size second 46 filters of size 3. Apply after every convolutional layer average pooling and batch normalization.
After a dropout layer, the network results are parsed from final fully connected layer of ten neurons.
4.2 PARALLEL ENSEMBLE TECHNIQUES
We parallel ensemble approach bagging to study the impact of variance and bias due to the ease of implementaiion using Python libraries such as scikit-learn and the ease of combining base learners' estimators' predictions improve model performance (Alelyani 2021)
Implementation Steps of Bagging:
Step 1:
Create multiple subsets from the original dataset with equal tuples, selecting observations with replacements.
Step 2: Create base model on each subset.
Step Each model is learned in parallel with each training set and independent of each other.
Step 4: Averaging the predictions from all the models to calculate the final prediction.
To implement the ensemble,
python libraries, including scikit-learn, etc. We have submitted all codes as supplementary materials.
5 EXPERIMENTS AND EVALUATION
5.1 DATASETS
We have tested the proposed method on three different datasets. Here, we review the characteristics of the tested datasets:

---

Under review as a conference paper at ICLR 2023
MNIST [37]: It's a handwritten digits database including training set of 60,000 examples and test set of 10,000 examples containing 28 x 28 grayscale images with ten classes.
Fashion-MNIST [38]: It's a dataset of images consisting of a training set of 60,000 examples and a test set of 10,000 examples. Each example is 28 28 grayscale image associated with label from 10 classes.
CIFARIO [39]: This dataset consists of 32 x 32 color images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
5.2 METRICS OF REPLACING RELU WITH POLYNOMIAL IN PRECISION, RECALL, AND FI-SCORE
In this section, we study the effect on Precision, Recall, and Fi-score by replacing ReLU with a polynomial. Following the definition of:
Precision: is the ratio of correctly predicted positive observations to the total predicted positive
observation (Sokolova et al. 2006), (Cook& & Ramadas 2020), (Yacouby & Axman 2020).
Precision = (P/)TP+FP)
The ratio of correctly predicted positive observations to all observations in the actual class observations to the total predicted positive observation (Sokolova et al. 2006). (Yacouby & Axman 2020
Recall:
Recall = TP/TP+FN
F1-score: is the weighted average of Precision and Recall. The score considers false positives and false negatives into account observations to the total predicted positive observation (Sokolova et al.,
2006), Yacouby & Axman 2020).
F1 Score = (Recal Precision) / (Recall + Precision)
where True Positives (TP), True Negatives (TN), False Positives (FP) False Negatives (FN) In this section, we calculate Precision, Recall, and Fl-score for all three datasets using ReLU and polynomials for activation function; we study the effect of replacing ReLU in hidden layers in each class of datasets. Table - calculates Precision, Recall, and F1-score for MINST, Fashion- -MIIST, and CIFAR1O by using ReLU and polynomial. Precision, Recall, and Fi-score results drop while using polynomials in the hidden layers as activation function, indicating a sharp performance decrease the result. For instance, in class in the dataset MINST, the F-1 score drops from 0.991 to 0.938 when using the polynomial.
Table Calculate Precision. Recall, and Fl-score for MINST, Fashion-MNIST, and CIFARIO by using ReLU and polynomial where C is the class.
C27
C-8
C.9
Precision
0.986
0.986
0.981
0.994
0.982
0.974
0.11
0.990
0.986
0.994
ReLU
Recall
0.996
0.998
0.972
0.959
0.987
0.927
0.930
0.991
0.968
0.992
MNIST
F1-Score
0.991
0.992
0.976
0.984
0.952
0351
0.980
0.958
Precision
0.930
0.833
0.911
0.839
0.897
0.871
0.896
0.896
0.896
0.896
Poly
Recall
0.966
0.57
0.200
0.856
0.809
0.910
0.820
0.946
0.808
0.801
F FI-Score
0.938
0.968
0.420
0.874
0.850
0.890
0.808
0.852
0.532
0.846
Precision
0.155
0.973
0.819
0.915
0.847
0.962
0.652
0.972
0.976
0.942
0.992
0.850
0.829
0.763
0.724
0.920
0.920
0.978
0.970
F-MNIST
F F--Score
0.822
0.982
0.834
0.870
0.803
0.971
0.686
0.945
0.997
01956
Precision
0.804
0.999
0.920
0.871
0.162
0.961
0.108
0.875
0.813
0.662
Poly
Recall
0.638
0.921
0.441
0.669
0.839
0.773
0.590
0.693
0.888
0.983
F F1-Score
0.739
0.944
0.944
0.773
0.272
0.857
0.138
0.774
0.849
0.762
Precision
0.724
0.225
0.548
0.553
0.645
0.673
0.750
0.761
0.871
ReLU
Recall
0.790
0.899
0.713
0.507
0.614
0.569
0.808
0.781
0.799
0.847
CIFAR_IO
F F1-Score
0.756
0.850
0.620
0.530
0.627
0.617
0.777
0.771
0.824
0.813
Precision
0.606
0.742
0.482
0.363
0.631
0.429
0.775
0.651
0.815
0.601
Poly
Recall
0.732
0.748
0.474
0.450
0.484
0.593
0.593
0.745
0.614
0.764
F FI-Score
0.663
0.745
0.478
0.413
0.548
0.498
0.498
0.649
0.700
0.762
5.2.
RESULT SEQUENTIAL ENSEMBLE TECHNIQUES
The Sequential ensemble techniques results are shown in Table 3 illustrating the resulting impact
on accuracy. The result of sequential ensemble techniques can be summarized as follows:
7

---

Under review as a conference paper at ICLR 2023
We measure the accuracy using polynomial and ReLU in the hidden layers. It decreases around 5%, 14%, and 10% in MINST, F-MINST, and CIFAR-10 when ReLU replaced with polynomial activation function.
After applying the sequential ensemble approach, the accuracy increased about 4%, 8%, and 2% in MNIST, F-MNIST and CIFIAR-10.
After five iterations, we successfully increased the accuracy and achieved performance at the same level ReL.U on average.
The reason behind the higher accuracy is due to the implementation of the combination of models by aggregating the output from each model with the effects that reduce the model error and maintain the model's generalization.
The sequential generation of base learners promotes the dependencies between the base learners. The model's performance is improved by assigning higher weights to previously misrepresented learners. Due the sequential generation of the models, the time complexity is high. The overall delay will be the total time for all the sequential sub-model classifications.
Table 3 iillustrates the resulting impact on accuracy using sequential ensemble techniques.
Table 3: The accuracy of the individual model with ReLU and polynomial activation function with sequential ensemble techniques where = number individual modes and n=l polynomial without applying ensemble.
MINST
MINST
CIFAR-10
ReLU Polynomial
99%
89%
83%
94F
73%
73%
Ensemble
95%
75%
Number of Individual model
N=4
96%
86%
77%
K=b
99%
89%
83.2%
subsectionResult of Parallel ensemble techniques The parallel ensemble's primary purpose to reduce variance without increasing bias due to model averaging. The parallel ensemble techniques results can summarized as follows:
We study the impact of parallel bagging ensemble techniques on average expected loss, bias and variance.
By replacing the ReLU with polynomial for both three datasets, the values in both bias and variance increase, which negatively affects the accuracy.
We notice a significant improvement in all datasets even after three estimators in the variance reduction without increasing the bias, which is the expected result to decrease the error. For instance, in 4able44iu the variance value drops from 0. 0.5 to 0.076 with a basis reduction from 1.748 1.130.
We achieve the same average result ReLU in variance and bias after 30 estimators without increasing bias. For instance the variance value is 0.019, almost the same with or better than ReLU 0.038 for Minst and Fashion-MNISS datasets.
We attain slight improvement after 100 estimates resulting in variance and bias. For instance, in table 5 the improvement is about 0,003, and no further improvement after that. There is trade- off between time complexity and accuracy. As described earlier, our goal is reduce bias and variance to increase accuracy performance.
For CIFAR-10, 10, it takes more time to reach the same result as ReLU about 100 estimators due to the complexity the data table T illuttratd the result.
The main reason for improving the variance value bagging that dividing the data into different groups makes the data more distinguishable linearly. As result, bagging can compensate for the performance gap between ReLU and polynomial activation functions in neural networks by decreasing the variance without increasing the bias.
In parallel ensemble techniques, base learners are generated in parallel format utilizing the parallel generation of base learners to encourage independence between the base learners. The independence of base learners significantly reduces the error due the application of averages. addition, due to the parallelism, execution will solve the sequential ensemble problem time computing.
8

---

Under review as a conference paper at ICLR 2023
The proposed bagging algorithm is experimentally proven to be very effective in improving classification accuracy. The results suggest that the bagging approach is very stable regarding feature selection due to the intrinsic power of reducing learning variance. Practically, the bagging concept creates new models, and the new mean shows a more actual picture of how those individual samples relate to each other in value. The more defined boundaries, the better the correlation between models, which reduces variance without increasing the bias. However, bagging slows down and grows more intensive the number of iterations increases; clustered systems or large number of processing cores will solve the problem of fast execution on large test sets.
Table 4 Table 5 and Table 6 and variance that accrued by using polynomial and bagging ensemble in MNIST, F-MNIST, and CIFAR-10.
illustrate illustaatt that the bagging ensemble reduces the average expected bias
Table 4: The average expected loss, bias, and variance for ReLU, polynomial, and bagging ensemble for Fashion-MNIST!
Average expected loss
Average bias
Average variance
ReLU Polynomial
I.677
1.378
0.038
1,784
0.175
Ensemble
N=3
1.106
1,030
0.076
Number of estimators
1.10
0,980
0.029
03899
0,970
0.019
NETOI
0.933
0,960
0.01T
Table 5:
The average expected loss, bias, and variance for ReLU, polynomial, and bagging ensemble for
MNIST.
Average expected loss
Average blas
Average variance
0.094
Polynomial
1.300
1.124
0.175
Ensemble
N=3
1.106
1.130
0.076
Number of estimators
1.101
0.029
N=30
0.989
0.970
0.019
N=100
0.981
0.963
0.016
Table 6: The average expected loss, bias, and variance for ReLU, polynomial, and bagging ensemble CIFAR-I0.
Average expected loss
Average bias
Average variance
ReLU Polynomial
7.483
7,381
0,102
[1.44]
9,816
1.625
Ensemble
N=3
8.687
8.643
0.043
Number of N-10
8332
8.211
0,641
estimators
N=100
7.64
7,375
0.032
Our goal is not outperform the state-of-the-art non-HE results on MNIST, Fashion-MNIST, and CIFAR-10. We aim to solve the accuracy problem and prove that the ensemble can close the accuracy gap caused by replacing the ReLU function with polynomial activation and improve the accuracy significantly to achieve equal performance as ReLU. Furthermore, the ensemble can be combined with other approaches, such as batch normalization, which makes the network more robust against gradient vanishing or exploding (Bjorck et al. 2018).
CONCLUSION
In this work, we apply an ensemble approach to solve the problem of decreasing accuracy when replacing the ReLU with a polynomial activation function in the hidden layers in HE-based ML. HE scheme cannot support comparison, division, and exponential operations in a straightforward manner; some functions used in deep learning. such as ReLU, Sigmod, and max-pooling, are incompatible with HE. Polynomial activation functions are considered to address this issue because an HE scheme can straightforwardly support polynomial-based calculations. However, using polynomials as an activation function causes low accuracy. We propose to apply the ensemble approach to solve the accuracy problem caused by replacing ReLU with low-degree polynomial. Our experiment results show increased accuracy in three datasets and achieve the same performance ReLU in parallel and sequential ensemble techniques.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Ahmad Al Badawi, Jin Chao, Jie Lin, Chan Fook Mun, Jun Jie Sim, Benjamin Hong Meng Tan, Xiao Nan, Khin Mi Mi Aung, and Vijay Ramaseshan Chandrasckhar. Towards the alexnet moment for homomorphic encryption: Henn, thefirst homomorphic enn on encrypted data with gpus. arXiv e-prints, pp. arXiv-1811, 2018.
Salem Alelyani. Stable bagging feature selection on medical data. Journal of Big Data, 8(1):1-18, 2021.
Ramy E Ali, Jinhyun So, and A Salman Avestimehr. On polynomial approximations for privacy- preserving and verifiable relu networks. arXiv preprint arXiv:201 2020.
Nils Bjorck, Carla P Gomes, Bart Selman, and Kilian Q Weinberger. Understanding batch normalization. Advances in neural information processing systems, 31, 2018.
Florian Bourse, Michele Minelli, Matthias Minihold, and Pascal Paillier. Fast homomorphic evaluation of deep discretized neural networks. In Annual International Cryptology Conference, pp. 482-512. Springer, 2018.
Leo Breiman. Bagging predictors. Machine learning, 24(2):123-140, 1996.
Leo Breiman. Random forests. Machine learming, 45(1):5-32, 2001.
Gavin Brown, Jeremy L Wyatt, Peter Tino, and Yoshua Bengio. Managing diversity in regression ensembles. Journal of machine learning research, 6(9), 2005.
Peter Bühlmann and Bin Yu. Analyzing bagging. The annals Statistics, 30(4):927-9611 2002.
John Bullinaria. Bias and variance, under-fiting and over-fitting Neural Computation: Lecture, 9:16, 2015.
Tom Bylander and Lisa Tate. Using validation sets avoid overfitting in adaboost. In Flairs conference, pp. $44-549. 2006.
Hervé Chabanne, Amaury De Wargny, Jonathan Milgram, Constance Morel, and Emmanuel Prouff. Privacy-preserving classification on deep neural network. Cryptology ePrint Archive, 2017.
Jonathan Cook and Vikram Ramadas. When consult precision-recall eurves. The Stata Journal, 13114433 2020.
Thomas G Dietterich. Ensemble methods in machine learning. In International workshop multiple classifier systems, pp. 1-15. Springer, 2000.
Pedro Domingos. A unified bias-nariance decomposition. In Proceedings of 17th international conference on machine learning. pp. 231-238. Morgan Kaufmann Stanford, 2000.
Xibin Dong, Zhiwen Yu, Wenming Cao, Yifan Shi, and Qianli Ma. A survey ensemble learning. Frontiers of Computer Science, 14(2):241-258, 2020.
Yoav Freund. Boosting weak learning algorithm by majority. Information and computation, 121 (2):256-285, 1995.
Stuart Geman, Elie Bienenstock, and René Doursat. Neural networks and the bias/variance dilemma. Neural computation, 4(1):1-58, 1992.
Ran Mllad-Bachrach. Nathan Dowlin, Kim Laine, Kristin Lauter, Michael Nachrig, and John Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In International conference on machine learning, pp. 201-210. PMLR, 2016.
Faliang Huang, Guoqing Xie, and Ruliang Xiao. Research on ensemble learning. In 2009 International Conference on Artificial Intelligence and Computational Intelligence, volume 3, pp. 249-252, 2009. doi: 10.1 109/AIC1.2009.233
10

---

Under review as a conference paper at ICLR 2023
Xiaoqian Jiang, Miran Kim, Kristin Lauter, and Yongsoo Song. Secure outsourced matrix computation and application to neural networks. In Proceedings of the 2018 ACM SIGSAC conference on computer and communications security, pp. 12009-122,, 2018.
Ron Kohavi, David H Wolpert, al. Bias plus variance decomposition for zero-one loss functions. In ICML, volume 96, pp. 275-83, 1996.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. The cifar-10 dataset. online: http://www. toronto. edu/kriz/cifar. haml, 55(5), 2014.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Communications the ACM, 60(6):84-90, 2017.
Yann LeCun. The mnist database of handwritten digits. http://yann. lecun. com/exdb/mmist// 1998.
John Loughrey and Pádraig Cunningham. Using early-stopping to avoid overfitting in wrapper- based feature selection employing stochastic search. Technical report, Trinity College Dublin, Department of Computer Science, 2005.
Srinath Obla. Effective activation functions for homomorphic evaluation deep neural networks. 2020.
Rising Odegua. An empirical study of ensemble techniques (bagging boosting and stacking). In Proc. Conf.: Deep Learn. IndabaXAr, 2019.
George Onoufriou, Paul Mayfield, and Georgios Leontidis. Fully homomorphically encrypted deep learning as service. Machine Learning and Knowledge Extraction, 3(4):819 9344,,43477 2021.
Joseph Rocca. Ensemble methods: bagging, boosting and stacking. Towards Data Science, 5, 2019.
Jason W Rocks and Pankaj Mehta. Memorizing without overfitting: Bias, variance, and interpolation in overparameterized models. Physical Review Research, 4(1):013201, 2022.
Omer Sagi and Lior Rokach. Ensemble learning: survey. Wiley Intendisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1249 2018.
S. S.loffeanC.. Szegedy. Bacchemmaliaaionnccceeeziiggggee training by reducing internal covariate shift. 2015.
Marina Sokolova, Nathalie Japkowicz, and Stan Szpakowicz. Beyond accuracy, f-score and roc: a family of discriminant measures for performance evaluation. In Australasian joint conference artificial intelligence, pp. 1015-1021. Springer, 2006.
Andreas Veit, Michael J Wilber, and Serge Belongie. Residual networks behave like ensembles of relatively shallow networks. Advances neural information processing systems, 29, 2016.
Geoffrey Webb and Zijian Zheng. Multistrategy ensemble learning: Reducing error by combining ensemble learning techniques. IEEE Transactions Knowledge and Data Engineering, 16(8): 980-991, 2004.
David H Wolpert. Stacked generalization. Neural networks, 5(2): 1992.
Han Xiao, Kashif Rasul, and Roland Vollgraf. Fashion-mnist: a novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.077441 2017.
Reda Yacouby and Dustin Axman. Probabilistic extension of precision, recall, and score for more thorough evaluation of classification models. In Proceedings of the first workshop evaluation and comparison of NLP systems, pp. 79-91, 2020.
Zhi-H Hua Zhou, Jianxin Wu, and Wei Tang. Ensembling neural networks: many could be better than all. Artificial intelligence, (37(1-2)2239-26 2002.
11


Reviews:
[{"review_id": "uE7m0bv8ii", "reviewer": "Reviewer_pCgq", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes to study the impact of ensemble learning on neural networks with polynomial approximations of the ReLU function. The motivation of using polynomial approximations comes from the need to apply Homomorphic encryption on inference (this is a standard technique in privacy preserving ML under homomorphic encryption).", "summary_of_the_review": "The paper applies ensemble techniques with neural networks as base models, with a caveat that the activation functions of the neural networks are polynomials approximating ReLU. I don't see anything novel here -- instead it is a rather weak engineering experiment.  ", "strength_and_weaknesses": "This is a strange paper. It appears to propose to use the well-known ensemble learning techniques with neural networks as base models. The first question that pops to my mind is – well, isn’t an ensemble of neural networks simply a wider (or deeper) neural network? \n\nIgnoring this – the paper is full of known machine learning (see for example Section 3.3 where they describe the advantages/disadvantages of ensemble learning) but there is really nothing new. Certainly not worthy of a top-conference such as ICLR.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "qH1gH92Fb9", "reviewer": "Reviewer_kymY", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "Homomorphic encryption allows to compute over encrypted data outsourcing to cloud servers securely. Hence, HE can be applied to deep learning by scaling efficiently and cost less. Since activation like ReLU is not compatible with HE, HE can apply polynomial activation function with merely addition and multiplication. Also, the lower-degree polynomial suffers from high bias, variance and low accuracy, so this work applies ensemble methods to enhance outputs from multiple weak learners.", "summary_of_the_review": "This work is less innovative. It is merely combines ensemble and polynomial replacement of relu together, and tests on relatively simple datasets. To be honest, I am not sure if there is a significant impact in this work on HE-based ML.", "strength_and_weaknesses": "Strength:\n\n1. Separating ensemble into sequential and parallel manner is good to take advantages of both approaches.\n\n2. This work shows limitation of ensemble instead of overrating it.\n\nWeakness:\n\n1. This work uses relatively shallow NN compared to existing HE-based DL methods.\n\n2. Lack of discussion on threat/security model since this work is based on cryptographic setting.\n\n3. Datasets used in this work are quite out-dated in the ML community.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "pi6A5YjOKFK", "reviewer": "Reviewer_WXCk", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the impact of ensemble modeling on the performance of homomorphic encryption compatible neural networks.", "summary_of_the_review": "Overall, I recommend against the acceptance of this paper due to:\n1 - low novelty (trivial ensembling)\n2 - analysis on the most important aspects of HE is missing", "strength_and_weaknesses": "Weaknesses:\n1 - The main idea of using ensemble of neural networks is trivial and very common in machine learning literature. The paper doesn't provide any specific adaptation to the homomorphic encryption domain.\n2 - The discussion on the homomorphic encryption schemes is completely missing. What type of HE do you use? \n3 - How do you preform majority voting in the encrypted domain? Most of HE schemes do not support argmax operation.\n4 - For sequential ensembling, it is important to study the effect of noise accumulation in the context of homomorphic encryption. This limitations prevents the use of even single deep neural networks on homomorphically encrypted data. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "divVA0fT4VF", "reviewer": "Reviewer_Kc3a", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the effect of ensembling techniques (sequential and parallel) on neural networks in the fully homomorphic encrypted (FHE) domain. In order to evaluate the neural networks using FHE, the ReLU activation function is approximated by a degree-2 polynomial. The authors find that this leads to a significant drop in accuracy. The authors verify that applying either sequential or parallel ensembling techniques can improve the accuracy. ", "summary_of_the_review": "The finding that ensembling techniques improve accuracy of neural networks, that use a degree 2 polynomial approximation to ReLU, is not by itself worthy of publication. However, this finding could an interesting starting point for further research in the area.", "strength_and_weaknesses": "Strengths:\n- Evaluating neural networks under FHE is a timely topic worthy of research.\n- The main hypothesis studied in this paper (that ensembling techniques improve accuracy) seems to be validated by the experimental results.\n\nWeaknesses:\nThe properties of ensembling techniques that are investigated here are widely known and can be found in any machine learning textbook. The main result here is that ensembling methods can improve accuracy by reducing bias and/or variance, which is by itself not worthy of publication even if it is applied in the FHE domain. I encourage the authors to view these results are a good starting point for further investigation. Some interesting research directions from here could be:\n1. How do ensembling techniques compare with other techniques for improving the accuracy in FHE domain such as increasing the degree of the polynomial approximation to ReLU? \n2. How do these methods compare in terms of training time and inference latency? \n3. What is the effect of using the same ensembling techniques in the non-encrypted domain? Do we see the same accuracy improvement or do these techniques somehow work especially well due to the approximation error in FHE?\n4. Were these neural networks actually implemented using an FHE library? From a quick glance at the code provided, it looks not. How would the accuracy and runtime be affected? Remember that FHE schemes for floating point arithmetic like CKKS are inherently noisy and may lead to different results (beyond just the effect of replacing the ReLU with a polynomial).  \n\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject