Title: On the Gini-impurity Preservation For Privacy Random Forests
Venue: NeurIPS 2023
Accepted: yes
Keywords: classification

random forests

privacy-preserving machine learng

data encrytion
Abstract: Random forests have been one successful ensemble algorithms in machine learning. Various techniques have been utilized to preserve the privacy of random forests from anonymization, differential privacy, homomorphic encryption, etc., whereas it rarely takes into account some crucial ingredients of learning algorithm. This work presents a new encryption to preserve data's Gini impurity, which plays a crucial role during the construction of random forests. Our basic idea is to modify the structure of binary search tree to store several examples in each node,  and encrypt data features by incorporating label and order information. Theoretically, we prove that our scheme preserves the minimum Gini impurity in ciphertexts without decrypting, and present the security guarantee for encryption. For random forests, we encrypt data features based on our Gini-impurity-preserving scheme, and take the homomorphic encryption scheme CKKS to encrypt data labels  due to their importance and privacy. We  conduct extensive experiments to show the effectiveness, efficiency and security of our proposed method.

On the Gini-impurity Preservation For Privacy Random Forests

Abstract

Random forests have been one of the successful ensemble algorithms in machine learning. Various techniques have been utilized to preserve the privacy of random forests, such as anonymization, differential privacy, homomorphic encryption, etc. This work takes one step towards data encryption by incorporating some crucial ingredients of learning algorithm. Specifically, we develop a new encryption to preserve data's Gini impurity, which plays an important role during the construction of random forests. The basic idea is to modify the structure of binary search tree to store several examples in each node, and encrypt the data features by incorporating label and order information. Theoretically, our scheme is proven to preserve the minimum Gini impurity in ciphertexts without decrypting, and we also present the security guarantee for encryption. For random forests, we encrypt data features based on our Gini-impurity-preserving scheme, and take the homomorphic encryption scheme CKKS to encrypt data labels owing to their importance and privacy. We finally present extensive empirical studies to validate the effectiveness, efficiency and security of our proposed method.

1 Introduction

From the pioneer work [1], random forests have been one successful ensemble algorithm [2-4], with diverse applications such as ecology [5], computational biology [6], objection recognition [7], remote sensing [8], computer vision [9], etc. The basic idea is to construct a large number of random trees individually and make prediction based on an average of their predictions. Numerous variants of random forests have been developed to improve performance under different settings [10-22], as well as theoretical understandings on the success of random forests [21, 23-27]. The splitting criterion, such as Gini impurity and information gain, has been one of the most important ingredient during the construction of random forests [1, 28].

Various techniques have been adopted to preserve the privacy of random forests, especially for sensitive tasks such as medical diagnosis, financial predictions, and so on. For example, differential privacy [29] has been successfully applied to preserve the privacy of random forests [30, 31] and decision trees [32-34], by adding certain noise perturbations. Another relevant approach is the secure multi-party computation for random forests and decision tree [35-39], where the privacy is preserved by multi-party joint computation over respective data inputs without leakage.

Homomorphic encryption [40-43] has been one of the most important cryptosystems in privacypreserving computing [44-47]. Based on such scheme, various algorithms have been developed to train privacy random forests and decision trees [48-52], while some other methods only considered inference without training due to computational costs [53-58]. In addition, LeFevre et al. [59] took

the anonymization [60] for random forests by grouping similar attributes so as to hardly identify specific individual information.

This work takes one step towards data encryption by incorporating some crucial ingredients of learning algorithm, and main contributions can be summarized as follows:

We present a new encryption to preserve data's Gini impurity, and the basic idea is to modify the structure of binary search trees to maintain several samples on each node, and encrypt data's features by incorporating label and order information. Our scheme could change the data frequencies, which is also beneficial for data security.

Theoretically, we prove the preservation of minimum Gini impurity in ciphertexts without decryption, which plays an important role on the construction of random forests. Our scheme also satisfies the security against Gini-impurity-preserving chosen plaintext attack.

We focus on the privacy random forests in the popular client-server protocol, and take our Gini-impurity-preserving encryption for data features. We adopt homomorphic encryption CKKS to encrypt data labels. Our encrypted decision tree takes smaller communication and computational complexities, as shown in Table 1.

Extensive experiments show that our encrypted random forests take significantly better performance than prior privacy random forests via encryption, anonymization and differential privacy, and are comparable to original (plaintexts) random forests without encryption. Our encrypted random forests make a good balance between computational cost and data security.

The rest of this work is constructed as follows: Section 2 introduces relevant work. Section 3 presents an encryption on data's Gini impurity. Section 4 proposes the encrypted random forests. Section 5 conducts extensive experiments. Section 6 concludes with future work.

2 Relevant Work

Homomorphic Encryption (HE) is a cryptosystem, which allows operations on encrypted data without access to a secret key [40]. We can perform some mathematical operations such as addition and multiplication operations on encrypted data without revealing sensitive information. Given an encryption function E ( · ) and a decryption function D ( · ) , the HE scheme provides two operators ⊕ and ⊗ such that, for every pair of plaintexts x 1 and x 2 ,

where + and × denote standard addition and multiplication operations, respectively.

Various HE schemes have been developed during the past years, e.g., ElGamal [67], Paillier [68], CKKS [42] encryption, etc. Relevant techniques have been successfully applied to machine learning tasks such as regression problem [69, 70], neural network [71-75], collaborative filtering [76], etc. Generally, HE schemes are accompanied with high computational costs, and one main challenge is to maintain a good trade-off among security, effectiveness and computational cost in real applications.

Z1

L2

Ls

a(1)

a(2)

a(k+1)

a(k+k2)

a(k+..+ks-1 +1)

a(n)

plaintext

ciphertext

(C1, e1,1) (C1, e1,2)

(C2, e2,1)

(C2, e2,k2)

(Cs, es,1)

(Cs, es,n-k-.-ks-1)

Secure Multi-Party Computation (SMC) [77] is another cryptographic technique to jointly compute a function from multiple private inputs with confidential, which has been used for machine learning to protect privacy data, such as neural network [78-80], k -means clustering [81-83], random forests and decision trees [35-39], etc. Differential privacy is introduced to preserve individual privacy by taking statistically inconsequential changes to data [84], and relevant techniques have been utilized in neural network [85-87], random forests [30, 31] and decision trees [32-34].

We introduce some notations used in this work. Write [ τ ] = { 1 , 2 , · · · , τ } for integer τ ≥ 2 . Let X ⊂ R d and Y = [ τ ] denote the feature and label space, respectively. A training sample is given by S n = { ( x 1 , y 1 ) , ( x 2 , y 2 ) , ..., ( x n , y n ) } . Let | A | be the cardinality of set A , and J · K denotes the corresponding encrypted value. Let N ( µ, σ 2 ) be a normal distribution of mean µ and variance σ 2 .

3 An Encryption for Gini Impurity

This section presents the first encryption to preserve the minimum Gini impurity over encrypted data without decryption. For simplicity, we give the detailed encryption on one-dimensional feature by incorporating label information, and make similar considerations for other dimensions.

3.1 Theoretical Analysis for Gini Impurity

Let A = { ( a 1 , y 1 ) , · · · , ( a n , y n ) } be a dataset with labels y i ∈ [ τ ] , and define the Gini value as

where p y denotes the proportion of the label y . Let A l a = { ( a i , y i ): a i ≤ a, ( a i , y i ) ∈ A } and A r a = { ( a i , y i ): a i > a, ( a i , y i ) ∈ A } be the left and right subsets of A w.r.t. a splitting point a , respectively. We define the Gini impurity w.r.t. dataset A and splitting point a as

where w l = | A l a | /n and w r = | A r a | /n . Let I ∗ G ( A ) be the minimum Gini impurity of dataset A , i.e.,

The minimum Gini impurity plays a crucial role on nodes splitting during the construction of random forests. We re-sort dataset A with a non-decreasing order for a 1 , a 2 , · · · , a n as follows:

where a ⟨ 1 ⟩ ≤ a ⟨ 2 ⟩ ≤ · · · ≤ a ⟨ n ⟩ , and y ⟨ 1 ⟩ , y ⟨ 2 ⟩ , · · · , y ⟨ n ⟩ denote their corresponding labels. By incorporating label information, we partition dataset A into several datasets I 1 , I 2 , · · · , I s as follows:

Here, any two adjacent datasets have different labels, and all samples have an identical label in one dataset I j , i.e., y ⟨ i ⟩ = y ⟨ i ′ ⟩ for every ( a ⟨ i ⟩ , y ⟨ i ⟩ ) ∈ I j and ( a ⟨ i ′ ⟩ , y ⟨ i ′ ⟩ ) ∈ I j .

̸

We consider two important factors in encryption: i) preservation of the minimum Gini impurity I ∗ G ( A ) over the encrypted data, and ii) a cryptosystem for encoding and decoding data. Based on such recognition, we introduce the following encryption, for every example ( a ⟨ i ⟩ , y ⟨ i ⟩ ) ∈ I j ,

Here, c 1 , c 2 , · · · , c s are random numbers s.t. c 1 < c 2 < · · · < c s , which aim to preserve the minimum Gini impurity. We take the homomorphic encryption scheme CKKS with a public key k pub for J a ⟨ i ⟩ K 2 = Enc ( k pub , i -k 1 -··· -k j -1 ) in Eqn. (5), and it is useful for decryption. Figure 1 presents a simple illustration for our encryption, and the detailed decryption is given in Appendix A.

We now present our main theorem as follows:

Theorem 1. We have I ∗ G ( A ) = I ∗ G ( A ′ ) , for re-sort dataset A by Eqn. (3) and for the corresponding encrypted dataset A ′ = { ( J a ⟨ 1 ⟩ K 1 , y ⟨ 1 ⟩ ) , · · · , ( J a ⟨ n ⟩ K 1 , y ⟨ n ⟩ ) } from Eqns. (4) -(5) .

This theorem shows that our encryption could preserve the minimum Gini impurity over encrypted data. The detailed proof is presented in Appendix B, which involves the proof of piecewise monotonicity of I G ( A,a ) w.r.t. splitting point a , and then solves the minimum splitting point on plaintexts, as well as the corresponding point on encrypted data.

3.2 Binary Search Tree for Encryption

We now present new binary search tree to encrypt a 1 , · · · , a n dynamically, especially for un-ordered dataset A = { ( a 1 , y 1 ) , · · · , ( a n , y n ) } , or when example ( a i , y i ) arrives in a streaming data. We begin with an alternative structure for binary search tree to maintain several samples on a node from Eqns. (4)-(5), rather than previous only one sample [88, 89]. Our new structure is given by

The samples stores one or multiple samples from A , and cipher 1 and cipher 2 are the first and second ciphertext in Eqn. (5), and left and right denote left and right child of the current node, respectively.

We initialize an empty tree BT = ∅ and set its cipher 1 = c max / 2 with c max = 2 λ log 2 n , and then we construct binary search tree iteratively. We maintain an interval [ t min , t max ] in each iteration so as

̸

̸

Algorithm 2 Splitting a node for encryption

Input: Output:

̸

to keep the increasing order of ciphertexts c 1 , c 2 , · · · , c s in Eqn. (5). During the i -th iteration, we receive a sample ( a i , y i ) , and then take two steps as follows:

Step-I: Search a node for sample ( a i , y i ) in binary search tree BT

Let t be a node pointer with the initialization of the root of BT . We search a path downward in BT by comparing with a i , and the search will terminate when t is a leaf node or an empty node.

For an internal node t , the search continues to its left child and updates t max = t. cipher 1 if

̸

and the search continues to its right child and updates t min = t. cipher 1 if

̸

otherwise, the search terminates. This procedure can be easily implemented with a while loop.

It is necessary to consider two special cases after the above search. We update t = t. left if

̸

In a similar manner, we update t = t. right if

̸

Step-II: Update the binary search tree BT

After Step-I, we could find a node t for sample ( a i , y i ) and the corresponding interval [ t min , t max ] . We directly append the example ( a i , y i ) into t . samples if y i = y j for every ( a j , y j ) ∈ t. samples ; otherwise, it is necessary to split the node t according to a i .

̸

We initialize an empty node l with l. samples = { ( a j , y j ) ∈ t. samples : a j < a i } , and it is sufficient to consider l. samples = ∅ . If t. left = ∅ , then we set

̸

and update l. left = t. left , t. left = l ; otherwise, we set

and update t. left = l . Here, ξ is a random number sampled from N (0 , 1) , and notice that we may randomly sample ξ multiple times so that the condition holds in Eqns (8)-(9), respectively.

̸

̸

̸

Algorithm 3 Finding the best splitting feature and position

Input: Encrypted datasets J S t n K , available splitting feature and position J s K ȷ i =1 , and secret key k sec Output: index i ∗

%%Server:

Calculate Gini impurity I G ( J S t n K , J s K i ) from Eqn. (12) w.r.t splitting feature and position J s K i end for

Send ciphertexts { I G ( J S t n K , J s K i ) } i ∈ [ ȷ ] to the client

%%Client:

Get the decrypted

Set

i

∗

=

{

Dec

-

1

if Dec

Send

J

(

, I

sec

(

k

, I

G

(

G

S

t

n

K

S

,

t

n

s

K

J

i

∗

(

k

sec to the server

We make similar update for the right child of node t : initialize an empty node r with r. samples = { ( a j , y j ) ∈ t. samples : a j > a i } , and consider r. samples = ∅ . If t. right = ∅ , then we set

̸

; otherwise, set

̸

and update r. right = t. right , t. right = r ; otherwise, we set

and update t. right = r . Algorithm 2 presents the detailed descriptions on the splitting of node t .

Algorithm 1 presents an overview of our Gini-impurity-preserving encryption, and the decryption is given in Appendix A. Our scheme does not only keep the minimum Gini impurity, but also change frequencies to prevent decryption from frequencies, which is also beneficial for encryption [90]. Our scheme takes an average of O ( n log n ) computational complexity, since it requires O (log n ) and O (1) computational complexities to search and update a node in each iteration, respectively. Finally, the average and worst space complexities are O (log n ) and O ( n ) for our encryption, respectively.

3.3 Security Analysis

For ciphertext vector J a K = ( J a K 1 , J a K 2 ) in Eqn. (5), it suffices to discuss the first ciphertext J a K 1 , since the security of J a K 2 has been analyzed in homomorphic encryption CKKS [42]. Following semantic security against chosen plaintext attacks [89, 91], we define a security game GameGIPCPA:

An adversary chooses two sequences with distinct plaintexts { a 0 1 , · · · , a 0 n } and { a 1 1 , · · · , a 1 n } , and sends them to a challenger;

The challenger flips an unbiased coin b ∈ { 0 , 1 } to select { a b 1 , · · · , a b n } , and randomly sets their corresponding labels { y b 1 , · · · , y b n } with each y b i drawn independently and uniformly over [ τ ] . The challenger encrypts { a b 1 , · · · , a b n } by Eqns. (4) and (5), and sends the ciphertexts to the adversary;

The adversary outputs a guess of b , i.e., which sequence is selected for encryption.

We then introduce the security against Gini-impurity-preserving chosen plaintext attack as follows. Definition 2. A scheme is said to be indistinguishable under Gini-impurity-preserving chosen plaintext attack if the probability of outputs with the correct guess b is negligible for the adversary A in GameGIPCPA, that is,

The following theorem shows that our encrypted plaintexts sequences are indistinguishable.

Theorem 3. Our scheme for the first ciphertexts J a 1 K 1 , J a 2 K 1 , · · · , J a n K 1 in Section 3.2 is security against Gini-impurity-preserving chosen plaintext attack.

The detailed proof is presented in Appendix C, and the basic idea is inspired from [88]. We take induction on n to show that data point ( a b i +1 , y i +1 ) affects the constructed binary search trees with the same probability as b = 0 and b = 1 , and then the ciphertexts of data points ( a b i +1 , y i +1 ) also follow the same distribution, i.e.,

,

i

i

s

))

}

)) = 0

i

∈

[

ȷ

]

for every

J

K

J

K

i

∈

[

ȷ

]

i

∗

by Eqn. (13)

4 Encrypted Random Forests

For encrypted random forests, we follow the popular client-server protocols [51, 65, 66, 88]. A client encrypts training and testing data, and transfers encrypted data to an honest-but-curious server. The server trains random forests from the encrypted data with the aid of client, and finally returns predictions on encrypted testing data.

Encryption for training and testing datasets

Recall training data S n = { ( x 1 , y 1 ) , · · · , ( x n , y n ) } with x i = ( x i, 1 , · · · , x i,d ) . The client constructs d binary search trees BT 1 , BT 2 , · · · , BT d according to Algorithm 1 over different dimensional features and labels in S n , where BT j is used to encrypt features { x 1 ,j , · · · , x n,j } for j ∈ [ d ] .

We take the homomorphic encryption CKKS [42] to encrypt training labels y 1 , · · · , y n . Each label y i is encoded with a vector of size τ by one-hot method, and we encrypt the vector by homomorphic encryption CKKS with a public key k pub . The ciphertexts J y i K = [ J y i, 1 K , · · · , J y i,τ K ] is given by

We obtain the final training data J S n K = { ( J x 1 K , J y 1 K ) , · · · , ( J x n K , J y n K ) } .

Construction on encrypted random forests

Let ˜ S n ′ = { ˜ x 1 , · · · , ˜ x n ′ } be a testing data with instance ˜ x i = (˜ x i, 1 , · · · , ˜ x i,d ) . For every plaintext ˜ x i,j with i ∈ [ n ′ ] and j ∈ [ d ] , we search a node t in the binary search tree BT j , similarly to the node search (Step-I) in Section 3.2, and obtain its ciphertext J ˜ x i,j K = [ t. cipher , Enc ( k pub , i )] . We have the encrypted testing data J ˜ S n ′ K = { J ˜ x 1 K , · · · , J ˜ x n ′ K } .

Encrypted random forests consist of individual decision trees DT 1 , · · · , DT m , where each tree DT i is constructed as follows. We first take a bootstrap sample J S ′ n K from J S n K , and initialize DT i with one node of data J S ′ n K . We repeat the following procedure recursively for each leaf node, until the number of training samples is smaller than α , or all instances have the same label in the leaf node:

Select a k -subset B from d available features randomly without replacement;

Find the best splitting feature in B and position by Gini impurity from the encrypted data;

Split the current node into left and right children via the best splitting position and feature.

Such construction is essentially similar to original random forests [1], whereas we require a different way to find the best splitting feature and position based on Gini impurity from the encrypted data.

Let t be the current leaf node for further splitting with the encrypted training data J S t n K ⊆ J S n K , and J s K 1 , · · · , J s K ȷ denote all possible splitting features and positions in the scope of the corresponding feature subset B from J S t n K . Here, the information of feature and position can be derived from the corresponding index i ∈ [ ȷ ] and subset B .

For each i ∈ [ ȷ ] , the server partitions the current encrypted training data J S t n K into left and right subsets, i.e., J S t n K l i and J S t n K r i , according to the splitting feature and position J s K i . Let n l and n r be the number of training examples in J S t n K l i and J S t n K r i , respectively, and denote by

From Eqn. (1), we have Gini impurity

where I G ( J S t n K l i ) = 1 ⊖ p l ⊙ p l and I G ( J S t n K r i ) = 1 ⊖ p r ⊙ p r , with

Here, ⊗ , ⊙ , ⊕ and ⊖ denote the CKKS element-wise homomorphic multiplication, dot, addition and subtraction functions, respectively, as in the work of [42].

The client gets plaintexts { Dec ( k sec , I G ( J S t n K , J s K i )) } ȷ i =1 by decrypting with the secret key k sec, when the server sends ciphertexts { I G ( J S t n K , J s K i ) } ȷ i =1 . If all instances have the same label in J S t n K , then we have Dec ( k sec , I G ( J S t n K , J s K i )) = 0 for each i ∈ [ ȷ ] , and we set i ∗ = -1 ; otherwise, we set i ∗ as

The client sends index i ∗ to the server for further splitting. Algorithm 3 presents the detailed descriptions on finding the best splitting feature and position.

For encrypted decision tree, the client requires the O ( κ ) computational complexity with κ leaves nodes, since the client performs constant basic operations for each node. The server takes the O ( κ ¯ ȷτn ) computational complexity for Eqn. (12), where ¯ ȷ is an average of number of possible splitting features and positions, and τ and n are the number of labels and training examples, respectively.

Our method takes O ( h ) communication rounds of O ( κ ¯ ȷ ) communication bandwidth to train an encrypted decision tree of height h . This is because we consider the breadth-first search and aggregate all nodes in the same height and send to the client with a single message at one time.

We do not require bootstrapping for homomorphic encryption in 3-depth homomorphic multiplicative, since we independently compute the splitting feature and position for each node from Eqn. (12). This is different from previous encrypted decision trees [51, 62], which could take expensive computational complexity for bootstrapping [40, 92].

Prediction on encrypted testing dataset

After getting decision trees DT 1 , · · · , DT m , we predict label J ˜ y i K = DT 1 ( J ˜ x i K ) ⊕··· ⊕ DT m ( J ˜ x i K ) for test instance J ˜ x i K ∈ J ˜ S n ′ K . The server sends ciphertexts { J ˜ y 1 K , · · · , J ˜ y n ′ K } to the client, and the client decrypts those ciphertexts, and gets the final plaintext label by ˜ y i = arg max j ∈ [ τ ] { Dec J ˜ y i,j K } .

During such prediction process, the server requires the O ( h ) computational complexity, since we search from the root to leaf node of tree. The client takes O (1) rounds of communication and communication bandwidth to transfer the testing data and predicting ciphertext without interaction.

miniboone

Our encrypted RFs

Original RFs

AnonyRFs

DiffPrivRFs

PPD-ERTs

OOCOCC

PivotRFs

MuIPRFs

HEldpRFs

Timeout

10

running time (seconds)

105

RRR

104

20

102

10

100

cancer

breast

diabetes

german

adver

bibtex

pendigits

phish

ailerons

house

a9a

amazon

bank

adult

mnist

runwalk

covtype

datasets

5 Experiment

We conduct experiments on 20 datasets 2 as summarized in Table 2. Most datasets have been wellstudied in previous random forests. In addition to the original (plaintexts) random forests [1], we compare with six state-of-the-art privacy-preserving random forests in recent years.

AnonyRFs: random forests based on anonymization with a top-down greedy search [59];

DiffPrivRFs: random forests based on differential privacy [93];

PPD-ERTs: extremely randomized trees from distributed structured data [64];

PivotRFs: random forests based on a hybrid of threshold partially homomorphic encryption and secure multiparty computation techniques [62];

MulPRFs: random forests based on the secure multiparty computation [94];

HEldpRFs: random forests with fully homomorphic encryption and low-degree polynomial approximations [51].

For all random forests, we train 100 individual decision trees, and randomly select ⌊ √ d ⌋ candidate features during node splitting. We set α = 10 for datasets of size smaller than 20,000 for our encrypted random forests; otherwise, set α = 100 , following [95]. For multi-class datasets, we take the one-vs-all method for MulPRFs, since it is limited to binary classification. Other parameters are set according to their respective references, and more details can be found in Appendix D.

Experimental comparisons

The performance is evaluated by five trials of 5-fold cross validation, and final prediction accuracies are obtained by averaging over these 25 runs, as summarized in Table 3. It is evident that our encrypted random forests take comparable performance with original random forests [1] on plaintexts, which nicely supports our Theorem 1 on the preservation of minimum Gini impurity in the construction of random forests. Our encrypted random forests are also comparable to MulPRFs if they can obtain results within 10 6 seconds (about 11.6 days), since MulPRFs are essentially similar to original random forests, yet with different implementation of secure multi-party computation.

As can be seen from Table 3, our random forests take significantly better performance than AnonyRFs and DiffPrivRFs, since the win/tie/loss counts show that our random forests win for most times and never lose. This is because AnonyRFs combine features by anonymization, while DiffPrivRFs add perturbations to features via differential privacy, therefore, both of them cause information lost in privacy process. Our random forests also achieve better performance than PivotRFs, since PivotRFs have to limit trees' depth for random forests due to heavy computations for HE and communications for secure multi-party computation.

Our random forests also outperform PPD-ERTs and HEldpRFs if results are obtained in 10 6 seconds, since PPD-ERTs adopt completely-random splitting, rather than selecting the minimum Gini impurity, while HEldpRFs take homomorphic encryption on features and employ low-degree polynomial approximation. Those approaches have modified the structures of original random forests.

Our scheme

Differential privacyAnonymizationOrder-pres.scheme

HE scheme

Our scheme

Differential privacyAnonymizationOrder-pres.scheme

HE scheme

Breast

品

OS0 0

0.8

0.6

1.0

Mnist

品

0105100

0.8

0.6

1.0

0.4

0.4

0.2

0.2

- 0.0

1.0

0.0

1.0

PhpB0

Rank

150 100 50 0 200 1

0.8

0.4

0.2

0.0

0.6

Phish

5500 20150 100

0.2

0.0

8'0-

0.6

·0.4

Pendigits

Rank

0.8

0.4

1.0

0.6

Covtype

0L0

一

8'0-

0.6

0.4

1.0

- 0.0

0.2

0.2

0.0

1234567

Bit

Bit

4

5

Bit

Bit

Bit

1234567

Bit

123

Bit

4

I8

Bit

Bit

Running time

All experiments are performed by c++ on the Ubuntu with 256GB main memory (AMD Ryzen Threadripper 3970X). We compare the training running time of our encrypted random forests and others, and the average CPU time (in seconds) is shown in Figure 2.

As expected, original random forests take the least running time over raw datasets without privacy preservation. Our encrypted random forests take larger running time than AnonyRFs and DiffPrivRFs because they are essentially similar to original random forests, yet with some simple modifications or perturbations on features. Our encrypted random forests take better performance and higher security.

Our encrypted random forests take smaller running time than PPD-ERTs, PivotRFs, MulPRFs and HEldpRFs, in particular for large datasets or high-dimensional datasets, where no results are obtained even after running out 10 6 seconds (almost 11.6 days). Because PPD-ERTs, PivotRFs and MulPRFs require expensive communication cost for multi-parity computation, while PivotRFs and HEldpRFs take heavy computation costs on HE scheme.

Security analysis

We present security analysis for the first ciphertext J a K 1 in ciphertext vector J a K = ( J a K 1 , J a K 2 ) , and the second ciphertext J a K 2 can be ensured by HE scheme. We compare with four state-of-the-art encryptions: differential privacy [93], anonymization [59], order-preserving scheme [96] and HE scheme [42]. Here, we present results of six datasets and randomly selecting one feature, and trends are similar on other dimensions and datasets. More results can be found in Appendix D.

Figure 3 shows the comparison results, and we take the bitwise leakage matrices to measure the security as in [97]: the more red the area, the higher the security. As expected, HE scheme presents the highest security, yet with heavy computational costs, for example, no results are obtained for datasets of size exceeding 3000 even after running out 10 6 seconds. It is also observed that our scheme presents higher security than the other three schemes, since those schemes simply present perturbations, compression or preserve the entire order information regardless of learning ingredients. In comparison, our scheme could make a good balance between security and computational cost.

6 Conclusion

This work takes one step on data encryption from some crucial ingredients of learning algorithm. We present a new encryption to preserve data's Gini impurity, which plays a crucial role during the construction of random forests. For random forests, we encrypt data features based on our Gini-impurity-preserving scheme, and take the homomorphic encryption scheme CKKS to encrypt data labels. Both theoretically and empirically, we validate the effectiveness, efficiency and security of our proposed method. An interesting work is to exploit other learning ingredients, such as gini index and information gain, for data encryption in the future.

References

L. Breiman. Random forests. Machine Learning , 45(1):5-32, 2001.

G. Biau and E. Scornet. A random forest guided tour. Test , 25(2):197-227, 2016.

L. Mentch and S. Zhou. Randomization as regularization: A degrees of freedom explanation for random forest success. Journal of Machine Learning Research , 21:1-36, 2020.

M. Fernández-Delgado, E. Cernadas, S. Barro, and D. Amorim. Do we need hundreds of classifiers to solve real world classification problems? Journal of Machine Learning Research , 15(1):3133-3181, 2014.

D. Cutler, T. Edwards Jr, K. Beard, A. Cutler, K. Hess, J. Gibson, and J. Lawler. Random forests for classification in ecology. Ecology , 88(11):2783-2792, 2007.

Y. Qi. Random forest for bioinformatics. In Ensemble Machine Learning , pages 307-323. Springer, 2012.

J. Shotton, A. Fitzgibbon, M. Cook, T. Sharp, M. Finocchio, R. Moore, A. Kipman, and A. Blake. Real-time human pose recognition in parts from single depth images. Communications of the ACM , 56(1):116-124, 2013.

M. Belgiu and L. Dragut. Random forest in remote sensing: A review of applications and future directions. ISPRS Journal of Photogrammetry and Remote Sensing , 114:24-31, 2016.

A. Criminisi and J. Shotton. Decision forests for computer vision and medical image analysis . Springer Science & Business Media, 2013.

S. Basu, K. Kumbier, J. Brown, and Bin B. Yu. Iterative random forests to discover predictive and stable high-order interactions. Proceedings of the National Academy of Sciences , 115(8): 1943-1948, 2018.

M. Denil, D. Matheson, and N. Freitas. Consistency of online random forests. In Proceedings of the 30th International Conference on Machine Learning , pages 1256-1264, 2013.

G. Louppe, L. Wehenkel, A. Sutera, and P. Geurts. Understanding variable importances in forests of randomized trees. Advances in Neural Information Processing Systems 26 , 2013.

P. Geurts, D. Ernst, and L. Wehenkel. Extremely randomized trees. Machine Learning , 63(1): 3-42, 2006.

B. Lakshminarayanan, D. Roy, and Y. Teh. Mondrian forests: Efficient online random forests. Advances in Neural Information Processing Systems 27 , pages 3140-3148, 2014.

X. Li, Y. Wang, S. Basu, K. Kumbier, and B. Yu. A debiased mdi feature importance measure for random forests. Advances in Neural Information Processing Systems 32 , 2019.

Y. Lin and Y. Jeon. Random forests and adaptive nearest neighbors. Journal of the American Statistical Association , 101(474):578-590, 2006.

B. Menze, B. Kelm, D. Splitthoff, U. Koethe, and F. Hamprecht. On oblique random forests. In Proceedings of the Joint European Conference on Machine Learning and Knowledge Discovery in Databases , pages 453-469, 2011.

J. Rodriguez, L. Kuncheva, and C. Alonso. Rotation forest: A new classifier ensemble method. IEEE Transactions on Pattern Analysis and Machine Intelligence , 28(10):1619-1630, 2006.

S. Wager, T. Hastie, and B. Efron. Confidence intervals for random forests: The jackknife and the infinitesimal jackknife. Journal of Machine Learning Research , 15(1):1625-1651, 2014.

Z.-H. Zhou and J. Feng. Deep forest. National Science Review , 6(1):74-86, 2019.

W. Gao, F. Xu, and Z.-H. Zhou. Towards convergence rate analysis of random forests for classification. Artificial Intelligence , 313:103788, 2022.

J.-Q. Guo, M.-Z. Teng, W. Gao, and Z.-H. Zhou. Fast provably robust decision trees and boosting. In Proceedings of the 39th International Conference on Machine Learning , pages 8127-8144, 2022.

G. Biau, L. Devroye, and G. Lugosi. Consistency of random forests and other averaging classifiers. Journal of Machine Learning Research , 9:2015-2033, 2008.

G. Biau. Analysis of a random forests model. Journal of Machine Learning Research , 13(1): 1063-1095, 2012.

E. Scornet, G. Biau, and J. Vert. Consistency of random forests. Annals of Statistics , 43(4): 1716-1741, 2015.

J. Mourtada, S. Gaïffas, and E. Scornet. Universal consistency and minimax rates for online mondrian forests. Advances in Neural Information Processing Systems 30 , pages 3758-3767, 2017.

C. Tang, D. Garreau, and U. von Luxburg. When do random forests fail. Advances in Neural Information Processing Systems 31 , pages 2987-2997, 2018.

Z.-H. Zhou. Ensemble Methods: Foundations and Algorithms . CRC Press, 2012.

C. Dwork. Differential privacy. In Proceedings of the 33rd International Colloquium on Automata, Languages, and Programming , pages 1-12, 2006.

A. Patil and S. Singh. Differential private random forest. In Proceedings of the 3rd International Conference on Advances in Computing, Communications and Informatics , pages 2623-2630, 2014.

X. Li, B. Qin, Y.-Y. Luo, and D. Zheng. A differential privacy budget allocation algorithm based on out-of-bag estimation in random forest. Mathematics , 10(22):4338, 2022.

A. Blum, C. Dwork, F. McSherry, and K. Nissim. Practical privacy: The sulq framework. In Proceedings of the 24th ACM SIGMOD-SIGACT-SIGART Symposium on Principles of Database Systems , pages 128-138, 2005.

A. Friedman and A. Schuster. Data mining with differential privacy. In Proceedings of the 16th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 493-502, 2010.

S. Fletcher and M. Islam. Decision tree classification with differential privacy: A survey. ACM Computing Surveys , 52(4):1-33, 2019.

S. Samet and A. Miri. Privacy preserving ID3 using gini index over horizontally partitioned data. In Proceedings of the 6th International Conference on Computer Systems and Applications , pages 645-651, 2008.

J. Vaidya, C. Clifton, M. Kantarcioglu, and A. Patterson. Privacy-preserving decision trees over vertically partitioned data. ACM Transactions on Knowledge Discovery from Data , 2(3):1-27, 2008.

S. Hoogh, B. Schoenmakers, and P. Chen. Practical secure decision tree learning in a teletreatment application. In Proceedings of the 18th International Conference on Financial Cryptography and Data Security , pages 179-194, 2014.

M. Joye and F. Salehi. Private yet efficient decision tree evaluation. In Proceedings of the 32nd Annual IFIP Conference on Data and Applications Security and Privacy , pages 243-259, 2018.

Y. Li, Z. Jiang, L. Yao, X. Wang, S. Yiu, and Z. Huang. Outsourced privacy-preserving C4.5 decision tree algorithm over horizontally and vertically partitioned dataset among multiple parties. Cluster Computing , 22(1):1581-1593, 2019.

C. Gentry. Fully homomorphic encryption using ideal lattices. In Proceedings of the 41st Annual ACM Symposium on Theory of Computing , pages 169-178, 2009.

L. Ducas and D. Micciancio. Fhew: Bootstrapping homomorphic encryption in less than a second. In Proceedings of the 34th Annual International Conference on the Theory and Applications of Cryptographic Techniques , pages 617-640, 2015.

J. Cheon, A. Kim, M. Kim, and Y. Song. Homomorphic encryption for arithmetic of approximate numbers. In Proceedings of the 23rd International Conference on Theory and Application of Cryptology and Information Security , pages 409-437, 2017.

I. Chillotti, N. Gama, M. Georgieva, and M. Izabachène. Tfhe: Fast fully homomorphic encryption over the torus. Journal of Cryptology , 33(1):34-91, 2020.

R. Gilad-Bachrach, N. Dowlin, K. Laine, K. Lauter, M. Naehrig, and J. Wernsing. Cryptonets: Applying neural networks to encrypted data with high throughput and accuracy. In Proceedings of the 33rd International Conference on Machine Learning , pages 201-210, 2016.

Q. Lou, W. Lu, C. Hong, and L. Jiang. Falcon: Fast spectral inference on encrypted data. Advances in Neural Information Processing Systems 33 , pages 2364-2374, 2020.

Z. Ghodsi, N. Jha, B. Reagen, and S. Garg. Circa: Stochastic ReLUs for private deep learning. Advances in Neural Information Processing Systems 34 , pages 2241-2252, 2021.

X. Li, R. Dowsley, and M. Cock. Privacy-preserving feature selection with secure multiparty computation. In Proceedings of the 38th International Conference on Machine Learning , pages 6326-6336, 2021.

A. Aloufi, P. Hu, H. Wong, and S. Chow. Blindfolded evaluation of random forests with multi-key homomorphic encryption. IEEE Transactions on Dependable and Secure Computing , 18(4):1821-1835, 2019.

J. Li, X. Kuang, S. Lin, X. Ma, and Y. Tang. Privacy preservation for machine learning training and classification based on homomorphic encryption schemes. Information Sciences , 526: 166-179, 2020.

L. Pulido-Gaytan, A. Tchernykh, J. Cortés-Mendoza, M. Babenko, and G. Radchenko. A survey on privacy-preserving machine learning with fully homomorphic encryption. In Proceedings of the 7th Latin American Conference on High Performance Computing , pages 115-129, 2021.

A. Akavia, M. Leibovich, Y. Resheff, R. Ron, M. Shahar, and M. Vald. Privacy-preserving decision trees training and prediction. ACM Transactions on Privacy and Security , 25(3):1-30, 2022.

K. Cong, D. Das, J. Park, and H. Pereira. Sortinghat: Efficient private decision tree evaluation via homomorphic encryption and transciphering. In Proceedings of the 29th ACM SIGSAC Conference on Computer and Communications Security , pages 563-577, 2022.

J. Brickell, D. Porter, V . Shmatikov, and E. Witchel. Privacy-preserving remote diagnostics. In Proceedings of the 14th ACM SIGSAC Conference on Computer and Communications Security , pages 498-507, 2007.

M. Barni, P. Failla, V . Kolesnikov, R. Lazzeretti, A. Sadeghi, and T. Schneider. Secure evaluation of private linear branching programs with medical applications. In Proceedings of the 17th European Symposium on Research in Computer Security , pages 424-439, 2009.

R. Tai, J. Ma, Y. Zhao, and S. Chow. Privacy-preserving decision trees evaluation via linear functions. In Proceedings of the 22nd European Symposium on Research in Computer Security , pages 494-512, 2017.

A. Tueno, F. Kerschbaum, and S. Katzenbeisser. Private evaluation of decision trees using sublinear cost. Proceedings on Privacy Enhancing Technologies , 2019(1):266-286, 2019.

Á. Kiss, M. Naderpour, J. Liu, N. Asokan, and T. Schneider. Sok: Modular and efficient private decision tree evaluation. In Proceedings of Privacy Enhancing Technologies , pages 187-208, 2019.

K. Sarpatwar, N. Ratha, K. Nandakumar, K. Shanmugam, J. Rayfield, S. Pankanti, and R. Vaculín. Privacy enhanced decision tree inference. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 154-159, 2020.

K. LeFevre, D. DeWitt, and R. Ramakrishnan. Mondrian multidimensional k-anonymity. In Proceedings of the 22nd International Conference on Data Engineering , pages 25-25, 2006.

L. Sweeney. k-anonymity: A model for protecting privacy. International Journal of Uncertainty, Fuzziness and Knowledge-based Systems , 10(05):557-570, 2002.

W. Du and Z. Zhan. Building decision tree classifier on private data. In Proceedings of the 14th IEEE International Conference on Privacy, Security and Data Mining , pages 1-8, 2002.

Y. Wu, S. Cai, X. Xiao, G. Chen, and B. Ooi. Privacy preserving vertical federated learning for tree-based models. CoRR/abstract , 2008.06170, 2020.

K. Hamada, D. Ikarashi, R. Kikuchi, and K. Chida. Efficient decision tree training with new data structure for secure multi-party computation. CoRR/abstract , 2112.12906, 2021.

A. Aminifar, F. Rabbi, K. Pun, and Y. Lamo. Privacy preserving distributed extremely randomized trees. In Proceedings of the 36th Annual ACM Symposium on Applied Computing , pages 1102-1105, 2021.

M. De Cock, R. Dowsley, C. Horst, R. Katti, A. Nascimento, W. Poon, and S. Truex. Efficient and private scoring of decision trees, support vector machines and logistic regression models

based on pre-computation. IEEE Transactions on Dependable and Secure Computing , 16(2): 217-230, 2017.

A. Tueno, Y. Boev, and F. Kerschbaum. Non-interactive private decision tree evaluation. In Proceedings of the 34th Annual IFIP Conference on Data and Applications Security and Privacy , pages 174-194, 2020.

T. ElGamal. A public key cryptosystem and a signature scheme based on discrete logarithms. IEEE Transactions on Information Theory , 31:469-472, 1985.

P. Paillier. Public-key cryptosystems based on composite degree residuosity classes. In Proceedings of the 18th Annual International Conference on the Theory and Applications of Cryptographic Techniques , pages 223-238, 1999.

K. Han, S. Hong, J. Cheon, and D. Park. Logistic regression on homomorphic encrypted data at scale. In Proceedings of the 33rd AAAI Conference on Artificial Intelligence , pages 9466-9471, 2019.

P. Fenner and E. Pyzer-Knapp. Privacy-preserving Gaussian process regression - A modular approach to the application of homomorphic encryption. In Proceedings of the 34th AAAI Conference on Artificial Intelligence , pages 3866-3873, 2020.

A. Sanyal, M. Kusner, A. Gascón, and V. Kanade. TAPAS: Tricks to accelerate (encrypted) prediction as a service. In Proceedings of the 35th International Conference on Machine Learning , pages 4497-4506, 2018.

Q. Lou, B. Feng, G. Fox, and L. Jiang. Glyph: Fast and accurately training deep neural networks on encrypted data. Advances in Neural Information Processing Systems 33 , pages 9193-9202, 2020.

Q. Lou and L. Jiang. HEMET: A homomorphic-encryption-friendly privacy-preserving mobile neural network architecture. In Proceedings of the 38th International Conference on Machine Learning , pages 7102-7110, 2021.

S. Pentyala, R. Dowsley, and M. Cock. Privacy-preserving video classification with convolutional neural networks. In Proceedings of the 38th International Conference on Machine Learning , pages 8487-8499, 2021.

E. Lee, J. Lee, J. Lee, Y . Kim, Y . Kim, J. No, and W. Choi. Low-complexity deep convolutional neural networks on fully homomorphic encryption using multiplexed parallel convolutions. In Proceedings of the 39th International Conference on Machine Learning , pages 12403-12422, 2022.

J. Wang, Q. Tang, A. Arriaga, and P. Ryan. Novel collaborative filtering recommender friendly to privacy protection. In Proceedings of the 28th International Joint Conference on Artificial Intelligence , pages 4809-4815, 2019.

AC. Yao. Protocols for secure computations. In Proceedings of the 23rd Annual Symposium on Foundations of Computer Science , pages 160-164, 1982.

S. Sayyad. Privacy preserving deep learning using secure multiparty computation. In Proceedings of the 2nd International Conference on Inventive Research in Computing Applications , pages 139-142, 2020.

R. Shokri and V. Shmatikov. Privacy preserving deep learning. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security , pages 1310-1321, 2015.

K. Bonawitz, V. Ivanov, B. Kreuter, A. Marcedone, HB. McMahan, S. Patel, D. Ramage, A. Segal, and K. Seth. Practical secure aggregation for privacy-preserving machine learning. In proceedings of the 24th ACM SIGSAC Conference on Computer and Communications Security , pages 1175-1191.

J. Vaidya and C. Clifton. Privacy-preserving k-means clustering over vertically partitioned data. In Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , pages 206-215, 2003.

X. Yi and YC. Zhang. Equally contributory privacy preserving k-means clustering over vertically partitioned data. Information Systems , 38(1):97-107, 2013.

Y. Fan, J. Bai, X. Lei, W. Lin, Q. Hu, G. Wu, J. Guo, and G. Tan. PPMCK: Privacy-preserving multi-party computing for k-means clustering. Parallel and Distributed Computing , 154:54-63, 2021.

C. Dwork. Differential privacy: A survey of results. In Proceedings of the 5th International Conference on Theory and Applications of Models of Computation , pages 1-19, 2008.

M. Abadi, A. Chu, I. Goodfellow, HB. McMahan, I. Mironov, K. Talwar, and L. Zhang. Deep learning with differential privacy. In Proceedings of the 23rd ACM SIGSAC Conference on Computer and Communications Security , pages 308-318, 2016.

T. Ha, TK. Dang, TT. Dang, TA. Truong, and MT. Nguyen. Differential privacy in deep learning: An overview. In Proceedings of the 12rd International Conference on Advanced Computing and Applications , pages 97-102, 2019.

B. Ghazi, N. Golowich, R. Kumar, P. Manurangsi, and CY. Zhang. Deep learning with label differential privacy. Advances in Neural Information Processing Systems 34 , pages 2713127145, 2021.

R. Popa, F. Li, and N. Zeldovich. An ideal-security protocol for order-preserving encoding. In Proceedings of the 34th IEEE Conference on Symposium on Security and Privacy , pages 463-477, 2013.

K. Florian. Frequency hiding order preserving encryption. In Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security , pages 656-667, 2015.

M. Islam, M. Kuzu, and M. Kantarcioglu. Access pattern disclosure on searchable encryption: Ramification, attack and mitigationjq. In Proceedings of the 19th Annual Network and Distributed System Security Symposium , page 12, 2012.

S. Goldwasser and S. Micali. Probabilistic encryption how to play mental poker keeping secret all partial information. In Proceedings of the 14th Annual ACM Conference on Symposium on Theory of Computing , pages 365-377, 1982.

I. Chillotti, N. Gama, M. Georgieva, and M. Izabachene. Faster fully homomorphic encryption: Bootstrapping in less than 0.1 seconds. In Proceedings of the 22nd International Conference on the Theory and Application of Cryptology and Information Security , pages 3-33, 2016.

N. Holohan, S. Braghin, P. Mac Aonghusa, and K. Levacher. Diffprivlib: The IBM differential privacy library. CoRR/abstract , 1907.02444, 2019.

K. Marcel. MP-SPDZ: A versatile framework for multi-party computation. In Proceedings of the 27th ACM SIGSAC Conference on Computer and Communications Security , pages 1575-1590, 2020.

P. Probst, M. Wright, and A. Boulesteix. Hyperparameters and tuning strategies for random forest. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 9(3):1301, 2019.

R. Agrawal, J. Kiernan, R. Srikant, and Y. Xu. Order preserving encryption for numeric data. In Proceedings of the ACM SIGMOD International Conference on Management of Data , pages 563-574, 2004.

C. Roy, B. Ding, S. Jha, W. Liu, and J. Zhou. Strengthening order preserving encryption with differential privacy. In Proceedings of the 29th ACM SIGSAC Conference on Computer and Communications Security , pages 2519-2533, 2022.

M. Tschantz, S. Sen, and A. Datta. Sok: Differential privacy as a causal property. In Proceedings of the 41st IEEE Conference on Symposium on Security and Privacy , pages 354-371, 2020.

B. Fuller, M. Varia, A. Yerukhimovich, E. Shen, A. Hamlin, V. Gadepally, R. Shay, J. Mitchell, and R. Cunningham. Sok: Cryptographically protected database search. In Proceedings of the 38th IEEE Conference on Symposium on Security and Privacy , pages 172-191, 2017.

Algorithm 4 Decryption

̸

A Detailed Decryption for Our Encryption Method

A.1 Decryption for Our Encryption in Section 3.1

We present the decryption for ciphertext J a i K = ( J a i K 1 , J a i K 2 ) in Eqn. (5) by the following steps:

Find the partition I j according to a i 1 ;

J K · Decrypt ciphertext J a i K 2 by the CKKS secret key k sec, and get index τ = Dec ( k sec , J a i K 2 ) in partition I j .

Obtain the plaintext a i as the τ -th sample in partition I j .

A.2 Decryption for Our Encryption of Binary Search Tree in Section 3.2

We decrypt a ciphertext J a i K = ( J a i K 1 , J a i K 2 ) based on binary search tree BT (in Section 3.2) and the CKKS secret key k sec by the following two steps, and Algorithm 4 presents the details of decryption:

Let t be a node pointer with the initialization of the root of binary search tree BT . We then search a path downward in BT by comparing with J a i K 1 . The search continues to its left child if J a i K 1 < t. cipher 1 and update t = t. left ; the search continues to its right child if a i 1 > t. cipher 1 and update t = t. right until a i 1 = t. cipher 1 .

J K J K · Decrypt ciphertext J a i K 2 by the CKKS secret key k sec, and get index τ = Dec ( k sec , J a i K 2 ) in t. samples . Then we use the index τ to get the plaintext a i = t. samples [ τ ] .

A.3 Formal Definition of Our Gini-impurity-preserving Encryption

We present a formal definition of our Gini-impurity preserving encryption as follows:

S ← KeyGen ( t max ) : Generate the secret state S by initializing binary search tree BT = ∅ , and a security parameter c max, which is a random number with c max > n . We maintain an interval [ t min , t max ] in each secret state S with t min = 0 and t max = c max in the initial stage, so as to keep the order of ciphertexts c 1 , c 2 , · · · , c s in Eqn. (5). In this way, the ciphertexts are random numbers with semi-order of plaintexts, and we have different ciphertext even for the same plaintexts.

S ′ , J a i K ← Encrypt ( S, a i ) : Encrypt a i and update the secret state to S ′ as for receiving a sample ( a i , y i ) as follows:

-Search a node for sample ( a i , y i ) in binary search tree BT as shown in Algorithm 1. Let t be a node pointer with the initialization of the root of BT . We search a path downward in BT by comparing with a i . The search will terminate when t is a leaf or an empty node.

-Update the binary search tree BT . We directly append the example ( a i , y i ) into t . samples if y i = y j for every ( a j , y j ) ∈ t. samples ; otherwise, it is necessary to split the node t according to a i . Algorithm 2 presents the detailed descriptions on the splitting of node t .

-Compute ciphertext J a i K and update the state from S to S ′ . Append example ( a i , y i ) into t. samples and update t. cipher 2 = Enc ( k pub , | t. samples | ) . We then compute the ciphertext J a i K = ( t. cipher 1 , t. cipher 2 ) , and update the state from S to S ′ through our BT .

a i ← Decrypt ( S ′ , J a i K ) : Solve plaintext a i for ciphertext J a i K based on state S ′ with binary search tree BT and the CKKS secret key k sec as follows:

-Let t be a node pointer with initialing the root of binary search tree BT . We then search a path downward in binary search tree BT by comparing with J a i K 1 . The search continues to its left child if J a i K 1 < t. cipher 1 and update t = t. left ; the search continues to its right child if J a i K 1 > t. cipher 1 and update t = t. right until J a i K 1 = t. cipher 1 .

-Decrypt ciphertext J a i K 2 by CKKS secret key k sec, and get index τ = Dec ( k sec , J a i K 2 ) in t. samples . Then we use the index τ to get the plaintext a i = t. samples [ τ ] .

B Proof of Theorem 1

Lemma 4. For dataset A = { ( a 1 , y 1 ) , · · · , ( a n , y n ) } , let I 1 , I 2 , · · · , I s be the corresponding partitions as defined by Eqn. (4) . There exists a splitting point a ∗ such that I G ( A,a ∗ ) = I ∗ G ( A ) and

where I G ( A,a ∗ ) and I ∗ G ( A ) are defined by Eqns. (1) and (2) , respectively.

Proof. Without loss of generality, we assume that a 1 , a 2 , · · · , a n are distinct elements. Our goal is to solve the optimal splitting point a ∗ ∈ arg min a ∈ R { I G ( A,a ) } , and we begin with some notations used in our proof. For every label j ∈ [ τ ] , we denote by

i.e., the number of the label j in dataset A . Let a be a splitting point, which splits A into left and right datasets A l a and A r a , that is,

For any given a ∈ R and j ∈ [ τ ] , we further denote by

i.e., the number of label j in subsets A l a . This follows that

where w l = | A l a | /n , and w r = 1 -w l . In the following, we will explore the monotonicity of function I G ( A,a ) when

for i = 2 , 3 , · · · , s -1 . It is easy to observe that ν j and ν l j keep constants except for ν l j ∗ , where j ∗ denotes the label of instances in I i . It remains to discuss the variable ν l j ∗ , and we have

̸

̸

It is easy to observe that

It is sufficient to consider two cases as follows:

We consider the first case

̸

and this follows that

We have and it holds that

Combining with Eqns. (14)-(16), we have

which proves the increasing function of I G ( A,a ) .

̸

̸

̸

̸

̸

̸

We now consider the second case

̸

and this follows that

We have

which proves the decreasing function of I G ( A,a ) .

In a summary, we prove the piecewise monotonicity of I G ( A,a ) for

̸

with i = 2 , 3 , · · · , s -1 . Moreover, it is easy to observe the monotonicity of I G ( A,a ) from ν l j = 0( j = j ∗ ) when

̸

and from ν j -ν l j = 0 ( j = j ∗ ) when

It is not necessary to consider the splitting point a ∗ > max { a k : ( a k , y k ) ∈ I s } with | A r a | = 0 , as well as the splitting point a ∗ < min { a k : ( a k , y k ) ∈ I 1 } with | A l a | = 0 , i.e., without splitting dataset A . This completes the proof.

Proof of Theorem 1

According to Lemma 6, we could find an optimal splitting point a ∗ such that

It is easy to observe that, for i ∈ [ s -1]

where c i is the identical ciphertext for those elements in ∈ I i , and we complete the proof.

Based on Theorem 1, our encryption with binary search trees (Algorithm 1) can also preserve the minimum Gini impurity over encrypted data, which can be shown by the following theorem:

Theorem 5. We have I ∗ G ( A ) = I ∗ G ( ˆ A ) , for re-sort dataset A by Eqn. (3) and for the corresponding encrypted dataset ˆ A = { ( J a ⟨ 1 ⟩ K 1 , y ⟨ 1 ⟩ ) , · · · , ( J a ⟨ n ⟩ K 1 , y ⟨ n ⟩ ) } from Algorithm 1.

Proof. Our constructed binary search tree BT (Algorithm 1) maintains several samples on a node. For each node t , we have t. cipher 1 < t. right . cipher 1 and t. cipher 1 > t. left . cipher 1 . In this way, we can obtain a monotone increasing sequence I 1 , I 2 , · · · , I s by inorder traversing the built Tree BT in Algorithm 1. Each I i for j ∈ [ s ] contains several samples as follows:

where a ⟨ i ′ ⟩ < a ⟨ j ′ ⟩ for ( a ⟨ i ′ ⟩ , y ⟨ i ′ ⟩ ) ∈ I i , ( a ⟨ j ′ ⟩ , y ⟨ j ′ ⟩ ) ∈ I j and i < j .

For each I i , if there is only one identical label, i.e., y ⟨ i ⟩ = y ⟨ i ′ ⟩ for every ( a ⟨ i ⟩ , y ⟨ i ⟩ ) , ( a ⟨ i ′ ⟩ , y ⟨ i ′ ⟩ ) ∈ I j , then we have I ∗ G ( A ) = I ∗ G ( ˆ A ) from Theorem 1. On the other hand, if the values are the same for all samples in I j ( j ∈ [ s ] ), i.e., a ⟨ i ⟩ = a ⟨ i ′ ⟩ for every ( a ⟨ i ⟩ , y ⟨ i ⟩ ) , ( a ⟨ i ′ ⟩ , y ⟨ i ′ ⟩ ) ∈ I j , then this splitting value is preserved without changing the minimum Gini-impurity of random forests. Hence, we also have I ∗ G ( A ) = I ∗ G ( ˆ A ) , and this completes the proof.

C Proof of Theorem 3

Given two sequences of distinct plaintext A 0 = { a 0 1 , a 0 2 , · · · , a 0 n } and A 1 = { a 1 1 , a 1 2 , · · · , a 1 n } , their corresponding labels are randomly set as follows:

Sort A b = { a b ⟨ 1 ⟩ , a b ⟨ 2 ⟩ , · · · , a b ⟨ n ⟩ } in ascending order, i.e., a b ⟨ 1 ⟩ < a b ⟨ 2 ⟩ < · · · < a b ⟨ n ⟩ for b ∈ { 0 , 1 } .

Set the corresponding labels { y ⟨ 1 ⟩ , y ⟨ 2 ⟩ , · · · , y ⟨ n ⟩ } randomly and independently from a uniform distribution on [ τ ] .

Then, we have

Lemma 6. For a b 1 < a b 2 < · · · < a b n with b = { 0 , 1 } , we have the same Gini impurity for two sequences A 0 = { ( a 0 1 , y 1 ) , ( a 0 2 , y 2 ) , · · · , ( a 0 n , y n ) } and A 1 = { ( a 1 1 , y 1 ) , ( a 1 2 , y 2 ) , · · · , ( a 1 n , y n ) } .

Proof. Let a b i be a splitting point for b ∈ { 0 , 1 } and i ∈ [ n ] , and we split A b into left and right datasets A l,b a b i and A r,b a b i according to a b i , that is,

For j ∈ [ τ ] , denote by ν l,b j and ν r,b j the cardinalities of subsets A l,b a b i and A r,b a b i with label j , respectively. Then, the Gini impurity of dataset A b and splitting point a b is given by

i

For b ∈ { 0 , 1 } , A l, 0 a 0 i and A l, 1 a 1 i have the same labels { y 1 , y 2 , · · · , y i } , and we have

Similarly, we have ∑ j ∈ [ τ ] ( ν r, 0 j ) 2 = ∑ j ∈ [ τ ] ( ν r, 1 j ) 2 , and this completes the proof.

We can show that adversary can not distinguish the ciphertext of { ( a 0 ⟨ 1 ⟩ , y ⟨ 1 ⟩ ) , · · · , ( a 0 ⟨ n ⟩ , y ⟨ n ⟩ ) } from that of { ( a 1 ⟨ 1 ⟩ , y ⟨ 1 ⟩ ) , · · · , ( a 1 ⟨ n ⟩ , y ⟨ n ⟩ ) } in a probabilistic perspective, i.e.,

We will prove Eqn. (18) by induction on n . We first have

For n = 1 , we have J a 0 ⟨ 1 ⟩ K = c 0 max / 2 and J a 1 ⟨ 1 ⟩ K = c 1 max / 2 with c 0 max = c 1 max = 2 λ log 2 n , according to the initialization in Algorithm 1. This follows that

We assume that Eqn. (18) holds for n = i ( i > 1) , that is,

Let us consider the case n = i +1 , and we add the sample ( a b ⟨ i +1 ⟩ , y ⟨ i +1 ⟩ ) in binary search tree BT b (Algorithm 1). It is sufficient to consider two cases as follows:

If we do not need to split a node for sample ( a b ⟨ i +1 ⟩ , y ⟨ i +1 ⟩ ) in Algorithm 1, then we have J a 1 ⟨ i +1 ⟩ K = J a 0 ⟨ i +1 ⟩ K . This is because a b ⟨ 1 ⟩ < a b ⟨ 2 ⟩ < · · · < a b ⟨ i +1 ⟩ for b = 0 and b = 1 , along with the same labels { y 1 , · · · , y i +1 } . Hence, we obtain the same ciphertext for a 0 ⟨ i +1 ⟩ and a 1 ⟨ i +1 ⟩ , i.e., t 0 . cipher 1 = t 1 . cipher 1 . This follows that

and

By induction assumption in Eqn. (19), we have

If we need to split the node for ( a b ⟨ i +1 ⟩ , y ⟨ i +1 ⟩ ) in Algorithm 1, then we assume that t 0 and t 1 are the corresponding splitting nodes. We firstly initialize the empty node l b and r b , and update the ciphertext l b . cipher 1 and r b . cipher 1 by Eqns (8)-(11), respectively. Notice that the random number ξ in Eqns (8)-(11) is sampled from N (0 , 1) , and thus l 0 . cipher 1 and l 1 . cipher 1 are sampled from the same distribution. We have

Similarly, r 0 . cipher 1 and r 1 . cipher 1 are sampled from the same distribution, and we have

For the ( i +1) -th iteration in Algorithm 1, we have

and

This follows that

This completes the proof.

D Experimental Details

Experimental settings

We now present some details of compared methods in this work.

Original RFs 3 : The orignal plaintext random forests [1] implemented by sklearn;

PPD-ERTs 4 : The extremely randomized trees algorithm for learning from distributed horizontal partition data [64];

PivotRFs 5 : A private and efficient solution for tree-based models in a vertical federated learning setting [62], based on a hybrid of threshold partially homomorphic encryption and secure multiparty computation techniques;

MulPRFs 6 : The original random forest [1] with the secure multiparty computation library MP-SPDZ [94], based on the sh2 protocol to support semi-honest two-party computation;

AnonyRFs 7 : The random forests based on anonymization library Mondrian, is a top-down greedy data anonymization algorithm for relational dataset [59];

DiffPrivRFs 8 : Random forests based on differential privacy library Diffprivlib [93].

Tables 4 and 5 summarizes some hyperparameters settings in our experiments. Except for parameters 'n_estimators' and ' α ' in leaf splitting, other parameters are set according to their respective references. We set security parameter λ > 6 . 4 according to privacy-preserving requisites as in [89].

Running Time

We give the prediction time comparisons(in seconds) for different methods as shown in Figure 4. As we can see, our encrypted random forests take comparable running time with original random forests, AnonyRFs and DiffPrivRFs, since our Gini-impurity preserving encryption method only requires O ( h ) time complexity without other additional operations, where O ( h ) denotes the height of binary search tree BT .

Furthermore, our encrypted random forests show superior efficiency compared to other methods, such as MulPRFs, PPD-ERTs, PivotRFs, and HEldpRFs, with the training time obtained in 10 6 seconds (almost 11.6 days). This is because MulPRFs, PivotRFs, and PPD-ERTs require expensive communication costs for multi-parity computation, while HEldpRFs takes heavy computation costs on HE scheme. We also present the orders of magnitude improvement of training and prediction time in Table 6 and Table 7, respectively.

Security

We analyze the security across fourteen datasets by randomly selecting an attribute that share a similar trend as other dimensions. We compare our Gini-impurity-preserving scheme with other four privacy-protection methods: differential privacy [93], anonymization [59], order-preserving scheme [96] and HE scheme [42]. The results are depicted in Figure 5.

Inspired from [97], we take the bitwise leakage matrix as our metric. An initial step is to scale and discretize the feature space into integers within the range of [0 , 2 7 ] , and then we sample 200 representative samples from each dataset to evaluate the security of the feature space. The primary objective in experiment is to safeguard as many bits of the plaintexts as possible. This quantitative

OurencryptedRFs

ZZ Original RFs

AnonyRFs

DiffPrivRFs

PPD-ERTs

PivotRFs

MuIPRFs

HEldpRFs

Timeout

103

running time (seconds)

101

102

品

-000

BRRSRR

38R

HSHHHS

BRRRRRR

100

101

104

cancer

breast

diabetes

german

adver

bibtex

pendigits

phish

ailerons

house

a9a

amazon

bank

anpe

mnist

miniboone

runwalk

covtype

datasets

assessment is visualized through a color map: the x -axis represents the individual bits (1 through 7), while the y -axis indicates the rank order of the 200 sampled datasets.

The color gradient, ranging from white to red, represents the degree of security, with white correlating to minimal security and red to maximal security. The security degree was normalized within a [0 , 1] range to ensure results' consistency. For instance, a security degree of 0 with white color indicates no security, while a security degree of 1 with red color suggests the highest level of security.

As expected, the HE scheme presents the highest security, yet with heavy computational costs. For example, datasets exceeding 3000 samples yielded no results under the HE scheme, even with an extended runtime of 10 6 seconds. It is also observed that our proposed scheme demonstrated superior security efficacy compared to the other three scheme: differential privacy [93], anonymization [59] and order-preserving scheme [96]. Since those schemes rely on mere data perturbations, compressions, or order information preservation. In comparison, our scheme makes a good balance between security and computational cost.

E Proof of Bitwise Leakage

In this section, we present a comprehensive evaluation of the security properties for our Gini-impurity preserving methods, full homomorphic encryption, anonymization technique, and differential privacy

methods. The security analysis is conducted in the feature space using the bitwise leakage matrix which is proposed by [97].

We focus on a discrete and finite feature space with a fixed size as in [97]. The feature space is defined as X = [0 , 2 m -1 ] , which means that the feature size is m bits, and the space ranges from 0 to 2 m -1 . Let D be the true distribution over the feature space, and dataset S = { a 1 , . . . , a n } are sampled independently and identically from distribution D .

The adversary A possesses two types of knowledge to achieve the goal of recovering plaintexts:

Auxiliary knowledge about a distribution D ′ over the feature space X [98], which provides additional information to the adversary.

Ciphertexts J S K corresponding to S , which represents the snapshot of the encrypted data store, as described in Fuller et al. [99].

We re-sort dataset S with a non-decreasing order, i.e., S = { a ⟨ 1 ⟩ , a ⟨ 2 ⟩ , · · · , a ⟨ n ⟩ } where a ⟨ 1 ⟩ ≤ a ⟨ 2 ⟩ ≤ · · · ≤ a ⟨ n ⟩ . Let S ⟨ i ⟩ be the i -th sample in S , and S ⟨ i ⟩ [ j ] be the j -th bit of S ⟨ i ⟩ with i ∈ [ n ] and j ∈ [ m ] . Then, we denote by b ⟨ i ⟩ [ j ] the adversary's guess for S ⟨ i ⟩ [ j ] through the auxiliary knowledge distribution D ′ as follows:

for i ∈ [ n ] and j ∈ [ m ] . The adversary aims to correctly guess the plaintext S ⟨ i ⟩ [ j ] using the auxiliary knowledge D ′ . Let L be a n × m matrix with

Similarly to [97], we have

and Pr D ( S ⟨ i ⟩ [ j ] = 1 ) = 1 -Pr D ( S ⟨ i ⟩ [ j ] = 0 ) , where s [ j ] denotes the j -th bit of s with S j 0 = { s | s ∈ S and s [ j ] = 0 } . This follows that

The variable L ( i, j ) represents the probability that an adversary can accurately guess the j -th bit of the plaintext S ⟨ i ⟩ . This metric can be considered as a measure of the information security for the ciphertexts J S ⟨ i ⟩ K , in the sense that a lower value of L ( i, j ) signifies a higher degree of security. Specifically, the bitwise information security of J S ⟨ i ⟩ K can be quantified as 1L ( i, j ) , and this metric provides a precise and quantitative assessment of the encryption scheme's security properties.

Specifically, we investigate the correlation among elements of L ( i, j ) , plaintexts, ciphertexts and secret keys. We explore the impact of different encryption parameters on the structure and behavior of L ( i, j ) . Our analysis reveals that the leakage pattern of L ( i, j ) is highly dependent on the specific encryption scheme. Therefore, it is crucial to carefully design and select the appropriate encryption scheme to minimize the risk of information leakage.

We now present the analysis of bitwise leakage matrix L ( i, j ) for our encryption method as follows.

Theorem 7. For our Gini-impurity-preserving encryption and plaintexts S , we have

Proof. Our Gini-impurity-preserving encryption transfers multiple plaintexts in I i ′ ( i ′ ∈ [ k ] ) to the identical first dimension ciphertext, i.e., c i ′ , as shown in Eqn. (5). Hence, the i ′ -th ciphertext c i ′ corresponds to multiple plaintexts, and the adversary will guess the true plaintext S ⟨ i ⟩ of ciphertext

Our scheme

DifferentialprivacyAnonymizationOrder-pres.scheme

HE scheme

Our scheme

Differential privacy Anonymization Order-pres.scheme

HE scheme

Wdbc

House

Cancer

A9a

Diabetes

Amazon

German

Bank

Adver

Adult

Bibitex

Runwalk

Covtype

Phish

567

5671234567

Hit

1234567123

Hi

12345671234567

Bit

Bit

Bit

Bit

c i ′ . Since the adversary only knows that i ′ -1 ciphertexts are smaller than c i ′ and k -i ′ ciphertexts are larger than c i ′ , the adversary can guess the plaintext S ⟨ i ⟩ from

with the same probability. In this way, the probability of adversary guessing S ⟨ i ⟩ [ j ] is

Let b ⟨ i ⟩ [ j ] be the adversary's guess for S ⟨ i ⟩ [ j ] , we have

The probability for the adversary correctly identifies the j -th bit of the plaintext S ⟨ i ⟩ is

and we complete the proof from Lemma 8.

Lemma 8 (Roy et al. [97]) . Let D be the input distribution and S = { a 1 , . . . , a n } denotes the dataset with each data point sampled i.i.d. from D , then we have

and

otherwise,

where

We now provide similar analysis of bitwise leakage matrix L for ϵ -local differential privacy.

Theorem 9. For ϵ -local differential privacy, we have

where S and S ′ denotes the plaintexts and ciphertexts, respectively, and

Proof. We concern ϵ -local differential privacy by adding noise to each individual value. If the adversary attempts to infer the original plaintext S ⟨ i ⟩ , then it relies on the ϵ -differential privacy disturbed data S ′ ( i ) and the auxiliary knowledge distribution D ′ . The adversary guesses the j -th bit of the i -th plaintext S ⟨ i ⟩ [ j ] through a process of deduction as follows:

This follows that

This completes the proof.

In order to gain a deeper understanding of the security for the k -anonymous algorithm, we conduct an analysis of the bitwise leakage matrix L . This matrix represents the amount of information leakage that occurs when the original data X is compressed into m partitions K 1 , K 2 , · · · , K t by the k -anonymous algorithm as follows:

The bitwise leakage matrix L quantifies the amount of information that can be inferred about an individual from the corresponding partitions. By analyzing this matrix, we can determine the level of privacy that is maintained by the k -anonymous algorithm and identify any potential vulnerabilities that could be exploited by an adversary. Then, we give the analysis of bitwise leakage matrix L for k -anonymous algorithm as follows.

Theorem 10. For k -anonymous algorithm and the plaintexts S , we have

where S ⟨ i ⟩ [ j ] denotes the j -th bit of S ⟨ i ⟩ with S ⟨ i ⟩ ∈ K q ( q ∈ [ t ]) and

Proof. The k -anonymity is a privacy-preserving technique that aims to protect the identity of individuals in a dataset. It works by grouping together individuals with similar attributes and pooling their data in a larger group, thus making it difficult for an adversary to identify any specific individual in the group. The k -anonymity ensures that each group has at least k individuals with the same attribute values, which further enhances the security of the data.

When the original data S ⟨ i ⟩ is pooled in the group K q ( q ∈ [ t ]) , the adversary attempts to guess the j -th bit of the plaintext S ⟨ i ⟩ using the auxiliary knowledge distribution D ′ and K q . To achieve this,

the adversary guesses b ⟨ i ⟩ [ j ] as the value corresponding to the maximum probability of the j -th bit in group K q as follows:

This completes the proof.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a novel encryption mechanism for isolation forest algorithms that preserves the Gini impurity metric. The authors provide theoretical evidence demonstrating that the proposed mechanism effectively preserves the Gini impurity and withstands attacks. Additionally, the authors have conducted comprehensive experiments to evaluate the accuracy and security of the proposed mechanism.\n\nSTRENGTHS:\nThis paper presents novel and intriguing findings. The proposed encryption mechanism is innovative, and the experiments conducted seem to be extensive and thorough. Additionally, the experimental results demonstrate the promising performance of the proposed mechanism in terms of both accuracy and security.\n\nWEAKNESSES:\n1.  It appears that there are concerns regarding the solidity of the proof for the theoretical guarantee in the security analysis (Theorem 3). After reviewing the proof provided in the appendix, it is evident that the proof relies on experimental analysis utilizing different random seeds rather than a traditional mathematical proof. This raises questions about the rigorousness of the proof and its validity.\n\n2. The paper requires improvement in terms of writing quality. In addition to the listed typos and grammar mistakes below, it is advisable for the authors to provide more background knowledge to enhance the reader's understanding. Furthermore, I suggest that the authors explain the motivation behind the proposed mechanism (Equations (4) and (5)) using a more informal language to aid comprehension.\n\nI have noticed some typos and grammar mistakes in the abstract and introduction sections. However, I also observed grammar errors throughout the entire paper. \n  1. Line 1, \"algorithms\" should be ``algorithm\".\n   \n  2. Line 3, \"from anonymization\" might be \"such as anonymization\".\n\n  3. Line 4, \"it rarely takes into account\" should be changed to \"they rarely take into account\". \n\n  4. Line 8, \"encrypt data features\" should be changed to \"encrypt the data features\".\n\n  5. Line 14, \"effectiveness, efficiency and security\" should be changed to \"effectiveness, efficiency, and security\".\n\n  6. In the first sentence (Line 16), \"one successful ensemble algorithms\" should be ``one successful ensemble algorithm\".\n\n  7. Line 30, \"cryptosystems\" should be \"cryptosystem\".\n\n  8. Line 22, \"ingredients\" should be \"ingredient\".\n\n  9.  Line 61, \"Given encryption function\" should be \"Given an encryption function\". \"and decryption function\" should be \"and a decryption function\". \"HE scheme\" should be \"the HE scheme\"\n\nQUESTIONS:\nBased on the concerns regarding the unsolid theoretical guarantee and unclear writing, it seems that the current version of the paper may not be suitable for publication in NeurIPS. While the paper presents interesting results, addressing these issues is crucial for ensuring the quality and suitability of the paper for a prestigious conference like NeurIPS. It is recommended that the authors revise and improve the theoretical guarantee and clarity of the writing before considering submission to NeurIPS or any other reputable publication venue.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 1, "DATE": "2023-07-02"}

comments: SUMMARY:
This work develops a new encryption to maintain the data's Gini impurity and combines it with CKKS to perform random forests training and predicting, which outperforms other SOTAs in terms of the trade-off between latency and security level.

STRENGTHS:
1. This paper gets solid technique details and corresponding theoretical proof.
2. The author gives extensive experiment results and a wide range of datasets which shows the scalability of the proposed methods.

WEAKNESSES:
1. One weakness would be the limited application scenario. It would be better to have some variants for other applications.

QUESTIONS:
same with weakness

LIMITATIONS:
same with weakness
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-07

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work investigates the privacy-preserving random forest, and the main contributions include an interesting property-preserving encryption method that can preserve data’s Gini impurity. Based on the proposed encryption method, the authors present an innovative privacy-preserving training and prediction algorithm for random forests in a client-server protocol, which can achieve the smallest communication and computational complexities in comparison with prior works. The method is supported by theoretical results, which provide justifications to preserve data’s Gini impurity and showed that the proposed algorithm is secure against Gini-impurity-preserving chosen plaintext attack. Finally, extensive experiments on various datasets are conducted to demonstrate the efficacy, efficiency, and security of the proposed method.\n\nSTRENGTHS:\nThis paper is well-written and easy to follow. It provides novel algorithms and solid theories with encouraging empirical results. The main idea of designing new encryption algorithm for decision trees that can preserve minimum Gini impurity is interesting and convincing for me.\n\n- Novel perspective on privacy-preserving machine learning. This paper introduces a novel perspective on privacy-preserving random forest by focusing on preserving minimum Gini impurity in encryption, since Gini impurity is exactly the information that random forests require during training and inference. This sheds new insights into the design of privacy-preserving machine learning algorithms with corresponding new property-preserving encryption methods, which is worthy of systemic further study for the potential enhancement of the efficacy and accuracy of privacy computing.\n- Innovative solution with solid analysis. The idea of the proposed minimum Gini impurity preserving encryption is innovative and interesting. It uses a new binary search tree to encrypt data dynamically by incorporating labels’ information, which is a novel way to encrypt data for decision trees, and the proposed encryption method incurs little additional computational cost.  The communication complexity, security and the correctness of proposed method are also well-analysed.\n- Well-founded experiments results. The empirical study provide accuracy and running time comparisons with various relevant methods, as well as a visual analysis of security. The encouraging result shows that the proposed method can effectively achieve a better balance among accuracy, communication, running time and data security.\n\nWEAKNESSES:\nDespite many strengths, there are several points where certain details could be further improved.\n\n-\tThe paragraph about ``Gini-impurity-preserving chosen plaintext attack’’ feels somewhat abstract. As it relates to the security analysis of proposed method, additional explanation could help reduce the reading difficulty for readers. Including a more detailed explanation in appendix might make this critical point more accessible.\n-\tThe interpretation of Figure 3 and the bitwise leakage matrices is a little confusing to me. It is not very clear to me what is the intuition of such matrices. It may be beneficial to provide a brief introduction on it to help readers comprehend Figure 3 more easily.\n-\tThe introduction about homomorphic encryption and CKKS seems to be brief. More details may be provided in appendix for readers unfamiliar with homomorphic encryption.\n-\tIn Figure 6 from the appendix, the ``Differential Privacy’’ at the top is blocked by the figure of bitwise leakage matrices.\n\nQUESTIONS:\n-\tHomomorphic encryption is used to encrypt labels, and it is known that homomorphic encryption has a high computation overhead; does this impact the efficacy of the proposed method?\n-\tAs shown in Table 3, the proposed encrypted random forests perform better than the original random forests on two datasets. Can you provide explanation for the improved efficacy on these datasets?\n-\tCan the proposed Gini-impurity-preserving encryption method be extended to Gini index or information gain?\n-\tIn Algo. 1 , will the selection of $c_{max}$ impact the security of proposed method?\n\nLIMITATIONS:\nThe authors have mentioned the limitations in security of the proposed method, such as the encryption method is secure against Gini-impurity-preserving chosen plaintext attack, and the server is honest-but-curious.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-09"}

comments: SUMMARY:
This work focuses on security and privacy for machine learning, especially random forest and its splitting criterion, Gini impurity. In order to do that, this work proposes a new encryption scheme for features of data, which is against chosen plaintext attack. Moreover, this work utilizes a homomorphic encryption CKKS to encrypt labels of data for secure random forests model.

STRENGTHS:
1. In Table 1, this work shows comparisons of asymptotical analysis on training/predictive communication/computation cost and demonstrates their work has advantages over other existing approaches.
2. As shown in Table 3, this work uses a lot of datasets to compare with existing approaches to convince audience on this approach in practice.
3. BST encryption avoids computationally expensive sorting.

WEAKNESSES:
1. Instead of putting proofs in appendix. It is better to use few sentences about how to prove theorems.
2. This work is lack of a related work section to walk through existing work.

QUESTIONS:
1. In section 4, authors mention [50, 72, 76, 77]. Since they relate to this work, do authors consider to update asymptotical analysis in table 1?

LIMITATIONS:
1. Chosen plaintext attack somehow is not sufficient in the real world application. If the work can guarantee chosen ciphertext attack, it could be a stronger scheme.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work focuses on the privacy of random forests. The authors develop a new encryption to preserve the Gini impurity of data, which plays an important role on the construction of random forests. Based on this new encryption, the authors introduce an effective algorithm for privacy random forests under the client-server protocol. Theoretical analysis is presented to guarantee the correctness and security of the proposed algorithm, and extensive experiments validates the effectiveness, efficiency and security of the proposed algorithm. In particularly, the privacy random forests take significantly better performance than previous privacy random forests via encryption, anonymization and differential privacy, and are even comparable to original plaintexts random forests without encryption.\n\nSTRENGTHS:\nRandom forests have been one successful algorithm with diverse applications, and this work focuses on the privacy of random forests, which is an important problem and direction in machine learning. Previous methods on this issue take anonymization, differential privacy and homomorphic encryption, to preserve the privacy of random forests, whereas this work explore the data encryption from the crucial ingredients of learning algorithm, i.e., the Gini impurity for random forests. \n1.\tAn interesting encryption with some potential direction: This work proposes an interesting encryption to preserve the minimum Gini impurity, which has been a crucial ingredient on the construction of random forests. The authors also propose an interesting binary search tree for data encryption with O(logn) computational complexity, and it is suitable for real-time or online implementation. This work may motivate us some new direction on data encryption from some crucial ingredients of learning algorithm, and such research could make a bridge between machine learning and encryption. \n2.\tAn effective and efficient privacy-preserved random forests: Based on the encryption of Gini-impurity preservation, the authors present an effective and efficient privacy-preserved random forests under the client-server protocol. As can be seen in Table 1, the proposed random decision tree takes the smallest communication and computational complexities during the training and predictive process of random forests in comparisons with previous privacy-preserving decision trees.\n3.\tSome theoretical supports for proposed algorithm: The authors prove the preservation of minimum Gini impurity in ciphertexts without decryption, which plays an important role on the construction of random forests. The proposed scheme also satisfies the security against Gini-impurity-preserving chosen plaintext attack. \n4.\tGood empirical results: Extensive experiments show that the proposed encrypted random forests take significantly better performance than previous privacy random forests via encryption, anonymization and differential privacy, and are comparable to original (plaintexts) random forests without encryption. The authors also show a good balance between computational cost and data security for the proposed algorithm.\n\nWEAKNESSES:\n1.\tThe English presentation should be improved over the whole submission.\n2.\tThe authors could present more detailed analysis on the Gini-impurity-preserving chosen plaintext attack, with necessary background knowledge on similar security threats.\n3.\tThe authors could present the space cost of the proposed encryption algorithm, and it remains unclear whether the algorithm's memory overhead during encryption is significant, especially when dealing with large-scale datasets.\n\nQUESTIONS:\n1. Is the proposed method able to handle multiclass data? Does the number of classes impact the efficiency of the encryption method? If the number of classes is large, could this potentially lead to a sparsity problem?\n2. In Algorithm 1, could the selection of the parameter c_max be inappropriate and subsequently affect the ciphertext?\n3. I notice that a few methods presented in Table 1 were not included in the experimental comparisons. Could the authors elaborate on this?\n\nLIMITATIONS:\nThe authors recognize certain constraints of their proposed method, including the security aspects highlighted in Theorem 3, and the general applicability across varying datasets as evidenced through experiments. However, I encourage the authors to review the proposed method to identify if any limitations persist, specifically those related to computational demands and sensitivity to hyperparameters.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}