Winning Privately: The Differentially Private Lottery Ticket Mechanism

1 INTRODUCTION . Learning while preserving the privacy of the contributing users is a priority for neural networks trained on sensitive data . Especially , when it is known that neural networks tend to “ remember ” training data instances ( such as a patient ’ s healthcare information ) ( Carlini et al. , 2018 ; Fredrikson et al. , 2015 ; Song et al. , 2017 ; Wu et al. , 2016 ) . Differential privacy ( Dwork et al. , 2006 ) has become the de facto standard for protecting an individual ’ s privacy in machine learning . Differentially private training of neural networks ensures that the model does not unduly disclose any sensitive information . The most often used approach to achieve this goal is the method of gradient perturbation , where we add controlled noise to the gradients during the training phase . Differentially Private Stochastic Gradient Descent ( DPSGD ) ( Abadi et al. , 2016 ) is the current state-of-the-art , used extensively for training privacy-preserving neural networks . DPSGD , however , falls short on the utility front , the main reason for which we discuss below . DPSGD , for a given minibatch , first computes the per-observation gradient , g ( xi ) , and then clips g ( xi ) in l2 norm , g ( xi ) /max ( 1 , ||g ( xi ) ||2/C ) ( Line 6 in Abadi et al . ( 2016 ) , Algorithm 1 ) . We can see that the norm will be large ( proportional to the number of model parameters ) , especially for a multi-layer neural network , leading to a large “ clipping impact ” on the gradient , resulting in “ smaller ” clippedgradient magnitude , easily overwhelmed by noise , required for preserving differential privacy . This leads to diminished utility , especially , for scenarios where we require tight privacy ( large noise ) . An obvious solution is to minimize the number of model parameters while maximizing the model ’ s utility . This , however , is non-trivial for differentially private neural networks , especially when we need to balance privacy and utility . Recently , it has been shown that there exist smaller sub-networks within large neural networks , which when trained in isolation provide similar utility as the large networks ( Frankle & Carbin , 2018 ; Frankle et al. , 2019 ) . The phenomenon , known as the lottery ticket hypothesis is an encouraging step towards finding small , high-utility architectures . But , directly using the lottery ticket hypothesis with differential privacy is non-trivial as we need to ensure the complete process ( from ticket selection to training the winning ticket ) is end-to-end differentially private . As a potential solution to improve the privacy-utility bottleneck in differentially private neural networks , we propose the Differentially Private Lottery Ticket Mechanism ( DPLTM ) . To ensure differential privacy in DPLTM , we use a three-tiered approach . In phase 1 , we create the lottery tickets following the principles of the lottery ticket mechanism . In phase 2 , we select the winning ticket with differential privacy , making sure that we pick the ticket with a small number of parameters and with high utility via our custom score function ( details in Section 3.3.1 ) . After differentially private selection of the winning ticket , our phase 3 trains the winning architecture with differential privacy . In summary , our main contributions in this work are as follows : 1 . We propose DPLTM , the differentially private lottery ticket mechanism . An end-to-end differentially private extension of the lottery ticket hypothesis . We design a custom score function to ensure the differentially private selection of “ best ” winners , which aids DPLTM to significantly improve the privacy-utility trade-off for differentially private neural networks . 2 . Due to the reduced noise impact in DPLTM , we show that DPLTM converges at a faster rate compared to DPSGD , leading to consumption of smaller privacy budget and better utility if early stopping is desired . 3 . We show that the winning tickets in DPLTM are “ transferable ” across datasets , domains , and model architectures . Leading to the possibility of ticket generation and selection on any public dataset , resulting in lower privacy cost . 4 . Using five real-life datasets , we show that DPLTM significantly outperforms DPSGD on all datasets , and for all privacy budgets . 2 PRELIMINARIES . We use this section to introduce differential privacy and the lottery ticket hypothesis . 2.1 DIFFERENTIAL PRIVACY . Differential privacy ( Dwork et al. , 2006 ) provides us with formal and provable privacy guarantees , with the intuition that a randomized algorithm behaves similarly on “ similar ” input datasets , formally Definition 1 . ( Differential privacy ( Dwork et al. , 2006 ) ) A randomized mechanismM : Dn → Rd preserves ( , δ ) -differentially privacy if for any pair of neighbouring databases ( x , y ∈ Dn ) such that d ( x , y ) = 1 , and for all sets S of possible outputs : Pr [ M ( x ) ∈ S ] ≤ e Pr [ M ( y ) ∈ S ] + δ Intuitively , Definition 1 states that for any pair of two neighboring datasets , x , y , differing on any one row , a randomized mechanism M ’ s outcome does not change by more than a multiplicative factor of e . Moreover , the guarantee fails with probability no larger than δ . If δ = 0 , we have pure- differential privacy . The Exponential Mechanism ( EM ) ( McSherry & Talwar , 2007 ) is a well-known tool for providing differential privacy . Defined by a rangeR , privacy parameter , and a score function u : XN ×R → R that maps a dataset to the utility scores , given a dataset D ∈ XN , the EM defines a probability distribution over R according to the utility score . In other words , the EM is more likely to output some r ∈ R with higher utility scores , formally Definition 2 . ( Exponential Mechanism ( McSherry & Talwar , 2007 ) ) The exponential mechanism M ( D , u , R ) selects and outputs an element r ∈ R with probability proportional to exp ( u ( D , r ) 2∆u ) where ∆u is the sensitivity ( ∆u = maxr∈RmaxX , X′ : ||X−X′||1≤1 |u ( X , r ) −u ( X ′ , r ) | ) of the score function . In terms of privacy guarantees , the exponential Mechanism provides - differential privacy ( McSherry & Talwar , 2007 ) 2.2 LOTTERY TICKET HYPOTHESIS . The Lottery Ticket Hypothesis was proposed by Frankle & Carbin ( 2018 ) , where interestingly , it was shown that randomly initialized neural networks contain small subnetworks , which when trained in isolation , can provide similar utility as the full network . To get the subnetworks , we train a network for I iterations , prune p % of its weights ( of smallest magnitude ) , and reset the weights of the pruned network to original initialization , to be trained again . This process ensures that for n rounds , each round prunes p1/n % of the weights . Such pruned , small subnetworks , with high utility are known as “ winning tickets ” from the lottery mechanism . 3 DIFFERENTIALLY PRIVATE LOTTERY TICKET MECHANISM . 3.1 OVERVIEW . We start by providing an overview of DPLTM . Using the input dataset , X , we generate and store multiple “ lottery-tickets ” , each with a varying number of model parameters . Then we use the exponential mechanism ( McSherry & Talwar , 2007 ) to pick a winning ticket using a custom utility function ( details follow ) , that provides us with the desired balance between the number of model parameters and the model utility . After selecting the winner , we train the winning architecture with differential privacy . Our total privacy cost , hence , is composed of two separate parts , selecting the winning ticket and training the winning ticket . We present the complete process succinctly as Algorithm 1 followed by a walk-through . Algorithm 1 Differentially Private Lottery Ticket Mechanism ( DPLTM ) Require : Dataset : X , Total privacy budget : ( , δ ) , Pruning percent : p , Number of tickets : T , Ticket training iterations : I , final model training iterations : I , Neural Network : f , Initial model parameters : θ0 , Initial mask : m , Privacy budget for ticket selection : 1 , Privacy budget for ticket training : 2 , δ , Constant for score function : ν , Minibatch size : L , Clipping factor : C , Learning rate : η Phase 1 - Generating Lottery Tickets 1 : procedure LTH ( f , m , θ0 ) 2 : Randomly initialize f ( X , m θ0 ) ; m = 1|θ0| . m is the initial mask 3 : for i ∈ T do 4 : Train f ( X , m θ0 ) for I iterations , to get f ( X , m θI ) 5 : Prune p % of parameters from θI , creating a new mask m′ . m′ [ j ] = 0 , where θI is pruned 6 : Reset the remaining parameters to their values in θ0 7 : f ( X , m′ θ0 ) is the lottery ticket 8 : Store the mask m′ , initial parameters θ0 9 : Store the proportion of remaining model parameters , c , in the lottery ticket 10 : Store the performance of the lottery ticket , a , on the test set 11 : Let m = m′ 12 : end for 13 : return C , A , M , θ0 . The collection of parameter proportions , model utilities , masks , and initial parameters , ci ∈ C , ai ∈ A , m′i ∈M ; i ∈ [ 1 , T ] 14 : end procedure Phase 2 - Selecting a Winning Ticket 15 : procedure DPWT ( C , A , 1 ) 16 : Calculate score S ( C , A ) = A ( 1− νC ) 17 : Select a winning ticket , T , with probability , P = exp ( 1S2∆ ) ∑ T exp ( 1S 2∆ ) 18 : return T . The winning ticket 19 : end procedure Phase 3 - Training the Winning Ticket 20 : procedure DPTWT ( T ) 21 : Initialize the network , f , with mask and initial values from the winning ticket 22 : for k ∈ I do 23 : Take a minibatch with sampling probability L/N . N is the total dataset size 24 : For each xi ∈ L , compute gradient gk ( xi ) = ∇θkL ( θk , xi ) 25 : ĝk = 1 L ( ∑ i gk ( xi ) /max ( 1 , ||gk ( xi ) ||2C ) +N ( 0 , σ2C2I ) ) 26 : θk+1 → θk − ηkĝk 27 : end for 28 : end procedure 3.2 DPLTM WALKTHROUGH . Next we provide phase by phase walkthrough of DPLTM . Phase 1 ( Generating lottery tickets ) : We start with generating lottery tickets required for DLPTM . At this stage , we are not yet concerned about privacy . So we use the non-private lottery ticket mechanism with our input dataset X , and generate T number of lottery tickets ( ti ; i ∈ [ 1 , · · · , T ] ) . Specifically , we use iterative pruning version of the lottery ticket mechanism , where using the pruning parameter , p , at each ticket iteration , we remove p % of model parameters with the smallest magnitude . This results in T tickets with a successively smaller number of parameters . Results from each ticket ( accuracy on the test set , Ai , i ∈ [ 1 , T ] ) are stored along with the fraction of parameters ( Ci , i ∈ [ 1 , T ] ) remaining in the model . For further use , we also store the mask m′i from each ticket and the randomly initialized initial parameters θ0 . Phase 2 ( Selecting a winning ticket with differentially privacy ) : After generating candidate tickets , we need to select a winning ticket that can be subsequently trained with differential privacy . But , picking a winner is non-trivial for two reasons , first as the tickets are generated using sensitive data , we can not directly pick a winner , that is , we need differential privacy for selecting the winning ticket , second , we need to pick a winner such that the winner has an adequate balance between the number of model parameters ( smaller the better ) and the model performance ( higher the better ) . As a solution , we use the Exponential Mechanism ( EM ) ( McSherry & Talwar , 2007 ) to pick our winner with differential privacy . And to balance the number of model parameters and the utility , we define our custom score function using the combination of the accuracy achieved by the ticket on the test set and the proportion of parameters left in the network ( more details in Sections 3.3.1 and 3.4 ) . Phase 3 ( Training the winning ticket with differential privacy ) : After we select our winning ticket with differential privacy in phase 2 , now we need to train our “ winner ” architecture so the final model is differentially private . We do so by using the method similar to the differentially private stochastic gradient descent ( DPSGD ) ( Abadi et al. , 2016 ) for the training of our winning ticket . This , in contrast to DPSGD ’ s naive implementation , now only trains a “ sub-network ” with a significantly small number of parameters . And hence provides significantly better utility , reasons for which we discussed at length in the Introduction . Our extensive empirical evaluation in Section 4 provides evidence for this claim . Next , we provide formal privacy guarantees for DPLTM . We start with introducing the EM for DPLTM with our custom score function .

Summary: This paper is a mash-up of recent work on differentially private stochastic gradient descent (DPSGD) and the lottery ticket hypothesis. Differential privacy is a paradigm for ensuring that statistical models learned from a large dataset do not disclose any particulars of individual elements of that dataset. DPSGD is a technique that applies differential privacy ideas to models trained with stochastic gradient descent, where privacy is guaranteed by clipping and adding noise to gradients. On the other hand, the lottery ticket hypothesis is a method for finding sparse sub-networks contained in larger dense neural network models that are at least as accurate as the underlying dense model. The authors propose and demonstrate that by combining DPSGD with the lottery ticket hypothesis that they can train end-to-end differentially private models that outperform the DPSGD technique.