Differential Privacy in Deep Learning | by Neeraj Rajkumar Parmaar | Towards Data Science

I would like to thank Mr. Akshay Kulkarni for guiding me on my journey in publishing my first-ever article.
As a large number of our day to day activities are being moved online the amount of personal and sensitive data being recorded is also on the rise. This surge in data has also led to the increased data analysis tools in the form of machine learning and deep learning that are permeating in every possible industry. These techniques are also used on sensitive user data to derive actionable insights. The objective of these models is to discover overall patterns rather than an individual’s habits.
Deep learning is evolving to become the industry standard in many of the automation procedures. But it is also infamous for learning the minute and fine details of the training dataset. This aggravates the privacy risk as the model weights now encode the finer user details which on hostile inspection could potentially reveal the user information. For example, Fredrikson et al. demonstrated a model-inversion attack that recovers images from a facial recognition system [1]. Given the abundance of data freely available it is safe to assume that a determined adversary can get hold of the necessary auxiliary information required to extract user information from the model weights.
Differential Privacy is a theory which provides us with certain mathematical guarantees of privacy of user information. It aims to reduce the impact of any one individual’s data on the overall result. This means that one would make the same inference about an individual’s data whether or not it was present in the input of the analysis. As the number of analyses increases on the data so does the risk of exposure of user information. The results of differentially private computations are immune to a wide range of privacy attacks.
This is achieved by adding carefully tuned noise (characterized by epsilon) during the computation making it difficult for hackers to identify any user. This addition of noise also leads to erosion in the accuracy of the computation. Hence there is a trade-off between the accuracy and privacy protection offered. The level of privacy is measured by epsilon and is inversely proportional to the extent of privacy protection offered. This means higher the epsilon the lesser is the degree of protection of the data and higher is the chance of revealing user information. Achieving epsilon differential privacy is an ideal case and is very difficult to achieve in a practical scenario and hence the (ε, δ)-differential privacy is used. By using (ε, δ)-differential privacy the algorithm is ε-differentially private with probability (1−δ). Hence, the closer δ is to 0, the better. Delta is usually set to the reciprocal of the number of training samples.
Mind you that we aim to safeguard the model and not the data itself from hostile inspection. This is done by adding noise during the calculation of model weights which has the additional advantage of regularization of the model. Specifically, in Deep Learning we integrate differential privacy by opting for differentially private optimizers because it is where most of the computation happens. The gradients are first calculated by taking the gradient of loss w.r.t the weights. These gradients are then clipped according to the l2_norm_clip parameter and noise is added to it which is controlled by the noise_multiplier parameter in the TensorFlow Privacy library.
We aim to preserve the privacy of the model weights by applying differential privacy in the form of a Differentially Private — Stochastic Gradient Descent (DP-SGD) optimizer. Here an attempt is made to apply differential privacy to a deep neural network VGG19 with the task of image classification on the CIFAR10 dataset to understand the impact on the model performance and privacy. If such a model were trained on sensitive and private images it would become paramount that these images are not leaked as they could be put to malicious use.
The TensorFlow Privacy library was used to implement the DP-SGD optimizer and compute epsilon. The hyperparameters are not very finely tuned to get the maximum accuracy for the benchmark as we need it for a comparative study only and excessive tuning might skew the comparison that is intended. The function that computes epsilon takes ‘number of steps’ as the input which is calculated as (epochs * number of training examples) / batch size. The number of steps is a measure of the number of times the model sees the training data. As the number of steps increases the more the model sees the training data and it incorporates the finer details into itself by overfitting which means the model has a higher chance of revealing user information.
num_classes = 10# data loadingx_train, y_train), (x_test, y_test) = cifar10.load_data()x_train = x_train.astype('float32')x_test = x_test.astype('float32')y_train = tf.keras.utils.to_categorical(y_train, num_classes)y_test = tf.keras.utils.to_categorical(y_test, num_classes)# network paramsnum_classes  = 10batch_size   = 128dropout      = 0.5weight_decay = 0.0001iterations   = len(x_train) // batch_size# dpsgd paramsdpsgd = Truelearning_rate = 0.15noise_multiplier = 1.0l2_norm_clip = 1.0epochs = 100microbatches = batch_sizeif dpsgd:   optimizer = DPGradientDescentGaussianOptimizer(   l2_norm_clip=l2_norm_clip,   noise_multiplier=noise_multiplier,   num_microbatches=microbatches,   learning_rate=learning_rate)   loss = tf.keras.losses.CategoricalCrossentropy(   reduction = tf.compat.v1.losses.Reduction.NONE)   # reduction is set to NONE to get loss in a vector form   print('DPSGD')else:   optimizer = optimizers.SGD(lr=learning_rate)   loss = tf.keras.losses.CategoricalCrossentropy()   print('Vanilla SGD')model.compile(loss=loss, optimizer=optimizer, metrics=['accuracy'])# where model is the neural network architecture you want to use# augmenting imagesdatagen = ImageDataGenerator(horizontal_flip=True,          width_shift_range=0.125,           height_shift_range=0.125,                  fill_mode='constant',cval=0.)datagen.fit(x_train)# start trainingmodel.fit(datagen.flow(x_train,y_train,batch_size=batch_size), steps_per_epoch=iterations,epochs=epochs,validation_data=(x_test, y_test))if dpsgd:   eps = compute_epsilon(epochs * len(x_train) // batch_size)   print('Delta = %f, Epsilon = %.2f'(1/(len(x_train)*1.5),eps))else:   print('Trained with vanilla non-private SGD optimizer')
The benchmark uses SGD provided by the Keras library with the same learning rate as DP-SGD. The network parameters are kept the same to make a fair comparison.
Word of caution: Since DP-SGD takes in weight decay as a parameter applying weight decay to each of the layers of the neural network shows an error.
A few observations:
The model training with DP-SGD is slower than the benchmark.
The model training with DP-SGD is noisier than the benchmark which may due to gradient clipping and addition of noise.
Eventually, the model with DP-SGD achieves a decent performance in comparison with the benchmark.
We try to simulate a real-world scenario by making an imbalanced dataset and then training the model. This is done by randomly assigning user-defined weights to each of the classes. We artificially create a severe imbalance that causes the model to underperform. To combat this imbalance we employ data level approaches instead of making changes to the model itself. The reduced number of instances causes the epsilon to increase. After oversampling with Synthetic Minority Oversampling TEchnique (SMOTE) we see an increase in accuracy. This enables us to deal with imbalance without making changes to the model itself.
Number of instances for each class (10 classes in total):[3000 3500 250 2000 1000 1500 500 50 5000 2500]
def imbalance(y, weights, x_train, y_train):    random.shuffle(weights)    indices = []    for i in range(10):       inds = np.where(y==i)[0]       random.shuffle(inds)       print(i,int(weights[i]*len(x_train)))       indices += list(inds[0:int(weights[i]*len(x_train))])       x_train = x_train[indices]       y_train = y_train[indices]    return x_train, y_trainweights = [0.01,0.05,0.1,0.2,0.3,0.4,0.5,0.6,0.7,1]x_train, y_train = imbalance(y_train,weights,x_train, y_train)# Implementing SMOTEfrom imblearn.over_sampling import SMOTEsmote = SMOTE(random_state=42)x_train = x_train.reshape(len(x_train),32*32*3) # where shape of image is (32,32,3)x_train, y_train = smote.fit_resample(x_train, y_train)x_train = x_train.reshape(len(x_train),32,32,3)
We observe that:
After oversampling the data with SMOTE, the model performance improves over the model trained over an imbalanced data set.
The model trained on the original balanced data supersedes the performance to the model trained on the oversampled data.
From experiments, we observe that data level approaches suffice to combat the problem of imbalance even with the differentially private optimizer.
The hyperparameter in the repertoire of the TensorFlow Privacy that has a direct impact on the epsilon is the noise multiplier.
Noise = 100.0, Epsilon = 0.18Noise = 10.0, Epsilon = 0.36Noise = 1.0, Epsilon = 3.44
We observe that:
Increasing the noise reduces the value of epsilon which should imply better privacy protection.
It is also known that privacy protection level is inversely proportional to model performance
But we observe no such relation in practice and that increasing the noise multiplier has almost no impact on the performance.
The departure from theory seen in tuning the noise_multiplier may be attributed to the large depth of the model where the gradient manipulation is rendered ineffective. From the code given in the TensorFlow Privacy source, we see that the value of epsilon is independent of the model training and only depends on noise_multiplier, batch_size, epochs, and delta values that we select. Epsilon has good theoretical backing for measuring the amount of risk to privacy. It takes into account all the factors that are important in determining the number of times the model sees the training data and it makes intuitive sense that the more the model sees the data the more the risk of user info exposure from model weights. But it still lacks in actually gauging how safe the model weights are to hostile inspection. This is a cause of concern because now we don’t have an idea if our model weights are immune to hacking. There is now a requirement for a metric that measures how opaque the model weights are about how little info they reveal about the users.
As discussed in the conclusion coming up with a metric to judge the privacy of the model is first and foremost. It can even be an attack model trying to derive user information from model weights. The number of training data points the model is able to identify correctly can be quantified as the risk to privacy. Once a reliable measure has been formalized, we can go about extending differentially private optimizers to more complex tasks in Computer Vision and Natural Language processing and to other architectures as well.
M. Fredrikson, S. Jha, and T. Ristenpart. “Model inversion attacks that exploit confidence information and basic countermeasures”. In CCS, pages 1322–1333. ACM, 2015.Abadi, Martin, et al. “Deep learning with differential privacy.” Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016.‘TensorFlow Privacy, https://github.com/tensorflow/privacy
M. Fredrikson, S. Jha, and T. Ristenpart. “Model inversion attacks that exploit confidence information and basic countermeasures”. In CCS, pages 1322–1333. ACM, 2015.
Abadi, Martin, et al. “Deep learning with differential privacy.” Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. 2016.
‘TensorFlow Privacy, https://github.com/tensorflow/privacy