Three Flavors of Private
Machine Learning
Nicolas Papernot*
University of Toronto & Vector Institute*presenting the work of many others
January 2021 – AAAI PPAI

@NicolasPapernot

Why is the trustworthiness of ML important? privacy

Membership inference attack
Adversary figures out whether data was in the training set from model predictions
Choquette-Choo et al. Label-Only Membership Inference Attacks2}How to define trustworthiness? A successful attempt with privacy
Randomized
Algorithm

Answer 1
Answer 2...
Answer nRandomized
Algorithm

Answer 1
Answer 2...
Answer nDifferential Privacy:
Dwork et al. Calibrating noise to sensitivity in private data analysis.

?? ?
?

A Metaphor
For Private
Learning

Slides adapted from Ulfar Erlingsson

An Individual’s Training Data

Slides adapted from Ulfar Erlingsson

An Individual’s Training Data
Each bit is flipped withprobability
25%

Slides adapted from Ulfar Erlingsson

Big Picture
Remains!

Slides adapted from Ulfar Erlingsson

How to train a model?
Initialize parameters θFor t = 1..T do
Sample batch B of training examples
Compute average loss L on batch B
Compute average gradient of loss L wrt parameters θUpdate parameters θ by a multiple of gradient averageA first flavor: How to train a model with differential privacy?
Initialize parameters θFor t = 1..T doSample batch B of training examples
Compute per-example loss L on batch B
Compute per-example gradients of loss L wrt parameters θEnsure L2 norm of gradients < C by clipping
Add Gaussian noise to average gradients (as a function of C)
Update parameters θ by a multiple of noisy gradient average
Deep Learning with Differential Privacy (CCS, 2016)
Abadi, Chu, Goodfellow, McMahan, Mironov, Talwar, Zhang

Our observation: DP-SGD leads to exploding
activations

Tempered sigmoids: a family of bounded activation
functionsscaletemperatureoffsetImproved privacy-utility tradeoffs withtempered sigmoids

MNIST

FashionMNISTAll 3D plots indicate accuracy using color (for a fixed privacy

CIFAR10

A particular case: tanh

MNIST

FashionMNISTCIFAR10

DP-SGD with tanh does not lead to exploding
activations

Improving the DP-SGD state-of-the-art withtanh

Tempered Sigmoid Activations for Deep Learning with Differential Privacy (AAAI 2021)
Nicolas Papernot, Abhradeep Thakurta, Shuang Song, Steve Chien, Úlfar Erlingsson

Slides adapted from Ulfar Erlingsson

16

Slides adapted from Ulfar Erlingsson

17

Slides adapted from Ulfar Erlingsson[arxiv:1910.13427] 18

Tension between differential privacy and fairness
Task

Utility on Long
Tailed Datasets

MIMIC-III
Mortality

Model

No Privacy

High Privacy (ε, 𝜹)Logistic Regression

0.82 ± 0.03

0.60 ± 0.04 (3.54, 10-5)

GRU-D

0.79 ± 0.03

0.53 ± 0.03 (2.65, 10-5)0.84 ± 0.00

0.49 ± 0.00 (0.84, 10-6)NIH Chest X-Ray
Finetuned DenseNetDisease
121
Prediction

Unfairness Due toOverinfluence ofMajority Subgroups

Chasing Your Long Tails: Differentially Private Prediction in Health Care Settings. (FAccT 2021)
Vinith Suriyakumar, Nicolas Papernot, Anna Goldenberg, Marzyeh Ghassemi.

A 2nd flavor: PATE aka Private Aggregation of Teacher Ensembles

Sensitive
Data

Partition 1

Teacher 1

Partition 2

Teacher 2

Partition 3

Teacher 3

...

...

Partition n

Teacher n

AggregatedTeacherPATE: Private Aggregation of Teacher Ensembles

Countvotes

Take maximum

PATE: Private Aggregation of Teacher Ensembles
If most teachers agree on the label,it does not depend on specific partitions,so the privacy cost is small.

If two classes have close vote counts,
the disagreement may reveal private information.

PATE: Private Aggregation of Teacher Ensembles

Countvotes

Add Laplaciannoise

Take maximum

PATE: Private Aggregation of Teacher Ensembles
Inaccessible by adversary

Data

Data 1

Teacher 1

Data 2Teacher 2

Data 3Teacher 3...
Data nNoisy
aggregation

Accessible by adversary

StudentQuery...
Teacher n

PATE: Private Aggregation of Teacher Ensembles (ICLR 2017)
Papernot, Abadi, Erlingsson, Goodfellow, Talwar

Public Data

Training
Prediction
Data feeding

Aligning privacy with generalization

Scalable Private Learning with PATE (Papernot*, Song* et al., ICLR 2018)

A third flavor: Confidential and Private Collaborative Learning• Few distributed participants, can use heterogeneous architectures• Evaluation shows improvements to accuracy and balanced accuracy (fairness)
CaPC Learning: Confidential and Private Collaborative Learning (ICLR 2021)
Christopher A. Choquette-Choo, Natalie Dullerud, Adam Dziedzic, Yunxiang Zhang, Somesh Jha, Nicolas Papernot, Xiao WangIs achieving trustworthy ML any different from real-world
computer security?
“Practical security balances the cost ofprotection and the risk of loss, which isthe cost of recovering from a loss times
its probability” (Butler Lampson, 2004)

Is the ML paradigm fundamentallydifferent in a way thatenables systematic approaches tosecurity and privacy?

Gradient masking

Practical Black-Box Attacks against Machine Learning.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, SomeshJha, Z.Berkay Celik, and Ananthram Swami.

vs.

Confidence masking

Label-Only Membership Inference Attacks
Christopher A. Choquette Choo, Florian
Tramer, Nicholas Carlini, Nicolas Papernot

Gradient masking

Practical Black-Box Attacks against Machine Learning.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh
Jha, Z.Berkay Celik, and Ananthram Swami.

vs.

Confidence masking

Label-Only Membership Inference Attacks
Christopher A. Choquette Choo, Florian
Tramer, Nicholas Carlini, Nicolas Papernot

Why is differential privacy in ML
successful?
• Definition of robustness to adversarial examples using simplistic
distances like L_p norms directly conflicts with generalization• Instead differential privacy encourages generalization
1. No necessary trade-off between privacy and ML objective
2. Degrades smoothly to not learning when it cannot be done
privatelyA fourth flavor of privacy?

Concrete problem: let’s say we- noticed one of our training points was poisoned- One of our users wants to delete their data
how do we patch a model once we’ve trained anddeployed it?
-> machine “unlearning”

GDPR

Consumer Privacy
ActPIPEDAIs differentially private training enough?

• Not really: differentially private training only bounds how much
we’ve learned from each training example• If we wanted to use differential privacy, we would have to set
epsilon to 0.

Why is machine unlearning difficult?
• Difficult to estimate influence of each training example onparameters and predictions• Stochasticity in• training algorithms: batch sampling, …
• learning itself: multiple minima• Training is incremental

What is machine unlearning?

Sharded Isolated Sliced Aggregated Training

Machine Unlearning. (IEEE SP 2021)
Lucas Bourtoule, Varun Chandrasekaran, Christopher Choquette-Choo, Hengrui Jia, Adelin Travers, Baiwu Zhang, David Lie, Nicolas Papernot

Resources:
cleverhans.io
github.com/cleverhans-lab/cleverhans
github.com/tensorflow/privacy

Private ML is an opportunity to make ML better

Contact information:
nicolas.papernot@utoronto.ca@NicolasPapernot?
I’m hiring at UofT & Vector:- Students and postdocs- Faculty positions at all ranks

Resources:
cleverhans.io
github.com/cleverhans-lab/cleverhans
github.com/tensorflow/privacy

Adam Dziedzic

Adelin Travers

Natalie Dullerud

Contact information:
nicolas.papernot@utoronto.ca@NicolasPapernot

Nick Jia

Ilia Shumailov

Lucas Bourtoule

Tejumade AfonjaVarun
Chandrasekaran

Mohammad Yaghini

Vinith Suriyakumar

I’m hiring at UofT & Vector:- Students and postdocs- Faculty positions at all ranks