AI and Security: Lessons, Challenges &Future Directions

Dawn SongUC BerkeleyAI and Security
EnablerAI

Security
Enabler• AI enables security applications• Security enables better AI• Integrity: produces intended/correct results (adversarial machine learning)• Confidentiality/Privacy: does not leak users’ sensitive data (secure, privacypreserving machine learning)• Preventing misuse of AI

AI and Security: AI in the presence of attacker

AI and Security: AI in the presence of attacker• Important to consider the presence of attacker• History has shown attacker always follows footsteps of new technology development (orsometimes even leads it)• The stake is even higher with AI• As AI controls more and more systems, attacker will have higher & higher incentives• As AI becomes more and more capable, the consequence of misuse by attacker will become more and more
severeAI and Security: AI in the presence of attacker• Attack AI• Cause learning system to not produce intended/correct results• Cause learning system to produce targeted outcome designed by attacker• Learn sensitive information about individuals• Need security in learning systems• Misuse AI• Misuse AI to attack other systems• Find vulnerabilities in other systems• Target attacks• Devise attacks• Need security in other systems

AI and Security: AI in the presence of attacker• Attack AI:• Cause learning system to not produce intended/correct results• Cause learning system to produce targeted outcome designed by attacker• Learn sensitive information about individuals• Need security in learning systems• Misuse AI• Misuse AI to attack other systems• Find vulnerabilities in other systems• Target attacks• Devise attacks• Need security in other systems

Deep Learning Systems Are Easily Fooled

ostrichSzegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., & Fergus, R. Intriguing properties of neural networks. ICLR 2014.

STOP Signs in Berkeley

Adversarial Examples in Physical WorldCan we generate adversarial examples in the physical world thatremain effective under different viewing conditions and viewpoints,
including viewing distances and angles?

10Adversarial Examples in Physical World
Subtle Perturbations

Evtimov, Ivan, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, and Dawn Song. “Robust
Physical-World Attacks on Machine Learning Models.” arXiv preprint arXiv:1707.08945 (2017).

11

Adversarial Examples in Physical World
Subtle Perturbations

12Adversarial Examples in Physical World
Camouflage Perturbations

13

Camouflage Perturbations

14

Adversarial Examples in Physical World
Adversarial perturbations are possible in physical world under
different viewing conditions and viewpoints, including viewing
distances and angles.
Deep loss function:

15

Adversarial Examples Prevalent in Deep Learning Systems• Most existing work on adversarial examples:• Image classification task
• Target model is known• Our investigation on adversarial examples:
Generative
ModelsDeep
Reinforcement
Learning
VisualQA/
Image-to-code

Other tasks and model classes

Blackbox
Attacks

New Attack
Methods
Provide more diversity of attacks

Weaker Threat Models(Target model is unknown)

Generative models
VAE-like models (VAE, VAE-GAN) use an intermediate latent
representation● An encoder: maps a high-dimensional input into lowerdimensional latent representation z.
● A decoder: maps the latent representation back to a highdimensional reconstruction.
●

Adversarial Examples in Generative Models●●

An example attack scenario:● Generative model used as a compression scheme

Attacker’s goal: for the decompressor to reconstruct a
different image from the one that the compressor sees.

Adversarial Examples for VAE-GAN in MNIST
Target Image

Original images

Adversarial examples

Reconstruction of original images

Reconstruction of adversarial examples

Jernej Kos, Ian Fischer, Dawn Song: Adversarial Examples for Generative Models

Adversarial Examples for VAE-GAN in SVHN
Target Image

Original images

Reconstruction of original images

Adversarial examples

Reconstruction of adversarial examples

Jernej Kos, Ian Fischer, Dawn Song: Adversarial Examples for Generative Models

Adversarial Examples for VAE-GAN in SVHN
Target Image

Original images

Adversarial examples

Reconstruction of original images

Reconstruction of adversarial examples

Jernej Kos, Ian Fischer, Dawn Song: Adversarial Examples for Generative Models

Deep Reinforcement Learning Agent (A3C) Playing Pong

Original Frames

Jernej Kos and Dawn Song: Delving into adversarial attacks on deep policies [ICLR Workshop 2017].

ScoreAdversarial Examples on A3C Agent on Pong

No. of steps
Jernej Kos and Dawn Song: Delving into adversarial attacks on deep policies [ICLR Workshop, 2017]

ScoreScore

Attacks Guided by Value Function

No. of steps
Blindly injecting adversarial perturbations
every 10 frames.

No. of steps

Injecting adversarial perturbations
guided by the value function.

Agent in Action

Original FramesWith FGSM perturbations(𝜖 = 0.005) inject inevery frameWith FGSM perturbations(𝜖 = 0.005) inject basedon value function

Jernej Kos and Dawn Song: Delving into adversarial attacks on deep policies [ICLR Workshop 2017].

Visual Q&A
Given a question and an image, predict the answer.

Studied VQA Models
Model 1: MCB ( https://arxiv.org/abs/1606.01847 )• Uses Multimodal Compact Bilinear pooling to combine the image feature andquestion embedding.

Studied VQA Models
Model 2: NMN ( https://arxiv.org/abs/1704.05526 )• A representative of neural module networks• First predicts a network layout according to the question, then predicts theanswer using the obtained network.

Question: What color is the sky? Original answer: MCB - blue, NMN - blue.
Target: gray.
Answer after attack: MCB - gray, NMN - gray.

benign image

adversarial image for MCB

adversarial image for NMN

Xiaojun Xu, Xinyun Chen, Chang Liu, Anna Rohrbach, Trevor Darell, Dawn Song:Can you fool AI with adversarial examples on a visual Turing test?

Question: Is it raining? Original answer: MCB - no, NMN - no.
Target: yes.
Answer after attack: MCB - yes, NMN - yes.

benign image

adv image for MCB

adv image for NMN

Question: What is on the ground? Original answer: MCB - sand, NMN - sand.
Target: snow.
Answer after attack: MCB - snow, NMN - snow.

benign image

adv image for MCB

adv image for NMN

Question: Where is the plane? Original answer: MCB - runway, NMN - runway.
Target: sky.
Answer after attack: MCB - sky, NMN - sky.

benign image

adv image for MCB

adv image for NMN

Question: What color is the traffic light? Original answer: MCB - green, NMN - green.
Target: red.
Answer after attack: MCB - red, NMN - red.

benign image

adv image for MCB

adv image for NMN

Question: What does the sign say? Original answer: MCB - stop, NMN - stop.
Target: one way.
Answer after attack: MCB - one way, NMN - one way.

benign image

adv image for MCB

adv image for NMN

Question: How many cats are there? Original answer: MCB - 1, NMN - 1.
Target: 2.
Answer after attack: MCB - 2, NMN - 2.

benign image

adv image for MCB

adv image for NMN

Adversarial Examples Prevalent in Deep Learning Systems• Most existing work on adversarial examples:• Image classification task
• Target model is known• Our investigation on adversarial examples:
Generative
ModelsDeep
Reinforcement
Learning
VisualQA/
Image-to-code

Other tasks and model classes

Blackbox
Attacks

New Attack
Methods
Provide more diversity of attacks

Weaker Threat Models(Target model is unknown)

A General Framework for Black-box attacks• Zero-Query Attack (Previous methods)• Random perturbation• Difference of means• Transferability-based attack• Practical Black-Box Attacks against Machine Learning [Papernot et al. 2016]• Ensemble transferability-based attack [Yanpei Liu, Xinyun Chen, Chang Liu, Dawn Song:Delving into Transferable Adversarial Examples and Black-box Attacks, ICLR 2017]• Query Based Attack (new method)• Finite difference gradient estimation• Query reduced gradient estimation• A general active query game model
The zero-query attack can be viewed as a special case for the query based attack,where the number of queries made is zeroQuery Based attacks• Finite difference gradient estimation• Given d-dimensional vector x, we can make 2d queries to estimate thegradient as below• An example of approximate FGS with finite difference

xadv = x + ✏ · sign (FDx (`f (x, y), ))• Query reduced gradient estimation• Random grouping• PCA[Bhagoji, Li, He, Song, 2017]

Similarly, we can also approximate for
logit-based loss by making 2d queries

Query Based Attacks
Finite Differences method outperforms other black-box attacks
and achieves similar attack success rate with the white-box attack

Gradient estimation method with query reduction performs approximately similar as
without query reductionBlack-box Attack on Clarifai

Original image, classified as “drug”
with a confidence of 0.99

Adversarial example, classified as“safe” with a confidence of 0.96

The Gradient-Estimation black-box attack on Clarifai’s Content Moderation Model

Adversarial Examples Prevalent in Deep Learning Systems• Most existing work on adversarial examples:• Image classification task
• Target model is known• Our investigation on adversarial examples:
Generative
ModelsDeep
Reinforcement
Learning
VisualQA/
Image-to-code

Other tasks and model classes

Blackbox
Attacks

New Attack
Methods
Provide more diversity of attacks

Weaker Threat Models(Target model is unknown)

Generating Adversarial Examples with
Adversarial Networks

Black-box can be performed
here via distillation

LGAN = Ex⇠Pdata (x) log D(x) + Ex⇠Pdata (x) log(1L = Lfadv + ↵LGAN + Lhinge[Xiao, Li, Zhu, He, Liu, Song, 2017]

D(x + G(x)))

Semi-white box attack on MNISTBlack-box attack on MNISTThe perturbed images are very close to the original ones. The original images lie on the diagonal.

The perturbed images are very close to the original ones. The original images lie on the diagonal.

Numerous Defenses Proposed• Input processing• Gaussian blur, median blur
• Quantization• Adversary re-training• Re-train with generated adversarial examples• Detecting adversarial examples• Detecting anomalous high-frequency patterns in input• Detecting anomalous activations• Detecting low confidence output

Numerous Defenses Proposed
Ensemble
Normalization
Distributional detection

Detection

PCA detection
Secondary classification
Stochastic
Generative
Training process
Architecture
Retrain
Pre-process input

Prevention

No Sufficient Defense Today• Strong, adaptive attacker can easily evade today’s defenses• Ensemble of weak defenses does not (by default) lead to strong defense• Warren He, James Wei, Xinyun Chen, Nicholas Carlini, Dawn Song [WOOT 2017]• Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methods• Nicholas Carlini and David Wagner

Adversarial Machine Learning• Adversarial machine learning:• Learning in the presence of adversaries• Inference time: adversarial example fools learning system• Evasion attacks• Evade malware detection; fraud detection• Training time:• Attacker poisons training dataset (e.g., poison labels) to fool learning system to learn wrong model• Poisoning attacks: e.g., Microsoft’s Tay twitter chatbot• Attacker selectively shows learner training data points (even with correct labels) to fool learning
system to learn wrong model• Data poisoning is particularly challenging with crowd-sourcing & insider attack• Difficult to detect when the model has been poisoned• Adversarial machine learning particularly important for security critical system

Security will be one of the biggest challenges in Deploying AI

Security of Learning Systems• Software level• Learning level• Distributed level

Challenges for Security at Software Level• No software vulnerabilities (e.g., buffer overflows & access control issues)• Attacker can take control over learning systems through exploiting software
vulnerabilities

Challenges for Security at Software Level• No software vulnerabilities (e.g., buffer overflows & access control issues)• Existing software security/formal verification techniques apply

Reactive Defense

Proactive Defense:
Bug Finding

Proactive Defense:
Secure by Construction

Automatic worm detection& signature/patch generation
Automatic malware
detection & analysis

Progression of different approaches to software security over last 20 years

Security of Learning Systems• Software level• Learning level• Distributed level

Challenges for Security at Learning Level• Evaluate system under adversarial events, not just normal events

Regression Testing vs. Security Testingin Traditional Software System
Regression Testing
Run program on normal
Operation
inputsGoal

Security Testing
Run program on
abnormal/adversarial inputs

Prevent normal users from Prevent attackers fromencountering errors
finding exploitable errors

Regression Testing vs. Security Testingin Learning System
Regression Testing
Train on noisy training data:
Training Estimate resiliency against
noisy training inputs

Security Testing
Train on poisoned training
data: Estimate resiliency
against poisoned training
inputsTest onTest on normal inputs:
abnormal/adversarial inputs:
Testing
Estimate generalization error Estimate resiliency against
adversarial inputs

Challenges for Security at Learning Level• Evaluate system under adversarial events, not just normal events• Regression testing vs. security testing• Reason about complex, non-symbolic programs

Decades of Work on Reasoning about Symbolic Programs• Symbolic programs:• E.g., OS, File system, Compiler, web application, mobile application• Semantics defined by logic• Decades of techniques & tools developed for logic/symbolic reasoning• Theorem provers, SMT solvers• Abstract interpretation

Era of Formally Verified Systems
Verified: Micro-kernel, OS, File system, Compiler, Security protocols, Distributed systems

IronClad/IronFleet
FSCQ
EasyCrypt

CertiKOS

miTLS/Everest

CompCertPowerful Formal Verification Tools + Dedicated Teams
Why3
Coq

Z3

No Sufficient Tools to Reason about Non-Symbolic Programs• Symbolic programs:• Semantics defined by logic• Decades of techniques & tools developed for logic/symbolic reasoning• Theorem provers, SMT solvers• Abstract interpretation• Non-symbolic programs:• No precisely specified properties & goals• No good understanding of how learning system works• Traditional symbolic reasoning techniques do not applyChallenges for Security at Learning Level• Evaluate system under adversarial events, not just normal events• Regression testing vs. security testing• Reason about complex, non-symbolic programs• Design new architectures & approaches with stronger generalization &security guarantees

Neural Program Synthesis
Can we teach computers to write code?
IntentExample Applications:•End-user programming•

Performance optimization of code•

Virtual assistantProgramSynthesizer

Program“Software is eating the world” --- az16
Program synthesis can automate this &democratize idea realization

Neural Program Synthesis
Training
data

123
234

452
345

612
367

Input

357

797

979

OutputNeural Program Synthesis
Training
data

123
234

452
345

612
367

Input

357

797

979

OutputNeural Program
Architecture
50
70
Test input

120

Learned neuralprogram

Test output

Neural Program Architectures
Neural Turing Machine(Graves et al)

Nov 2014

Stack Recurrent Nets (Joulin et al)

May 2015

Dec 2015

Reinforcement Learning
Neural Turing Machines(Zaremba et al)

May 2016

Learning Simple Algorithms from
Examples (Zaremba et al)June 2016

Neural Programmer (Neelankatan et al)
Neural Programmer-Interpreter (Reed et al)
Neural GPU (Kaiser et al)

Oct 2016

Differentiable NeuralComputer (Graves et al)

Neural Program Synthesis Tasks: Copy, Grade-school addition, Sorting, Shortest Path

Challenge 1: Generalization
Training
data

length = 3

123
234

452
345

612
367

Input

357

797

979

OutputNeural Program
Architecture
length = 5

34216
24320
Test input

54321

Learned neuralprogram

Test output

Challenge 2: No Proof of Generalization
length = 3

Training
data

123
234

452
345

612
367

Input

357

797

979

OutputNeural Program
Architecture✔

length = 5

34216
24320
Test input

58536

Learned neuralprogram

Test output

Our Approach: Introduce Recursion

Learn recursive neural programs

Jonathon Cai, Richard Shin, Dawn Song: Making Neural Programming Architectures Generalize via Recursion[ICLR 2017, Best Paper Award ]Recursion• Fundamental concept in Computer Science and Math
• Solve whole problem by reducing it to smaller subproblems (reduction rules)• Base cases (smallest subproblems) are easier to reason about

QuicksortOur Approach: Making Neural Programming Architectures Generalize via Recursion• Proof of Generalization:••

Recursion enables provable guarantees about neural programs
Prove perfect generalization of a learned recursive program via a verification procedure• Explicitly testing on all possible base cases and reduction rules (Verification set)

• Learn & generalize faster as well
• Trained on same data, non-recursive programs do not generalize wellAccuracy on Random Inputs for Quicksort

Jonathon Cai, Richard Shin, Dawn Song: Making Neural Programming Architectures Generalize via Recursion [ICLR 2017, Best Paper Award ]Lessons• Program architecture impacts generalization & provability• Recursive, modular neural architectures are easier to reason, prove, generalize• Explore new architectures and approaches enabling strong generalization & security
properties for broader tasks

Challenges for Security at Learning Level• Evaluate system under adversarial events, not just normal events• Reason about complex, non-symbolic programs• Design new architectures & approaches with stronger generalization &security guarantees• Reason about how to compose componentsCompositional Reasoning• Building large, complex systems require compositional reasoning• Each component provides abstraction• E.g., pre/post conditions• Hierarchical, compositional reasoning proves properties of whole system• How to do abstraction, compositional reasoning for non-symbolic programs?

Security of Learning Systems• Software level• Learning level• Evaluate system under adversarial events, not just normal events• Reason about complex, non-symbolic programs• Design new architectures & approaches with stronger generalization & security guarantees• Reason about how to compose components• Distributed level• Each agent makes local decisions; how to make good local decisions achieve good global decision?

AI and Security: AI in the presence of attacker• Attack AI• Integrity:• Cause learning system to not produce intended/correct results• Cause learning system to produce targeted outcome designed by attacker• Confidentiality:• Learn sensitive information about individuals• Need security in learning systems• Misuse AI• Misuse AI to attack other systems• Find vulnerabilities in other systems• Target attacks• Devise attacks• Need security in other systems

Misused AI can make attacks more effective

Deep Learning Empowered
Bug Finding

Deep Learning Empowered
Phishing Attacks

Misused AI for large-scale, automated, targeted manipulation

Future of AI and Security
How to better understand what security means for AI, learning systems?

How to detect when a learning system has been fooled/compromised?
How to build better resilient systems with stronger guarantees?

How to build privacy-preserving learning systems?

Security will be one of the biggest challenges in Deploying AI