Special Topics in Mechano-Informatics II

Interpretable and Adversarial Machine
Learning
Qibin Zhao

m

 

P

Tensor Learning Tea
RIKEN AI
https://qibinzhao.github.io

Machine Learning: The Success Story
Machine Learning: The Success Story

Image classification
Reinforcement Learning

ImageNet: An ML Home Run

ILSVRC top-5 Error on ImageNet
30
25

Machine translation

AlexNet

20
15
10
5
0
2010

2011

2012

2013

2014

Human

2015

2016

But what do these results really mean?

2

2017

ML Achieves Superhuman Performance3

Evolution of ML
Big Data

Data is growing...

Computational
Resources

Open Images:
9M images
ImageNet:
1M images

MNIST:
60k images

1998

CIFAR:
60k images

2009

2017

4

MachineLearning

Machine
in PhysicalWorld
ML Learningin PhysicalWorld

Autonomous Driving

Healthcare

Malware Classification

Fraud Detection

Smart City

Biometrics Recognition
2

5

Consequencesin
Real-world
Applications
NJV̐aY\VMLYX]N[_NXLN]
3NZVYcSXQ<;SX^RN\NJVaY\VMRJ]\NJV̐aY\VMLYX]N[_NXLN]

3NZVYcSXQ<;SX^RN\NJVa$NERPIMOI$NERPIMOI

6

Privacy: Deep Leakage from Gradientslearning: model is moving‣ Federated
while private training data neverleaves local devic
training data can be‣ However,leaked by publicly shared gradients

Privacy of training data

X

θ X̂(Goodfellow 2018)(Ligeng Zhu et al., Deep Leakage from Gradients. NeurIPS 2019)e7

Building Trust between Human and AI
Explanations on how and
why AI systems make thedecisions for different
stakeholders

Training models off ofsensitive data needs
privacy preserving
safeguards

Enable equitable andunbiased decision
makingProtect AI systems
from potential risks
that may cause
physical and digital
harm

Clearly determine who is
responsible for the outputof AI system decisions

Ability to produce
consistent and reliable
outputs
8

Interpretable/ExplainableMachine LearningBlack-box AI Creates Confusions%ODFNER[$,FUHDWHVFRQIXVLRQDQGGRXEW'ER-XVYWXSYV%-
HIGMWMSRW#&YWMRIWW3[RIV;L]-EQKIXXMRKXLMWHIGMWMSR#,S[HS-ERW[IVXLMWGYWXSQIVGSQTPEMRX#'YWXSQIV7YTTSVX

4SSV(IGMWMSR,S[HS-QSRMXSVERHHIFYKXLMWQSHIP#&PEGOFS\%--8 3TIVEXMSRW,S[GER-KIXEFIXXIVHIGMWMSR#(EXE7GMIRXMWXW-WXLMWXLIFIWXQSHIPXLEXGERFIFYMPX#%VIXLIWI%-W]WXIQHIGMWMSRWJEMV#-RXIVREP%YHMX6IKYPEXSVW

10Black-box ModelßɫȴŷȋƟȝࡩ  ȝȴƌƟȋɻ ŒɫƟ ȴɠŒɧʞƟॼ5æßīǞˈ ƌȴƟɻ ʉǞƟ ££ ɠɫƟƌǩƁʉ Œ ƁŒʉࡲīǞǩƁǞ ǇƟŒʉʞɫƟɻ ȝŒʉʉƟɫࡲɫƟ ʉǞƟɫƟ ǊƟȣƟɫŒȋ ƟˇɠȋŒȣŒʉǩȴȣɻࡲࡲ‣ Internals are unknown to observe‣ Internals are known but uninterpretabler11ࠂ ࡷ ࠁ߽

Explanations in ML world
Medical Diagnosis

Credit Evaluation

Fair lending laws [ECOA, FCRA] require credit decisions to be explainable

12What is Explainable AI:KDWLV([SODLQDEOH$,"%ODFN%R[$,'DWD%ODFN%R[$,'HFLVLRQ
5HFRPPHQGDWLRQ$,SURGXFW&RQIXVLRQZLWK7RGD\¶V$,%ODFN
%R[
Ɣ
Ɣ
Ɣ
Ɣ

:K\GLG\RXGRWKDW":K\GLG\RXQRWGRWKDW"
:KHQGR\RXVXFFHHGRUIDLO"
+RZGR,FRUUHFWDQHUURU"([SODLQDEOH$,)HHGEDFN(EXE)\TPEMREFPI%-)\TPEMREFPI%-4VSHYGX&OHDU 7UDQVSDUHQW3UHGLFWLRQV'HFLVLRQ([SODQDWLRQ

Low-to-High Interpretability
Significance:
Strong impacts

Ɣ
Ɣ
Ɣ
Ɣ

Manipulability:
Controllable effects

Low Interpretability,XQGHUVWDQGZK\,XQGHUVWDQGZK\QRW,NQRZZK\\RXVXFFHHGRUIDLO,XQGHUVWDQGVR,WUXVW\RXComplexity:
Gaining insights

High Interpretability

Interpretability means that the cause
13 and effect can be determined.

What is Interpretable/Explainable ML
There is no mathematical de nition of interpretability. Two
proposed de nitions in the literature areis the degree to which a human can understand‣ Interpretability
the cause of a decision. — Tim Milleis the degree to which a human can consistently‣ Interpretability
predict the model’s result. — Been Kim

:

r

fi

fi

14

Why Explainability
Generating Explanation for the End-User

Weld, D., et al, The challenge of crafting intelligible intelligence, Communications of ACM (2018)?

15

Why Explainability?
Debug (Mis)-Prediction

 16Why Explainability: Verify the ML Model/System

17Why Explainability? Improve ML Model

18

Why Explainability: Learn New Insights

19

Why Explainability: Learn Insights in the Sciences

20

Why Interpretability: Find Bias and Fairness

21

What kind of Interpretation?
Which dimensions of the data are most
‣ Data:relevant for the task

What concept does a particular neural‣ Model:encodeExplain why a certain instance‣ Prediction:
has been classi ed as a certain class

?

fi

?

22

Model-based vs. Post Hoc Interpretability

Model-based‣ Simpler model to t the dat
predictive accuracy but‣ Lowerhigher descriptive accurac
Post hoor visualize information of‣ Analyze
a trained model‣ Unchanged predictive accuracy
De nitions, methods, and applications in interpretable machine learning (Murdoch et al. PNAS 2019)

a

y

fi

 

c

fi

23

Global vs. Local Explanations

24

Global vs. Local Interpretation
Global interpretatio

Local interpretatio

how a lamp
Understanding why this‣ Understanding‣
typically looks like
image contain a lamp

n

n

25

Taxonomy of Interpretability Methods
Model-speci c
Salienc
Maps

NN Laye
Visualization
Occlusio
Maps

XGBoosFeaturImportances

Featur
Weights

Local

GlobalInput
Gradients

LIME

ShapleValuesinterpretation for‣ Local:speci c instancinterpretation for‣ Global:model output

PartiaDependency
Plots

Model-agnostic

c: only for speci c‣ Model-speci
model class, access to modelinternal
for any models,‣ Model-agnostic:
post hoc, analyzing input andoutput without access to modelinternals

fi

e

fi

fi

 

s

fi

r

n

t

y

y

e

e

l

 

26

SODLQ"$FFXUDF\YV([SODLQDELOLW\[SODLQ"$FFXUDF\YV([SODLQDELOLW\
Accuracy vs. Explainability
1HXUDO1HW
1HXUDO1HW

*$1

&11
&11

*$1

511
511

(QVHPEOH
0HWKRG(QVHPEOH
0HWKRG;*%

5DQGRP ;*%5DQGRP)RUHVW'HFLVLRQ'HFLVLRQ
7UHH
7UHH

)RUHVW

6WDWLVWLFDO
6WDWLVWLFDO
0RGHO
0RGHO

$2*
$2*

*UDSKLFDO0RGHO*UDSKLFDO0RGHO
690
690%D\HVLDQ%D\HVLDQ%HOLHI1HW%HOLHI1HW
6/5
6/5
&5)
&5) +%1+%1
0/1
0/1
0DUNRY
0DUNRY
0RGHO
0RGHO

/LQHDU/LQHDU
0RGHO
0RGHO

27

Explaining Decision28

Sensitivity Analysis

29

Saliency via Backpropagation
Sensitivity analysis of target neuron w.r.t. input pixels

The “salient” pixels
usually light up
Deep inside convolutional networks, Simonyan, Vedaldi, Zisserman, ICLR, 201430

Saliency Map
Saliency maps provide a visual representation of the input sensitivity ofan output class
ReLU<latexit sha1_base64="4/MCg3DV+j3FoZtMWFZPwksTHTw=">AAACNnicbVDLSgMxFM34rPU16tJNsAh1U2aqoCspunFZ0T6gU4ZMmmlDMw+SO9IyzN/4EX6DW924EVyIWz/B9IHY1gOBwzn35iTHiwVXYFlvxtLyyuraem4jv7m1vbNr7u3XVZRIymo0EpFsekQxwUNWAw6CNWPJSOAJ1vD61yO/8cCk4lF4D8OYtQPSDbnPKQEtuealI5gPJez4ktDUiYkETgRO71xaHJxk2a+UDrLMkbzbA0ffB246cK0Mu2bBKllj4EViT0kBTVF1zQ+nE9EkYCFQQZRq2VYM7XSUQQXL8k6iWExon3RZS9OQBEy10/E/M3yslQ72I6lPCHis/t1ISaDUMPD0ZECgp+a9kfif10rAv2inPIwTYCGdBPmJwBDhUWm4wyWjIIaaECq5fiumPaIbA13tTIoXZLoTe76BRVIvl+zTUvn2rFC5mraTQ4foCBWRjc5RBd2gKqohih7RM3pBr8aT8W58Gl+T0SVjunOAZmB8/wBCrq37</latexit>@Sc (x)
@x x0

X
k

↵kc =
<latexit sha1_base64="+V9QTyW79ZqgEwQFUCIMT8oUMXk=">AAACc3icbVFNb9QwEHXCR8vy0RSOvVhdkIoEq6QgtRekFi4cOBTEthXr3cjxTnbd2ElkTxArKz+UY38Ed5xtJPrBSJbevDfPYz1ntZIW4/h3EN67/+DhxuajweMnT59tRdvPT23VGAFjUanKnGfcgpIljFGigvPaANeZgrOs+NTpZz/BWFmV33FVw1TzRSlzKTh6Ko0cQ/iF7ht8GbdMQY57zDY6LSjjql7ytJgJejzzrZGLJb5mbHBN+EBZbrhwSet+tHRtdPLNRdvTrOYGJVd0NRPtv+7YD120s6JNo2E8itdF74KkB0PS10kaXbJ5JRoNJQrFrZ0kcY1T190rFLQD1liouSj4AiYellyDnbp1SC195Zk5zSvjT4l0zV53OK6tXenMT2qOS3tb68j/aZMG88Opk2XdIJTialHeKIoV7RKnc2lAoFp5wIWR/q1ULLkPCP2/3NiS6S6T5HYCd8Hp/ih5N9r/+n549LFPZ5PskF2yRxJyQI7IZ3JCxkSQy2AjiILt4E+4E+6GL69Gw6D3vCA3Knz7F8Uevz0=</latexit>↵kc Ak

!

1 X @y c
Z i,j @Akij

Sanity Checks for Saliency Maps (Adebayo et al., NeurIPS 2018)
Deep Inside Convolutional Networks (Simonyan et al., ICLR 2014)31

Sanity Check-1‣ When randomizing weight, model gives random predictio‣ Does saliency map change?

Sanity Checks for Saliency Maps (Adebayo et al. NeurIPS 2018)n32

Class Activation Maps (CAM)

33

Grad-CAM

34

Extremal Perturbations[Fong et al., ICCV 2019]35

Uncertainty Map
Sensing
uncertainty
Modeling
uncertainty

What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?
(Kendall et al. NeurIPS 2017)36

Interpreting Model‣ Find prototypical example of a categor‣ Find pattern maximizing activity of a neurony37Activation Maximization
Visualize the exemplar of class (output layer) or representation (hidden layer)by optimization w.r.t. inputlmax hi,j,c (x)xmax Sc (x)<latexit sha1_base64="JwqVfyYMRxwozIXm935PQG+x6Ow=">AAACFnicbVBLSwMxGMzWV62vVY+CBItQD5bdKuix6MVjffQB7bJks2kbmmSXJCstS2/+CH+DVz17E69ePfpPTB8H2zoQmMx8w5dMEDOqtON8W5ml5ZXVtex6bmNza3vH3t2rqSiRmFRxxCLZCJAijApS1VQz0oglQTxgpB70rkd+/ZFIRSPxoAcx8TjqCNqmGGkj+fZhi6O+n/aH8N7Hhf4JPIUtZuIhgnfm6tt5p+iMAReJOyV5MEXFt39aYYQTToTGDCnVdJ1YeymSmmJGhrlWokiMcA91SNNQgThRXjr+xxAeGyWE7UiaIzQcq38TKeJKDXhgJjnSXTXvjcT/vGai25deSkWcaCLwZFE7YVBHcFQKDKkkWLOBIQhLat4KcRdJhLWpbmZLwIemE3e+gUVSKxXds2Lp9jxfvpq2kwUH4AgUgAsuQBncgAqoAgyewAt4BW/Ws/VufVifk9GMNc3sgxlYX7+QFp5F</latexit><latexit sha1_base64="5v+klG9fndcKES/fq8KGwJOo69A=">AAACEHicbVDLSsNAFJ34rPUVdSO4GSxChVKSKuiy6MZlBfuANobJdNKOnUnCzERaQvwIv8Gtrt2JW//ApX/itM3Cth64cDjnXu69x4sYlcqyvo2l5ZXVtfXcRn5za3tn19zbb8gwFpjUcchC0fKQJIwGpK6oYqQVCYK4x0jTG1yP/eYjEZKGwZ0aRcThqBdQn2KktOSahx2Ohm4yTGH/nrkJLT2UcFocnkLXLFhlawK4SOyMFECGmmv+dLohjjkJFGZIyrZtRcpJkFAUM5LmO7EkEcID1CNtTQPEiXSSyQcpPNFKF/qh0BUoOFH/TiSISzninu7kSPXlvDcW//PasfIvnYQGUaxIgKeL/JhBFcJxHLBLBcGKjTRBWFB9K8R9JBBWOrSZLR5PdSb2fAKLpFEp22flyu15oXqVpZMDR+AYFIENLkAV3IAaqAMMnsALeAVvxrPxbnwYn9PWJSObOQAzML5+AYaLnMY=</latexit>

x

R(x)

https://distill.pub/2017/feature-visualization/；https://distill.pub/2018/building-blocks/
38

Multifaceted Feature Visualization
max Sc (x)Class maximization w.r.t. inputs<latexit sha1_base64="JwqVfyYMRxwozIXm935PQG+x6Ow=">AAACFnicbVBLSwMxGMzWV62vVY+CBItQD5bdKuix6MVjffQB7bJks2kbmmSXJCstS2/+CH+DVz17E69ePfpPTB8H2zoQmMx8w5dMEDOqtON8W5ml5ZXVtex6bmNza3vH3t2rqSiRmFRxxCLZCJAijApS1VQz0oglQTxgpB70rkd+/ZFIRSPxoAcx8TjqCNqmGGkj+fZhi6O+n/aH8N7Hhf4JPIUtZuIhgnfm6tt5p+iMAReJOyV5MEXFt39aYYQTToTGDCnVdJ1YeymSmmJGhrlWokiMcA91SNNQgThRXjr+xxAeGyWE7UiaIzQcq38TKeJKDXhgJjnSXTXvjcT/vGai25deSkWcaCLwZFE7YVBHcFQKDKkkWLOBIQhLat4KcRdJhLWpbmZLwIemE3e+gUVSKxXds2Lp9jxfvpq2kwUH4AgUgAsuQBncgAqoAgyewAt4BW/Ws/VufVifk9GMNc3sgxlYX7+QFp5F</latexit>

x

R(x)

Multifaceted Feature Visualization: (Nguyen et al. ICML 2016 Best Paper Award)
39

Activation Maximization

https://distill.pub/2017/feature-visualization/；https://distill.pub/2018/building-blocks/
40

LIME (Local Interpretable Model-Agnostic Explanations)models are trained to approximate the predictions of‣ Surrogate
the underlying black box model (model-agnostic approach‣ Explain the decision by evidence of interpretable region“Why Should I Trust You?” Explaining the Predictions of Any Classi er (Ribeiro et al. KDD 2016)
Model-Agnostic Interpretability of Machine Learning (Ribeiro et al. AAAI 2018)

)

fi

41

LIME: More Examples

42

In uence Functions‣ In uence of model’s prediction by training point‣ Identify the training points “responsible” for a given predictio‣ How predictions change if removing a training point z<latexit sha1_base64="jcPl1mJepOWnhivvYHIpURud3Dw=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjIB8wHJEfY2e8mS3b1jd084j/wCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66Rdq3pX1VrzulK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuPvZOY</latexit>r✓ L(ztest , ✓)T H✓ 1 r✓ L(z, ✓)n
X1r2✓ L(zi , ✓)Hessian H✓ =n i=1I(z, ztest ) =<latexit sha1_base64="6oRtHVnqo3ZhhpInVEmz9SguEO0=">AAACznicfVFNaxsxENVuvxL3y22PuYiahgQcs+sW2kMNob2k0IMLcRKw7EUra2NhrXaRZkMdIXrtLyz0mH8S2d6GblI6IHi8eU8zvElLKQxE0e8gvHf/wcNHW9utx0+ePnvefvHyxBSVZnzEClnos5QaLoXiIxAg+VmpOc1TyU/TxedV//SCayMKdQzLkk9yeq5EJhgFTyXtXySnMGdU2i9u77KLLxNLgH8HDNyA29/FA3yAiaKppAmBOQeKbxxfvaMh724U+9NjfFSrp/Ygdv/xd3HtIaT1x4N3B5hkmjIbO6scMVWeWDGI3VQ1V5n2m7uIm9+SdifqRevCd0Fcgw6qa5i0r8isYFXOFTBJjRnHUQkTSzUIJrlrkcrwkrIFPedjDxXNuZnYdf4Ov/HMDGeF9k8BXrN/OyzNjVnmqVeu9jW3eyvyX71xBdmHiRWqrIArthmUVRJDgVfHxDOhOQO59IAyLfyumM2pTw78yRtT0tz5TOLbCdwFJ/1e/LbX//auc/ipTmcL7aDXaA/F6D06REdoiEaIBR+DNFgEMhyGF6ELf2ykYVB7XqFGhT+vAQKx4MM=</latexit>‣ How predictions change if a training point z is modi ed<latexit sha1_base64="jcPl1mJepOWnhivvYHIpURud3Dw=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjIB8wHJEfY2e8mS3b1jd084j/wCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66Rdq3pX1VrzulK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuPvZOY</latexit><latexit sha1_base64="XJGmyOdBZGtl8G2DyMwEF0qynxc=">AAACiXicfVHRShtBFJ1da2vV1qiPpTA01CrEsGsFRSiIvljog4JRIRuXu5MbMzg7u8zcFeOyH9HP62O/wtdOkqU0Kr0wcDj3nLmXc5NcSUtB8Mvz517Nv36z8HZxafnd+5XG6tqFzQojsCMylZmrBCwqqbFDkhRe5QYhTRReJrfH4/7lHRorM31Ooxx7KdxoOZACyFFx42eUAg0FqPJ7tfnQ4g9xGRHeEye0VG1t8G98m0caEgVxREMk4H8dP5xjRt6aKrauz/lJrb4ut8Oq9t//76MWr81xoxm0g0nx5yCsQZPVdRo3fkf9TBQpahIKrO2GQU69EgxJobBajAqLOYhbuMGugxpStL1yEl3FPzumzweZcU8Tn7D/OkpIrR2liVOOt7VPe2PypV63oMF+r5Q6Lwi1mA4aFIpTxsd34H1pUJAaOQDCSLcrF0MwIMhda2ZKklYuk/BpAs/BxU47/NreOdttHh7V6SywD+wT22Qh22OH7ISdsg4T7NH76G14X/wlP/T3/YOp1PdqzzqbKf/4D9zSxgs=</latexit>

I(z, ztest ) =r✓ L(ztest , ✓)T H✓ 1 rx r✓ L(z, ✓)z 7! z + ,r L(ztest , ✓0 ) = L(z, ztest )T<latexit sha1_base64="VCsMEKx7+Z5Tbvv72g7n5CV/coI=">AAACa3icbZHBTttAEIbXbmkDpW0ot7aHVSNUUFFkUyR6QUJw6aEHKhFAilNrvJ6QFeu1uztGTSw/KEcegGcoayeHBjrSSr/++XZn9G9SKGkpCG49/9nzlRcvO6trr9Zfv3nb3Xh3bvPSCByIXOXmMgGLSmockCSFl4VByBKFF8n1SdO/uEFjZa7PaFrgKIMrLcdSADkr7lo+41EGhaWcz75EKSqCXR79LiHlkYZEQTw3G4omAlT1o96exVVE+Ic4oaXa8TRBgs87h0vQLp/FLVa12M6vs/lTcbcX9IO2+FMRLkSPLeo07t5FaS7KDDUJBdYOw6CgUQWGpFBYr0WlxQLENVzh0EkNGdpR1YZT8y3npHycG3c08db990YFmbXTLHFks7x93GvM//WGJY2/jSqpi5JQi/mgcam4S7JJmqfSoCA1dQKEkW5XLiZgQJD7j6UpSVa7TMLHCTwV53v98Gt/7+d+7+h4kU6HfWCf2DYL2QE7Yt/ZKRswwW7ZX6/jrXr3/qb/3v84R31vcWeTLZW/9QCzOLxq</latexit>‣ Poising attack

Understanding Black-box Predictions via In uence Functions (Koh and Liang, ICML 2017)
n

?

?

fi

s

fl

fl

fl

43

In uence Functions

Understanding Black-box Predictions via In uence Functions (Koh and Liang, ICML 2017)

fl

fl

44

Counterfactual Explanations
Credit Evaluation‣ What do I need to change for the bank to approve my loan
‣ Which symptoms would lead to a different medical diagnosis?
0
min
max
(f
(x
)
✓
0
x

<latexit sha1_base64="WSB8T5SZYYH/DCkg6lH+UOmib04=">AAACWXicbVBNbxMxEHWWrxA+GuiRy4gIJRU02g1IcEGqyoVjkUhbKQ6rWa+3sWp7V/YsSrTk9/Ebqp76A7jCFScNgraMZPm9N/M0o5dVWnmK47NWdOv2nbv32vc7Dx4+erzVffL00Je1E3IsSl264wy91MrKMSnS8rhyEk2m5VF2+mHVP/oqnVel/UyLSk4NnlhVKIEUpLSL3CibNvP+ErjBecp18OYIf/5BkXKaScLBvL+zC4v+zpcRvIR8ME/jVxA0zjt/CbwH/i0Q2A0swLRJlmm3Fw/jdcFNkGxAj23qIO1e8LwUtZGWhEbvJ0lc0bRBR0pouezw2ssKxSmeyEmAFo3002YdxRJeBCWHonThWYK1+q+jQeP9wmRh0iDN/PXeSvxfb1JT8W7aKFvVJK24XFTUGqiEVa6QKycF6UUAKJwKt4KYoUNBIf0rWzKzyiS5nsBNcDgaJq+Ho09venv7m3Ta7Bl7zgYsYW/ZHvvIDtiYCfad/WA/2a/WedSK2lHncjRqbTzb7EpF278ByJiw3A==</latexit>y 0 )2 + d(x0 , x0 )d(x0 , x0 ) = kx0x0 k1‣ Adversarial example with sparsity of perturbations
Counterfactual explanations without opening the black box (Wachter et al. 2017)

 

?

45

Is Google’s DeepDream Art?

46

Deep Generative Representation

47

Disentangled Representations‣ Factorize distribution over the latent variable‣ Single change in factor should lead to single change representationss48

Application: Image Translation‣ Image resynthesis by manipulating latent factors

Multi-Attribute Transfer via Disentangled Representation (Zhang et al., AAAI 2019)49

gAdversarial Machine Learnin(Reliability and Robustness)

Extreme Reliability and Safety

Air traf c control

Autonomous vehicles

Surgery robots

Medical diagnosisfi51

Problem: DNNs are Brittle

Published as a conference paper at ICLR 2015

+ .007 ⇥=xsign(rx J(✓, x, y))“panda”
57.7% confidence“nematode”
8.2% confidencex+✏sign(rx J(✓, x, y))“gibbon”
99.3 % confidence

Figure 1: A demonstration of fast adversarial example generation applied to GoogLeNet (Szegedy
et al., 2014a) on ImageNet. By adding an imperceptibly small vector whose elements are equal to
the sign of the elements of the gradient of the cost function with respect to the input, we can change
GoogLeNet’s classificationof the image.
Here ourbetween✏ of .007 corresponds
to theMLmagnitude of theInconsistentperception
human andsmallest bit of an 8 bit image encoding after GoogLeNet’s conversion to real numbers.

(Goodfellow
Let ✓ be the parameters of a model,x the inputettoal.,
the ICLR
model,2015)
y the targets associated with x (formachine learning tasks that have targets) and J(✓, x,y) be the cost used to train the neural network.
52
We can linearize the cost function around the current value of ✓, obtaining an optimal max-norm

Reliability: Medical Diagnosis(Finlayson et al. Science 2019)
Adversarial attacks on medical machine learning

53

Robust Physical-World Attacks(Eykholt et al., Robust physical-world attacks on deep learning visual classi cation, CVPR 2018)
fi

54

→ Security

More Examples

Recognized Input Image[Carlini Wagner 2018]:
Voice commands that are
unintelligible to humans[Sharif Bhagavatula Bauer Reiter 2016]:
Glasses that fool face recognition• Adversarial patch•[Sharif Bhagavatula Bauer Reiter 2016]:
Glasses that fool face recognition

Why Is This Brittleness of ML aWhy
Problem?
IBM ResearchAI
Is ThisBrittlenessof ML a(Brown et al., 2017)→ Security[Carlini Wagner 2018]:
Voice commands that are
unintelligible to humans→ Security[Carlini Wagner 2018]:
Voice commands that are
unintelligible to humans[Simen Thys et al., 2019
Fooling automated surveillance cameras:
adversarial patches to attack person
detection]

55

Robustness

Reward-driven model robustness
Accuracy vs. Adversarial Robustness
mprovement

Accuracy(D. Su et al., Is Robustness the Cost of Accuracy? - A Comprehensive Study on the Robustness of 18
Deep Image Classi cation Models, ECCV 2018)fi

56

Limitation
ofML
Framework
A Limitation of the (Supervised) ML Framework

A Limitation of the (Supervise

Measure of performance:
Fraction of mistakes during testing
But: In reality, the distributions
we use ML on are NOT the ones
we train it on
Training

Inference

All training and testing data
examples drawn
independently from same
distribution

Training

=

Frac

B
w

Inference

Real-world application
57

Implication of Adversarial Examples
has high score of accuracy but not suf ciently‣ ML
intelligen

‣
Risky
for
safetycriticalapplication‣
LimitationsofcurrentMLmethod‣Trustbetween
human
and
AI
‣

Distinct principles between human perception and M

L

fi

s

s

t

58

Attacks on ML Pipeline
Recovery of sensitive
training data
Training phrase

Training data

(X, Y )

Learning
Algorithm

Inference phrase

Mode

θ

Test outputy*Poisoning
Training Set

Test inpux*

Adversarial
Examples

 

 

t

 

l

59

Model TheftPoisoning Attack‣ By poisoning training data, the model will be compromise
backdoors in training data, such that the data with‣ Planting
backdoors will be misclassi ed
Datapoisoning
Learning
process

Weight poisoning
Clean Model

Poisoned Model
d

fi

 

 

60

Trojan Attack
Modified Samples
Target Label: 4

Label 4
(Target)Trigger:Label 7

...
...

Train

...

...

Modified Training Set

Backdoor
Configura!on

Inputsw/ Trigger

Backdoored DNN

Inputsw/o Triggera) Trainingb)

Fig. 1. An illustration of backdoor attack. The backdoor target is label 4, and the trigger pattern is a white squar
backdoor, part of the training set is modified to have the triggerLabelstampedand label modified to the target label.
4
the model will recognizeInputs
samples with trigger as the target label. Meanwhile,
theLabel)model can still recognize corr
(TargetTrain

w/ Trigger

Label 4labels can be infected, but we assume the majorityLabelof labels
A
B
5
Inputsremain uninfected. By their nature, these backdoors prioritize(CorrectClean
Labels)w/oTrigger
Model
Backdoored DNN
Label
7
stealth, and an attacker is unlikely to risk detection by emMinimum
model. The attacker
misclassify ag bedding many backdoors into a singleb) Inference
can also use one or multiple triggers to infect the same target
Trigger
et is label
4, andthe trigger
pattern
is a whitesquare andon thebottomBackdoor
right corner.
When
injecting(BolunWang
et al., Neural
Cleanse:Identifying
Mitigating
Attacksin Neural
Networks.
label.
A

Dimension
ger stamped and label modified to the target
label. After
trained 2019)with the modified
training set,IEEE Security
and Privacy,
B. DefensetheAssumptions
andGoals correct label for any sample withoutabel. Meanwhile,model can stillrecognize
trigger.
Infected
61

ModelCLR 2020

Backdoor Attack against Federated Learning
DBA: distributed backdoor attack
Gt+1
federated learning aggregator…

poisonker

distributed attackers

benign participants…

…
local trigger 1rGt

t setting)poisonlocal trigger 2

local trigger 3

local trigger 4

local triggers(b) DBA: distributed backdoor attack (ours)(Chulin Xie, et al., DBA: Distributed Backdoor Attacks against Federated Learning. ICLR 2020)nd distributed backdoor attacks (DBA)on FL. The aggregator at
62

Evasion Attack: Adversarial Examples

Loss Function for Attack
e.g. cat
𝑥
close
𝑥

?

Network
𝑓𝜃

𝑦

𝑓𝜃 𝑥

DNN

𝑦

𝑓𝜃 𝑥

close 𝑦 𝑟far
𝑦 𝑓𝑎 𝑠close
e.g. fishk𝑟 attac 𝑥 fixedvs.
non-targeted• Training: 𝐿 Target
𝜃
𝐶
𝑦
,
𝑦
𝑟𝑎𝑖𝑛
White-box• Non-targeted
Attack: vs.
𝐿 𝑥black-box𝐶attack
𝑦 ,𝑦 𝑟𝜃 fixed63• Targeted Attack:

sThe
used
in previous
work that
attemptedthe
to analyze
neuralnetworks
thatfigureshows images
that maximize
activations
in a random
direction
onbasis).
problems.
These
works
activationof a hidden unit as am
Imageswithineachinterpretrow shareansemantic
properties.
k for input images which maximize the activation value of this single

Intriguing Properties of NN (1)Naturalbasis distribution
vector
mages
the
data
that the network
not trained
can befrom
formally
statedas w.r.t.
visual inspectionof imageswas
x0 , which
satisfyon
i-th hidden unit
inablevalue):with the i-th hiddenhidden
orassociated
unit.
layer
(a) Unit sensitive to white flowers.

(b) Unit sensitive to postures.

(a) Unit sensitive to white flowers.

(b) Unit sensitive to postures.

activations0x =arg maxvh 2(x),iy random
direction
Rnei gives
rise to similarly interpretable selly, we find thatx2I
images x0 are semanticallyrelatedtospecieachother,
forBasis
activation has
c semantic
property
Input images(c) Unit senstive to round, spiky flowers.

hidden layer
activations2
x = arg maxh (x), vi
0x2I

Random vector(d) Unit senstive to round green or yellow
objects.
(c) Unit senstive to round, spiky flowers.
(d) Unit senstive to round green or yellow
objects.
Figure 3: Experiment performed on ImageNet. Images stimulating single unit most (maximum stimulation innaturalbasis
direction).performed
Images withineach rowImages
share many
semantic
properties.
Figure 3:
Experimenton ImageNet.
stimulating
singleunit most (maximum stimulation innatural basis direction). Images within each row share many semantic properties.

(a) Direction sensitive to white, spread
flowers.
(a) Direction sensitive to white, spread
flowers.

(b) Direction sensitive to white dogs.
(b) Direction sensitive to white dogs.

(c) Direction sensitive to spread shapes.
(c) Direction sensitive to spread shapes.

(d) Direction sensitive to dogs with brownheads.
(d)Direction sensitive to dogs with brownheads.
Figure 4: Experiment performed on ImageNet. Images giving rise to maximum activations in a random direc
activations
also
hasgiving
speci
c semantic
propertytion(maximumstimulation
in a randombasis).
Imageswithineachtorow
share many
semantic
Figure4:Random
Experiment
performed
on ImageNet.
Imagesrise
maximum
activationsin aproperties.
random direction (maximum stimulation in a random basis). Images within each row share many semantic properties.

basis is not better than a random basis for inspecting the propertiesn the notion that neural networks disentangle variation factors acrossmodel for weakly-supervised localization). Such global analyses are useful in that they can make u
understandbetter the input-to-output
mapping
represented
by thearetrainedmodel for weakly-supervised
localization).
Suchglobal analyses
usefulnetwork.
in that they can make us
understand better the input-to-output mapping represented by the trained network.
Generally speaking, the output layer unit of a neural network is a highly nonlinear function of it
input.
When
it is trainedwith layer
the cross-entropy
lossnetwork(using isthea highly
Softmax
activation
function),
Generally
speaking,
the output
unit of a neuralnonlinearfunctionof itsi
represents
a conditional
distributionof the label given
the input
the training
set function),presented itso
input. When
it is trainedwith the cross-entropy
loss (using
the(andSoftmax
activationfar).
It hasabeen
argued distribution[2] that the ofdeep
non-linear
and sotherepresents
conditional
the stacklabel ofgiven
the inputlayers(and inthebetween
training theset input
presented
far). Itunit
has ofbeen
argued[2] thatarethea deep
stack
non-linear
layersin betweengeneralization
the input andprio
theoutputa neural
network
way fortheofmodel
to encodea non-local
outputof space.
a neuralInnetwork
are a itway
for the model
encode fora non-local
generalization
overtheunit
input
other words,
is assumed
that istopossible
the output
unit to assignpriornonover the input
space.
In other non-epsilon)
words, it is assumed
that istopossible
the input
outputspace
unit tononsignificant(and,presumably,probabilities
regions forof thethatassign
containno(Szegedy et al. Intriguing propertiesof examples
neuralnetworks,
ICLR
2014)
significant(and, presumably,
non-epsilon)
probabilities
to regionsforofinstance,
the inputthe
space
thatobjects
containfrom
notraining
in their vicinity.
Such regions
can represent,
same
training examples
in their
Such regions
can represent,
forwhich
instance,
thenonethelesssame objects
fromdifferent
viewpoints,
whichvicinity.
are relatively
far (in pixel
space), but
share
both
the
64
different
viewpoints,
which
are
relatively
far
(in
pixel
space),
but
which
share
nonethelessboththelabel and the statistical structure of the original inputs.
label and the statistical structure of the original inputs.

Uninterpretable and counter-intuitive properties of DN

‣

N

fi

fi

distinction
between individuallevel units
and onrandomactivations
claimNousing
a convolutional
neuralhigh
networktrainedMNIST.
We
Figure 1 shows images that maximize the activations in the natural
ges that maximize the activation in random directions. In both cases

(x, l) is a hard problem, so we approximate it by using a box-consnd an approximation of D(x, l) by performing line-search to find t
Intriguing
Properties
of fNN(2)
minimizer
r of the following
problem satisfies(x + r)= l.
• Minimize c|r| + lossf (x + r, l) subject to x + r 2 [0, 1]
Optimization of
Perturbation

Adversarial
Example

m

Wrong
Label

penalty function method would yield the exact solution for Ds, howeverneuralnetworks
arenon-convex
ingeneral,so
weend
Ostrich, struthio, camelus
ase.
Experimental results(a)(a) Even columns: adversarial examples for a linear (FC) classifier (stddev=0.06)(b) Even columns: adversarial examples for a 200200-10 sigmoid network(stddev=0.063)(c) Randomly distorted
samples by Gaussian noisewith stddev=1. Accuracy:
51%.

(b)

“minimimum distortion” function D has the following intriguing p
Uninterpretable and counter-intuitive properties of DN
by informal
evidenceandquantitative
experimentsin
this
section:Hardly perceptible perturbation can cause misclassi cation of networ
Figure 7: Adversarial examples for a randomly chosen subset of MNIST compared with randomly distorted

Figure 5: Adversarial examples generated for AlexNet [9].(Left) is a correctly predicted sample, (center) difexamples.
ference between correct image, and image predicted incorrectly magnified by 10x (values shifted by
128 and Odd columns correspond to original images, and even columns correspond to distorted counterparts.
clamped), (right) adversarial example. All images in the right column are predicted to be an “ostrich,
StruthioThe
adversarial examples generated for the specific model have accuracy 0% for the respective model. Notecamelus”. Average distortion based on 64 examples is 0.006508. Plase refer to http://goo.gl/huaGPbthat while the randomly distorted examples are hardly readable, still they are classified correctly in half of thefor full resolution images. The examples are strictly randomly chosen. There is not any postselection involved.

cases, while the adversarial examples are never classified correctly.

‣
These distorted images or adversarial examples generalize fairly well even‣
1. For toall
the models
networks
wedifferent
studieddifferent
trained bydataset(MNIST, QuocNet [10],Model Name

(a)

(b)

Training error

Test error

Av. min. distortion

= 10

4

6.7%

7.4%

0.062

Softmax with

= 10

2

10%

9.4%

0.1

FC10(1)

Softmax with

=1

21.2%

20%

0.14

FC100-100-10

Sigmoid network

0%

1.64%

0.058

0%

1.54%

0.065

0.57%

1.9%

0.086

FC10(10

4

FC10(102

Description))

Softmax withple, we havealwaysmanagedtogeneratevery
close,
visu(Szegedy et al. Intriguing properties of neural networks, ICLR 2014)versarial examples that are65 misclassified by the original

Figure 6: Adversarial examples for QuocNet [10]. A binary car classifier was trained on top of the last layer
features without fine-tuning. The randomly chosen examples on the left are recognized correctly as cars, while
the images in the middle are not recognized. The rightmost column is the magnified absolute value of thedifference between the two images.

FC200-200-10
AE400-10

Sigmoid network= 10 5 , 10 5 , 10 6
= 10

Autoencoder with Softmax

5

, 10

5

= 10

6

, 10

6

Table 1: Tests of the generalization of adversarial instances on MNIST.
k

N

fi

the original training set all the time. We used weight decay, but no dropout for this network. Forensor or data storage apparatus associated with our problem.

Whyvector
Do Adversarial
ExamplesHappen?
en a weightw and an adversarial
example x̃:x̃ = x +> ⌘>>w x̃ = w x + w ⌘.
<latexit sha1_base64="Hr70mVolgGLagrCx+ufPcejnrJc=">AAACGnicbZDLSgMxFIYzXmu9VV26iRZBEMpMFXQjiG5cVrAX6JSSSU9rMJkZkjNiGbr2JXwFt7p3J27duPVJzLRd2NYDIR//fw4n+YNYCoOu++3MzS8sLi3nVvKra+sbm4Wt7ZqJEs2hyiMZ6UbADEgRQhUFSmjEGpgKJNSD+6vMrz+ANiIKb7EfQ0uxXii6gjO0Uruw56OQHaCpHyhKHwf0nGb0eDS8fEDWLhTdkjssOgveGIpkXJV24cfvRDxRECKXzJim58bYSplGwSUM8n5iIGb8nvWgaTFkCkwrHX5lQA+s0qHdSNsTIh2qfydSpozpq8B2KoZ3ZtrLxP+8ZoLds1YqwjhBCPloUTeRFCOa5UI7QgNH2bfAuBb2rZTfMc042vQmtgRqkLeheNMRzEKtXPKOS+Wbk+LF5TieHNkl++SQeOSUXJBrUiFVwskTeSGv5M15dt6dD+dz1DrnjGd2yEQ5X79B9Z90</latexit>

Input
Data

Adversaria
ExampleT

Linearlayer

TTw x̃ = w x + w ⌘<latexit sha1_base64="P/LkIPqytcbEYCCbZyyc7RhA6T4=">AAACMXicbVDLSgMxFM34rPVVdalIUARBKDN1oRtBdOOyBWsLnVoy6a0Gk5khuaOWoUt/w59w0w9wq/vupFt/wkyr4OtAyLnn3MtNThBLYdB1B87E5NT0zGxuLj+/sLi0XFhZvTBRojlUeSQjXQ+YASlCqKJACfVYA1OBhFpwc5r5tVvQRkThOXZjaCp2FYqO4Ayt1Coc+oGid5fnlPooZBvSrL7v0SP6ZYwEuvdVZ5cPyFqFbbfojkD/Eu+TbB9v9CvvD5v9cqsw9NsRTxSEyCUzpuG5MTZTplFwCb28nxiIGb9hV9CwNGQKTDMd/bBHd6zSpp1I2xMiHanfJ1KmjOmqwHYqhtfmt5eJ/3mNBDuHzVSEcYIQ8vGiTiIpRjSLi7aFBo6yawnjWti3Un7NNONoQ/2xJVC9vA3F+x3BX3JRKnr7xVLFpnNCxsiRdbJFdolHDsgxOSNlUiWcPJJn8kJenSdn4Lw5w3HrhPM5s0Z+wHn/ABVPq50=</latexit>

Perturbation

T maximize this increase
uses the activation to grow by w ⌘.Wecanw ⌘ / ✏mn
aint on ⌘max
by assigning⌘=sign(w).
Ifw
hasn
dimensions
and
the
T
w ⌘
⌘
nt of the weightvector is m, then the activation
willDimensionality
grow by ✏mn.
Averagek⌘k1 < ✏th the dimensionalityof the problemmagnitude
but theof wchange ofininput
activationn grow linearly with n, then for high dimensional problems, we can
Early explanations for adversarial examples is highly nonlinearity and over tting of
es to ‣the
input
that
add
up
to
one
large
changetotheoutput.
We
NN (is it wrong?
ccidental
steganography,”wherea linear model
is models
forcedto attend
Adversarial
samplesarecausedbyhigh-dimensionality
of
input
and
being
‣
tooclosely
linear rather
than toogns most
withitsnonlinea
weights, even if multiple signals are present
eater amplitude.
‣ Linear models lack the capacity to resist adversarial perturbatio><latexit sha1_base64="nYUPdclBbxyAgkTqAx9mvVOpzNE=">AAACHnicbVA7T0IxGO3FF+ILddSYRkLiRO7FQUeiiyMkvBIumt5StKGvtL0aQhid/BP+BTeiu5tx1dXVP2EvMIh4kqan5ztfvn4nUowa6/ufXmphcWl5Jb2aWVvf2NzKbu/UjYw1JjUsmdTNCBnCqCA1Sy0jTaUJ4hEjjah3ntQbt0QbKkXV9hVpc3QtaJdiZJ10lc3DMOLw7rKaXCGxCIZKS2WleyhDmRSQQ+fL+QV/DDhPginJlfZHle/7g1H5KvsVdiSOOREWM2RMK/CVbQ+QthQzMsyEsSEK4R66Ji1HBeLEtAfjdYYw75QO7ErtjrBwrP7uGCBuTJ9HzsmRvTF/a4n4X60V2+5pe0CFii0ReDKoGzPotk2ygR2qCbas7wjCmrq/QnyDNMLWJTgzJeLDjAsl+BvBPKkXC8FxoVhx6ZyBCdJgDxyCIxCAE1ACF6AMagCDB/AEnsGL9+i9em/e+8Sa8qY9u2AG3scP3FullA==</latexit><latexit sha1_base64="iBKNDkKQNg65ExJ87YMA3empe6A=">AAACD3icbVC7TsMwFHV4lvIKILGwWFSVmKqkDDChAgOMRepLakrkuE5r1U4i2wGq0I/gF1hBrGyIlT+AlYnPwGk70JYjXencc+7VvTpexKhUlvVlzM0vLC4tZ1ayq2vrG5vm1nZNhrHApIpDFoqGhyRhNCBVRRUjjUgQxD1G6l7vPPXrN0RIGgYV1Y9Ii6NOQH2KkdKSa+46HN25DlEIOh6Ht9cVmDaumbMK1hBwlthjkiudXLzkTz9/yq757bRDHHMSKMyQlE3bilQrQUJRzMgg68SSRAj3UIc0NQ0QJ7KVDP8fwLxW2tAPha5AwaH6dyNBXMo+9/QkR6orp71U/M9rxso/biU0iGJFAjw65McMqhCmYcA2FQQr1tcEYUH1rxB3kUBY6cgmrnh8kNWh2NMRzJJasWAfFopXOp0zMEIG7IF9cABscARK4BKUQRVgcA8ewRN4Nh6MV+PNeB+NzhnjnR0wAePjF64Xn6c=</latexit><latexit sha1_base64="U7FiCq9xMiqzkTj8tvjGJrO4hYM=">AAAB/XicbVC7SgNBFL3rM66vqKXNYBCswm4stBGDNpYRzAOSEGYns8mYmdllZlYJS/AXbLUVO7H1E/wEsfVLnDwKk3jgwuGce7n3niDmTBvP+3YWFpeWV1Yza+76xubWdnZnt6KjRBFaJhGPVC3AmnImadkww2ktVhSLgNNq0Lsc+tU7qjSL5I3px7QpcEeykBFsrFRBjUCg+1Y25+W9EdA88Sckd/7pnsUvX26plf1ptCOSCCoN4Vjruu/FppliZRjhdOA2Ek1jTHq4Q+uWSiyobqajawfo0CptFEbKljRopP6dSLHQui8C2ymw6epZbyj+59UTE542UybjxFBJxovChCMToeHrqM0UJYb3LcFEMXsrIl2sMDE2oKktgRi4NhR/NoJ5Uink/eN84drLFS9gjAzswwEcgQ8nUIQrKEEZCNzCIzzBs/PgvDpvzvu4dcGZzOzBFJyPX6H2mH8=</latexit><latexit sha1_base64="1FPVZbCVgfjSyYj3iWo04SJkPl4=">AAACGXicbVC7SgNBFJ2NrxhfUUubIUEQhLAbCy0sgjaWEUwiZEOYncwmQ2Zml5m7whLT+gV2/oKt9nZia5XWL3HyKEzigYHDOedy75wgFtyA646czMrq2vpGdjO3tb2zu5ffP6ibKNGU1WgkIn0fEMMEV6wGHAS7jzUjMhCsEfSvx37jgWnDI3UHacxaknQVDzklYKV2HvuP2A8k9hkQy9s+VyGk+NIKseFiHCm6JXcCvEy8GSlWCv7p86iSVtv5H78T0UQyBVQQY5qeG0NrQDRwKtgw5yeGxYT2SZc1LVVEMtMaTH4yxMdW6eAw0vYpwBP178SASGNSGdikJNAzi95Y/M9rJhBetAZcxQkwRaeLwkRgiPC4FtzhmlEQqSWEam5vxbRHNKFgy5vbEshhzpbiLVawTOrlkndWKt/adq7QFFl0hAroBHnoHFXQDaqiGqLoCb2iN/TuvDgfzqfzNY1mnNnMIZqD8/0Lfk2jKA==</latexit>

Generalization of adversarial examples across different models can be explained as‣
the perturbations
with the weightvectors of model
mple linear
model being
can highly
havealigned
adversarial
examples
if its input has suf-

fi

n

r

)

 

l

explanationsforadversarial
examplesinvokedhypothesizedprop(Goodfellow et al. Explaining and Harnessing Adversarial Examples, ICLR 2015)
as their supposed highly non-linear
nature. Our hypothesis based66

FGSM: Fast Gradient Sign Methodx̃ = x + ⌘Adversarial
Examples<latexit sha1_base64="Hr70mVolgGLagrCx+ufPcejnrJc=">AAACGnicbZDLSgMxFIYzXmu9VV26iRZBEMpMFXQjiG5cVrAX6JSSSU9rMJkZkjNiGbr2JXwFt7p3J27duPVJzLRd2NYDIR//fw4n+YNYCoOu++3MzS8sLi3nVvKra+sbm4Wt7ZqJEs2hyiMZ6UbADEgRQhUFSmjEGpgKJNSD+6vMrz+ANiIKb7EfQ0uxXii6gjO0Uruw56OQHaCpHyhKHwf0nGb0eDS8fEDWLhTdkjssOgveGIpkXJV24cfvRDxRECKXzJim58bYSplGwSUM8n5iIGb8nvWgaTFkCkwrHX5lQA+s0qHdSNsTIh2qfydSpozpq8B2KoZ3ZtrLxP+8ZoLds1YqwjhBCPloUTeRFCOa5UI7QgNH2bfAuBb2rZTfMc042vQmtgRqkLeheNMRzEKtXPKOS+Wbk+LF5TieHNkl++SQeOSUXJBrUiFVwskTeSGv5M15dt6dD+dz1DrnjGd2yEQ5X79B9Z90</latexit>

Adversaria
ExamplePerturbation
Adversarial
AttackInput
Data

Perturbation
Model parameter θ

ModelParameters

Input
Data

Label

⌘ = ✏ sign(rx J(✓, x, y))<latexit sha1_base64="XWG1iBeewZazep36G1YKDPtmhRc=">AAACOnicbVDLSiNBFK12fE18Zcalm8IgRJDQrYJuBEc34krBqJAO4XblJimsqm6qbktCk++Zn5hfcKu4dSdu5wOsxCx8HSg4nHMu99ZJMiUdheFjMPVjemZ2bv5naWFxaXml/Ov3pUtzK7AuUpXa6wQcKmmwTpIUXmcWQScKr5Kb45F/dYvWydRc0CDDpoaukR0pgLzUKv+JkYAfcB5j5qRKTbzFY8I+FU52zZBXYwOJglafn1bjRHuv5wd8xvP+Fh9sbrbKlbAWjsG/kmhCKmyCs1b5KW6nItdoSChwrhGFGTULsCSFwmEpzh1mIG6giw1PDWh0zWL81SHf8Eqbd1LrnyE+Vt9PFKCdG+jEJzVQz332RuJ3XiOnzn6zkCbLCY14W9TJFaeUj3rjbWlRkBp4AsJKfysXPbAgyLf7YUuihyVfSvS5gq/kcrsW7dS2z3crh0eTeubZGltnVRaxPXbITtgZqzPB/rI7ds8egn/BU/AcvLxFp4LJzCr7gOD/K480q8g=</latexit>

Gradient of loss function
w.r.t. input(Goodfellow et al. Explaining and Harnessing Adversarial Examples, ICLR 2015)
 

 

l

67

stead uses inputs that are unlikely to occur naturally but that expose flaws in the ways thconceptualizes its decision function. At the time, this procedure was never demonstra
ve beyond dropout on a state of the art benchmark. However, this was partially because
lt to experiment extensively with expensive adversarial examples based on L-BFGS.

Objective of Adversarial Training

und that training with an adversarial objective function based on the fast gradient sign mAdversarial
Training
effective
regularizer:
˜ x, y) = ↵J(✓, x, y) + (1J(✓,↵)J(✓, x + ✏sign (rx J(✓, x, y)) .

of our experiments, we used ↵ = 0.5. Other values may work better; our initial guess oarameterLoss
workedthat wefordid not
feel the need to explore more. This app
for well enoughRegularizer
Adversaria
Perturbationthat wetraining
continually
our supply of adversarial
examples, to make them resist the cu
dataupdateRobustness
Examplen of the model. Using this approach to train a maxout network that was also regularized
ut, we were able to reduce the error rate from 0.94% without adversarial training to 0
Adversarial examples are continually updated given current mode
dversarial training.

‣
The larger model capacity is required to reduce error on adversarial‣
served that
we were not reaching zero error rate on adversarial examples on the trainin
exampleed this problem by making two changes. First, we made the model larger, using 1600 uni
shows
greatnetwork
robustness
to adversarial
example adver
ather thanAdversariallythe 240 usedtrainedby themodel
original
maxoutfor thisproblem. Without
g, this causes
the model
to overfit
slightly,
andandgetinterpretablean error rate of 1.14% on the tes
The weightof modelare more
localized
dversarial training, we found that the validation set error leveled off over time, andAdversarial
trainingmaxout= Activeresult
learning
ow progress.
The originaluses early stopping, and terminates learning
idation set error
rate has not decreased for 100 epochs. We found that while the validati(Goodfellow et al. Explaining and Harnessing Adversarial Examples, ICLR 2015)
was very flat, the adversarial validation set68error was not. We therefore used early sto
s

l

 

l

 

s

‣
‣
‣

Optimization for Adversarial Attack
min E(x,y)⇠D L(✓, x, y)Standard
training<latexit sha1_base64="nWKvdnzh9EApVTggbL3wZKrHzL8=">AAACQ3icbVDLSgMxFM34tr6qLt0Ei1ChlJkq6LL4AEEXClaFThkyaWpDk8yQ3BGHYT7Kj/AbxJ2u3YlbwfSBaPVA4Nxz7+HenDAW3IDrPjsTk1PTM7Nz84WFxaXlleLq2pWJEk1Zg0Yi0jchMUxwxRrAQbCbWDMiQ8Guw95hv399x7ThkbqENGYtSW4V73BKwEpB8dSXXAU+dBkQ7EsC3TDMjvMgK99X0m3fcDlUKRHZUZ5jv/JdnuW4PDRW8H0Fp9tBseRW3QHwX+KNSAmNcB4UX/12RBPJFFBBjGl6bgytjGjgVLC84CeGxYT2yC1rWqqIZKaVDT6d4y2rtHEn0vYpwAP1pyMj0phUhnayf7EZ7/XF/3rNBDr7rYyrOAGm6HBRJxEYItxPELe5ZhREagmhmttbMe0STSjYnH9tCWVuM/HGE/hLrmpVb6dau9gt1Q9G6cyhDbSJyshDe6iOTtA5aiCKHtATekGvzqPz5rw7H8PRCWfkWUe/4Hx+AeBXsVU=</latexit>

Loss Input Label
Model
Parameter

min k kp

Goal:✓<latexit sha1_base64="12L/VfrQDnKx2Jkwsxhc2w38uoE=">AAACSHicbVBNaxRBFOxZjcYYzapHL00WIUEYZpKAAS9BLx5XyCaB7WV40/Mm26S7Z9L9JmSZ7N/yR/gPBC8e9OxNckvvxyEfFjQUVfV4ryuvtfKUJD+izqPHK0+erj5be77+4uVG99XrI181TuJAVrpyJzl41MrigBRpPKkdgsk1Hudnn2f+8QU6ryp7SJMaRwZOrSqVBApS1u0Lo2wmCtQEXFwtiLjKai7OGyi4ILyk1scUT8VHXmaCxkiwdfl+kdzmwuL5LX076/aSOJmDPyTpkvTYEv2s+0sUlWwMWpIavB+mSU2jFhwpqXG6JhqPNcgzOMVhoBYM+lE7//mUvwtKwcvKhWeJz9XbEy0Y7ycmD0kDNPb3vZn4P2/YULk/apWtG0IrF4vKRnOq+KxGXiiHkvQkEJBOhVu5HIMDSaHsO1tyMw2dpPcbeEiOduJ0N975utc7+LRsZ5W9ZZtsi6XsAztgX1ifDZhk39hP9pv9ib5Hf6N/0fUi2omWM2/YHXQ6N2xPsps=</latexit>

Perturbation

s.t. f✓ (x + ) 6= f✓ (x)
Model

Input

Model parameter

max L(✓, x + , y) s.t. k kp  ✏Optimization:<latexit sha1_base64="31c6EWNJxCfxGVtOs3RUTlGA4ew=">AAACVnicbVA9TxtBEF1fQvjIlyFlmhUoChHodEcKkNygUCQFBUgYkLyONbce4xV7H+zOIVvHtfyZ/I2U/AAoaWmhRuz5KAJkpJXevDdPM/uiTCtLQXDV8F69nnozPTM79/bd+w8fm/ML+zbNjcS2THVqDiOwqFWCbVKk8TAzCHGk8SA63qr0g1M0VqXJHo0z7MZwlKiBkkCO6jV/ixhGvUL0UROU3HU0lKCL7XJZ0BAJVvlopVZX+fgbFy3R4oJwRIX1yefO4vqzekKc9TIuNJ5wgZlVulqwFPjBpPhLED6Cpc3Wzz/nK18vdnrNa9FPZR5jQlKDtZ0wyKhbgCElNZZzIreYgTyGI+w4mECMtltMcij5F8f0+SA17iXEJ+y/jgJia8dx5Carf9rnWkX+T+vkNNjoFirJcsJE1osGueaU8ipU3lcGJemxAyCNcrdyOQQDklz0T7ZEcekyCZ8n8BLsr/nhd39t14Xzg9U1wz6zRbbMQrbONtkvtsPaTLK/7IbdsrvGZePem/Km61Gv8ej5xJ6U13wAIcS4sQ==</latexit>

Loss Adversaria
ExampleGradient:r✓ L(✓, x, y)<latexit sha1_base64="ZDDkzsNCTZnp2T1hyPisX2Di0Ik=">AAACJnicbVDLSgNBEJz1/Tbq0ctgEBQl7EZBj6IXDx4imChkQ+iddMzg7Owy0yuGJX/hR/gNXvXsTcSb/omTx8FXQUNR1U13V5Qqacn3372x8YnJqemZ2bn5hcWl5cLKas0mmRFYFYlKzFUEFpXUWCVJCq9SgxBHCi+jm5O+f3mLxspEX1A3xUYM11q2pQByUrNQCjVECpphCxUBD2OgjgCVn/W2QuogwS6/2xmau7y73SwU/ZI/AP9LghEpshEqzcJn2EpEFqMmocDaeuCn1MjBkBQKe3NhZjEFcQPXWHdUQ4y2kQ/+6vFNp7R4OzGuNPGB+n0ih9jabhy5zv7d9rfXF//z6hm1Dxu51GlGqMVwUTtTnBLeD4m3pEFBqusICCPdrVx0wIAgF+WPLVHcc5kEvxP4S2rlUrBXKp/vF4+OR+nMsHW2wbZYwA7YETtlFVZlgt2zR/bEnr0H78V79d6GrWPeaGaN/YD38QV3nqWn</latexit>

 

l

 

69
p

Kee
Inperceptibler L(✓, x + , y)<latexit sha1_base64="1n+g4XysLE8vzn9xTmOC9fojLFI=">AAACH3icbVDLSgNBEJyNr/iOevQyGMQIEnajoMegFw8eIhgNZEPonUzMkNnZZaZXDEs+wY/wG7zq2Zt4zdE/cfI4mMSChqKqm+6uIJbCoOsOnMzC4tLySnZ1bX1jc2s7t7N7b6JEM15lkYx0LQDDpVC8igIlr8WaQxhI/hB0r4b+wxPXRkTqDnsxb4TwqERbMEArNXNHvoJAQtPHDkegfgjYYSDTm35hLJ3Q5xPaO27m8m7RHYHOE29C8mSCSjP347ciloRcIZNgTN1zY2ykoFEwyftrfmJ4DKwLj7xuqYKQm0Y6eqhPD63Sou1I21JIR+rfiRRCY3phYDuHB5tZbyj+59UTbF80UqHiBLli40XtRFKM6DAd2hKaM5Q9S4BpYW+lrAMaGNoMp7YEYd9m4s0mME/uS0XvtFi6PcuXLyfpZMk+OSAF4pFzUibXpEKqhJEX8kbeyYfz6nw6X873uDXjTGb2yBScwS8O7aLc</latexit>

e

Tru
Label

l
Geometry of p-Norm<latexit sha1_base64="jHJvyDVhTUHaLRCZxsbPW+B7F/c=">AAAB+XicbVC7SgNBFL0bXzHxEbW0GYyCVdiNhZZBG8uI5gFxCbOT2WTIzOwyMxsISz7BVms7sRX8Ab/C0o/Q2smjMIkHLhzOuZd77wlizrRx3U8ns7K6tr6R3czlt7Z3dgt7+3UdJYrQGol4pJoB1pQzSWuGGU6bsaJYBJw2gv7V2G8MqNIskndmGFNf4K5kISPYWOmWt+N2oeiW3AnQMvFmpFg5/n7/GOR/qu3C130nIomg0hCOtW55bmz8FCvDCKej3H2iaYxJH3dpy1KJBdV+Ojl1hE6s0kFhpGxJgybq34kUC62HIrCdApueXvTG4n9eKzHhhZ8yGSeGSjJdFCYcmQiN/0YdpigxfGgJJorZWxHpYYWJsenMbQnEyGbiLSawTOrlkndWKt/YcC5hiiwcwhGcggfnUIFrqEINCHThAR7hyUmdZ+fFeZ22ZpzZzAHMwXn7BQzkmLI=</latexit>

Example x<latexit sha1_base64="TqraGdCnTtWa8f24z5/8QO6xAvA=">AAAB93icbVA9SwNBEJ2LXzF+RS1tFoNgFe6ioGXQxjIB8wHJEfY2e8mS3b1jd088jvwCW63txNafY+k/cZNcYRIfDDzem2FmXhBzpo3rfjuFjc2t7Z3ibmlv/+DwqHx80tZRoghtkYhHqhtgTTmTtGWY4bQbK4pFwGknmNzP/M4TVZpF8tGkMfUFHkkWMoKNlZrPg3LFrbpzoHXi5aQCORqD8k9/GJFEUGkIx1r3PDc2foaVYYTTaamfaBpjMsEj2rNUYkG1n80PnaILqwxRGClb0qC5+nciw0LrVAS2U2Az1qveTPzP6yUmvPUzJuPEUEkWi8KEIxOh2ddoyBQlhqeWYKKYvRWRMVaYGJvN0pZATG0m3moC66Rdq3pX1VrzulK/y9MpwhmcwyV4cAN1eIAGtIAAhRd4hTcndd6dD+dz0Vpw8plTWILz9QuMl5OW</latexit>

k kp  ✏<latexit sha1_base64="J8xbmptfLQCJAzq7mxRa+7T9Pf0=">AAACEnicbVC7TgJBFJ3FF4IP1E6biWhiRXax0JJoY4mJPBKWkNnZC0yYfTgzS0IWEj/Cb7DVysLO0PoDln6E1s4ChYAnucnJOffm3nuckDOpTPPTSK2srq1vpDcz2a3tnd3c3n5VBpGgUKEBD0TdIRI486GimOJQDwUQz+FQc3rXiV/rg5As8O/UIISmRzo+azNKlJZauUN7aLvAFbGHrRDbHO6xDaFkPDHzZsGcAC8Ta0bypZPv13E/+1Nu5b5sN6CRB76inEjZsMxQNWMiFKMcRhk7khAS2iMdaGjqEw9kM578MMKnWnFxOxC6fIUn6t+JmHhSDjxHd3pEdeWil4j/eY1ItS+bMfPDSIFPp4vaEccqwEkg2GUCqOIDTQgVTN+KaZcIQpWObW6L4410JtZiAsukWixY54XirQ7nCk2RRkfoGJ0hC12gErpBZVRBFD2gJ/SMXoxH4814N8bT1pQxmzlAczA+fgFPRqKn</latexit>

lp ball<latexit sha1_base64="pqU85S+lNnUPLYFbRBi3V2M25C4=">AAAB+XicbVBNSwMxEJ34WetX1aOXYBE8ld0q6LHoxWNF+wHtUrJptg1NskuSFcrSn+BVz97Eq7/Go//EtN2DbX0w8Hhvhpl5YSK4sZ73jdbWNza3tgs7xd29/YPD0tFx08SppqxBYxHrdkgME1yxhuVWsHaiGZGhYK1wdDf1W89MGx6rJztOWCDJQPGIU2Kd9Ch6Sa9U9ireDHiV+DkpQ456r/TT7cc0lUxZKogxHd9LbJARbTkVbFLspoYlhI7IgHUcVUQyE2SzUyf43Cl9HMXalbJ4pv6dyIg0ZixD1ymJHZplbyr+53VSG90EGVdJapmi80VRKrCN8fRv3OeaUSvGjhCqubsV0yHRhFqXzsKWUE5cJv5yAqukWa34l5Xqw1W5dpunU4BTOIML8OEaanAPdWgAhQG8wCu8oQy9ow/0OW9dQ/nMCSwAff0CCs2UbQ==</latexit>

k k1  ✏<latexit sha1_base64="lD34OJtmHwBAJoBBxG5vilh11SE=">AAACF3icbVC7SgNBFJ31Gd9RSwuHBEEQwq4WWgZtLCOYKGSWMDu5q4Ozs+vMXWGJKf0DG7/BVms7sbVM6Z84SSx8HRg4nHMu986JMiUt+v7Am5icmp6ZLc3NLywuLa+UV9daNs2NgKZIVWrOI25BSQ1NlKjgPDPAk0jBWXR1NPTPbsBYmepTLDIIE36hZSwFRyd1ypvslnVBIWe3HSZ1jAVlCq4pg8xKNUxU/Zo/Av1Lgi9SrVfYzv2gXjQ65Q/WTUWegEahuLXtwM8w7HGDUijoz7PcQsbFFb+AtqOaJ2DD3ugjfbrllC6NU+OeRjpSv0/0eGJtkUQumXC8tL+9ofif184xPgh7Umc5ghbjRXGuKKZ02ArtSgMCVeEIF0a6W6m45IYLdN392BIlfddJ8LuBv6S1Wwv2arsnrpxDMkaJbJAK2SYB2Sd1ckwapEkEuSOP5Ik8ew/ei/fqvY2jE97XzDr5Ae/9E9J2o7U=</latexit>

k k2  ✏<latexit sha1_base64="31LRNbT8x6oYU6BEiHyADPQK6Y0=">AAACEnicbVDJSgNBEO1xjXGLetNLkyAIQpiJBz0GvXiMYBbIDKGnpyZp0rPY3SMMk4AfkW/wqmdv4tUfyNE/sbMcTOKDgsd7VVTVc2POpDLNsbG2vrG5tZ3bye/u7R8cFo6OGzJKBIU6jXgkWi6RwFkIdcUUh1YsgAQuh6bbv5v4zWcQkkXho0pjcALSDZnPKFFa6hRO7YHtAVfEHnQq2ObwhG2IJeMTs2SWzSnwKrHmpFQt2pejcTWtdQo/thfRJIBQUU6kbFtmrJyMCMUoh2HeTiTEhPZJF9qahiQA6WTTH4b4XCse9iOhK1R4qv6dyEggZRq4ujMgqieXvYn4n9dOlH/jZCyMEwUhnS3yE45VhCeBYI8JoIqnmhAqmL4V0x4RhCod28IWNxjqTKzlBFZJo1K2rsqVBx3OLZohh85QEV0gC12jKrpHNVRHFL2gV/SG3o2R8WF8Gl+z1jVjPnOCFmB8/wIwu6Ev</latexit>70

k k1  ✏<latexit sha1_base64="t39tNPvSLaREhKb8QmnmO5GtuzM=">AAACEnicbVDJSgNBEO1xjXGLetNLkyAIQpiJBz0GvXiMYBbIDKGnpyZp0rPY3SMMk4AfkW/wqmdv4tUfyNE/sbMcTOKDgsd7VVTVc2POpDLNsbG2vrG5tZ3bye/u7R8cFo6OGzJKBIU6jXgkWi6RwFkIdcUUh1YsgAQuh6bbv5v4zWcQkkXho0pjcALSDZnPKFFa6hRO7YHtAVfEHnQsbHN4wjbEkvGJWTLL5hR4lVhzUqoW7cvRuJrWOoUf24toEkCoKCdSti0zVk5GhGKUwzBvJxJiQvukC21NQxKAdLLpD0N8rhUP+5HQFSo8Vf9OZCSQMg1c3RkQ1ZPL3kT8z2snyr9xMhbGiYKQzhb5CccqwpNAsMcEUMVTTQgVTN+KaY8IQpWObWGLGwx1JtZyAqukUSlbV+XKgw7nFs2QQ2eoiC6Qha5RFd2jGqojil7QK3pD78bI+DA+ja9Z65oxnzlBCzC+fwEvGqEu</latexit>Target Attacks
Goal:

min k kps.t. f✓ (x + ) = y

0

DNN model

Target Label<latexit sha1_base64="DgSA7cunuijp9IiwKhH+syLebhU=">AAACQnicbVBNSxxBFOwxJhrztUmOuTQuAUNgmDGCgRCQ5BI8KbgqbG+GNz1v3MbunrH7TXAZ9z/lR+Q3BHKKueYmXj3Y+3GImoKGoqoe73XltVaekuRXtHBv8f6DpeWHK48eP3n6rPP8xb6vGiexJytducMcPGplsUeKNB7WDsHkGg/y488T/+AbOq8qu0ejGgcGjqwqlQQKUtbZFkbZTBSoCbg4mxFxltVcnDRQcEF4Sq2PKR6LD7zMBA2RYO307Sz5hn/ko6+idspg1ukmcTIFv0vSOemyOXayzh9RVLIxaElq8L6fJjUNWnCkpMbximg81iCP4Qj7gVow6Aft9M9j/jooBS8rF54lPlX/nWjBeD8yeUgaoKG/7U3E/3n9hsr3g1bZuiG0craobDSnik8K5IVyKEmPAgHpVLiVyyE4kBRqvrElN+PQSXq7gbtkfz1O38XruxvdrU/zdpbZK7bK1ljKNtkW+8J2WI9J9p39ZL/ZefQj+htdRJez6EI0n3nJbiC6ugacdrFL</latexit>

Optimization problem:max{L(✓, x + , y)<latexit sha1_base64="ssY+oVa7lTDeAyvy4n81ZCYu+rc=">AAACf3ichVFNb9QwEHXCVylfS3ukSBYrRCvaKCmVWsSlohcOHIrEtpXWy2rizHatOolrT9Cu0pz5jRz5AfwHnN0caIvESJae35vRGz+nRitHcfwzCO/cvXf/wcrD1UePnzx91nu+duLKykocyFKX9iwFh1oVOCBFGs+MRchTjafpxVGrn35H61RZfKW5wVEO54WaKAnkqXHvh8hhNhYZagJR+wtNJej6c7MpaIoE23z2dqlu8/nWDv9PxzdhrMpxSzRcXFaQcUE4o9pFFDXiAxdXndPV2HCh8ZILNE7pdpN+HMWL4rdB0oE+6+p43PslslJWORYkNTg3TGJDoxosKamxWRWVQwPyAs5x6GEBObpRvQis4a89k/FJaf0piC/YvydqyJ2b56nvbN/rbmot+S9tWNHkYFSrwlSEhVwaTSrNqeRt+jxTFiXpuQcgrfK7cjkFC5L8H11zSfPGZ5LcTOA2ONmNknfR7pe9/uHHLp0V9oK9YpssYfvskH1ix2zAJPsdrAcbwcswCN+EURgvW8Ogm1ln1yp8/weKEMHv</latexit>

Loss w.r.t. true label

0L(✓, x + , y )}Loss w.r.t. target label

71

s.t. k kp  ✏Targeted Attacks: Example

72

White-box Attacks
Fast approache‣‣

Fast gradient sign<latexit sha1_base64="aIA66Q8v7rGLWWsGSLG1TN/jGWA=">AAACOXicbVBNSyNBEO1x1/V7jevRS2NYiCBhRgUFWVD34sGDglEhHUJNp5I09vQM3TWSMMzv2R/hb/CqePS2ePUP2Ik5rLoPCh7vVVFVL860chSGj8HUl6/T32Zm5+YXFpe+L1dWfly4NLcSGzLVqb2KwaFWBhukSONVZhGSWONlfP175F/eoHUqNec0zLCVQM+orpJAXmpXDkUHNQH/xQVmTunUcLEvCAdUuJ4peU0YiDW0B1wkQH0Jujgpa4L6SLA52BxubLQr1bAejsE/k2hCqmyC03blSXRSmSdoSGpwrhmFGbUKsKSkxnJe5A4zkNfQw6anBhJ0rWL8asl/eqXDu6n1ZYiP1X8nCkicGyax7xzd6z56I/F/XjOn7l6rUCbLCY18W9TNNaeUj3LjHWVRkh56AtIqfyuXfbAgyaf7bkuclD6T6GMCn8nFVj3arm+d7VQPjibpzLI1ts5qLGK77IAds1PWYJL9YXfsnj0Et8FT8Dd4fmudCiYzq+wdgpdXvrmtaA==</latexit>= ✏ sgn(rx L(✓, x, y))=✏Fast gradient<latexit sha1_base64="svNFd2U8JGq8Dn8Qw7H/kmXryEU=">AAACaXichVFNa9tAFFypzUfTtHHaS2kvS03BgWAkJ5BAKYT00kMPCcRJwGvE0/rJXrJaKbtPIUbR7yw99pzfUOja8SFf0IGFYWYe7zGbllo5iqLfQfji5dLyyuqrtdfrb95utDbfnbqishL7stCFPU/BoVYG+6RI43lpEfJU41l68X3mn12hdaowJzQtcZjD2KhMSSAvJa1LMUJNwL9xgaVTujBcfOVCY0YdkVmQtTCQakiuuciBJhJ0/bPpCJogwfb19nSrqcXN/zLiJuk1wqrxhLaSVjvqRnPwpyRekDZb4Chp/RGjQlY5GpIanBvEUUnDGiwpqbFZE5XDEuQFjHHgqYEc3bCeV9PwL14Z8ayw/hnic/X+RA25c9M89cnZ6e6xNxOf8wYVZfvDWpmyIjTyblFWaU4Fn/XMR8qiJD31BKRV/lYuJ+ALJf8bD7akeeM7iR838JSc9rrxTrd3vNs+OFy0s8o+sc+sw2K2xw7YD3bE+kyyX+xvsBysBLfhZvgh/HgXDYPFzHv2AGH7H7Jdu24=</latexit>✓rx L(✓, x, y)
krx L(✓, x, y)k2◆

Iterative approac

max L(✓, x + , y)k kp<latexit sha1_base64="5tEj5L4J++sZ+a1PKxOpJ6l99bc=">AAACO3icbVDJSgNBEO1x34169NIYBEUNMyroMejFgwcFo0ImDDU9FdPYs9BdI4ZJPsiP8Bu86lG8iVfvdpaDW0HD471XVV0vzJQ05Lovzsjo2PjE5NT0zOzc/MJiaWn50qS5FlgTqUr1dQgGlUywRpIUXmcaIQ4VXoW3xz396g61kWlyQe0MGzHcJLIpBZClgtKxH8N94EeoCLjF1BKgitPuhk8tJNjm91sDcZu3N/kO95WdHVlrZ0D7nSALSmW34vaL/wXeEJTZsM6C0qsfpSKPMSGhwJi652bUKECTFAq7M35uMANxCzdYtzCBGE2j6B/b5euWiXgz1fYlxPvs944CYmPacWidvXPMb61H/qfVc2oeNgqZZDlhIgaLmrnilPJecjySGgWptgUgtLR/5aIFGgTZfH9sCeOuzcT7ncBfcLlb8fYqu+f75erRMJ0ptsrW2Abz2AGrshN2xmpMsAf2xJ7Zi/PovDnvzsfAOuIMe1bYj3I+vwDp9K3w</latexit>

Target speci c optimization

min L(✓, x + , y 0 ) + k kp
<latexit sha1_base64="N9NUpAjw2gXhqoeuLyuzgPmIBk8=">AAACPHicbVBNSxxBEO3R+P21MUcvjYuoKMuMCnqUmIOHHAy4KuwsQ01Prdtsd8/QXSNZxv1D/oj8Bq/mFvAWcs05vR+H+FHQ8HjvVVXXSwslHYXhUzA1/WFmdm5+YXFpeWV1rfZx/crlpRXYFLnK7U0KDpU02CRJCm8Ki6BThddp72yoX9+hdTI3l9QvsK3h1siOFECeSmpfYi1NEmeoCHisgboCVPV1sBNTFwn2+fe9sbjP+9u7fI/Hyg/PvPd+zMf3SZHU6mEjHBV/C6IJqLNJXSS1X3GWi1KjIaHAuVYUFtSuwJIUCgeLcemwANGDW2x5aECja1ejawd8yzMZ7+TWP0N8xP7fUYF2rq9T7xze415rQ/I9rVVS56RdSVOUhEaMF3VKxSnnw+h4Ji0KUn0PQFjp/8pFFywI8gG/2JLqgc8kep3AW3B10IgOGwffjuqnnyfpzLMNtsl2WMSO2Sk7ZxesyQR7YI/sif0MfgTPwe/gz9g6FUx6PrEXFfz9B1ltrh0=</latexit>

Need to know model f✓<latexit sha1_base64="PAn6XdOHZomlIVjg/0fDZmIpnsE=">AAAB73icbVBNS8NAEJ3Ur1q/qh69LBbBU0mqoMeiF48V7Ae0oWy2m3bpZhN3J0IJ/RNePCji1b/jzX/jts1BWx8MPN6bYWZekEhh0HW/ncLa+sbmVnG7tLO7t39QPjxqmTjVjDdZLGPdCajhUijeRIGSdxLNaRRI3g7GtzO//cS1EbF6wEnC/YgOlQgFo2ilTtjv4Ygj7ZcrbtWdg6wSLycVyNHol796g5ilEVfIJDWm67kJ+hnVKJjk01IvNTyhbEyHvGupohE3fja/d0rOrDIgYaxtKSRz9fdERiNjJlFgOyOKI7PszcT/vG6K4bWfCZWkyBVbLApTSTAms+fJQGjOUE4soUwLeythI6opQxtRyYbgLb+8Slq1qndRrd1fVuo3eRxFOIFTOAcPrqAOd9CAJjCQ8Ayv8OY8Oi/Ou/OxaC04+cwx/IHz+QMe2ZAF</latexit>

 

 

h

s

fi

73

Adversarial Examples with Spatial Constraints

Optimizing Spatial Constraints(Handling Limits on Imperceptibility)
Perturbation/Noise Matrix

Adversarial Target Labeln
X
Lp norm (L-0, L-1, L-2, …) Loss FunctionminL(✓, xi + , y 0 ) + k kp
<latexit sha1_base64="g8rsC/tTW463Mg7OoenM9khHixc=">AAACS3icbVBNTxRBFOxZQQG/Fj166bAxYiCbGTTBCwnBiwcPGF0g2V4nb3resh26eybdbwybYX4YP8If4IlwwzM34sHej4OAL+mkUq8q1a+yUitPcfwraj1YWHz4aGl55fGTp8+et1dfHPiichJ7stCFO8rAo1YWe6RI41HpEEym8TA7+TjZH/5A51Vhv9G4xIGBY6uGSgIFKm1/FUbZVOSoCbjwlUlrtZM03y0XBmgkQdefm3VBIyTY5Kep2phpN/n4zVu+wYUOWXmwns14cZaWabsTd+Pp8PsgmYMOm89+2r4UeSErg5akBu/7SVzSoAZHSmpsVkTlsQR5AsfYD9CCQT+op8c3/HVgcj4sXHiW+JT911GD8X5ssqCcXOTv7ibk/3b9ioYfBrWyZUVo5SxoWGlOBZ80yXPlUJIeBwDSqfBXLkfgQFLo+1ZKZprQSXK3gfvgYKubvOtufXnf2d2bt7PEXrE1ts4Sts122Se2z3pMsnN2wa7Y7+hndB3dRH9m0lY097xkt6a1+Bf4f7NH</latexit>

min<latexit sha1_base64="5edA6qoMrg6Bw7350FOH4GUnya4=">AAACXXicbVFNTxRBEO0dQQERVz1w4NJxY8RANjNoohcTghcPkGDiAsn2OqnpqWU7dPdMumsMm2F+Ib+AE/EneNULvR8kAlbSyct7VfW6X2elVp7i+KoVPVpYfPxkaXnl6eqzteftFy+PfFE5iT1Z6MKdZOBRK4s9UqTxpHQIJtN4nJ19mejHP9F5VdjvNC5xYODUqqGSQIFK2yiMsqnIURNw4SuT1upz0vywXBigkQRd7zebgkZIsM3PU7XFD9JzIfOC+Gxqm4/fvuNbXOjgmoclF7cNM11cpGXa7sTdeFr8IUjmoMPmdZi2f4m8kJVBS1KD9/0kLmlQgyMlNTYrovJYgjyDU+wHaMGgH9TTOBr+JjA5HxYuHEt8yv47UYPxfmyy0Dl5o7+vTcj/af2Khp8GtbJlRWjlzGhYaU4Fn2TLc+VQkh4HANKpcFcuR+BAUviBOy6ZaUImyf0EHoKjnW7yvrvz7UNnd2+ezhLbYK/ZJkvYR7bLvrJD1mOSXbLf7A/727qOFqPVaG3WGrXmM6/YnYrWbwDe0LeS</latexit>

n
X
i=1

Subtle Poster
Camouflage Sticker

i=1

Optimizing Spatial Constraints(Handling
Limits on Imperceptibility)
013L(✓, xi + Mx · , y ) + kMx · kp

Subtle Poster
Camouflage Sticker
Mimic vandalism“Hide in the human
psyche”
74

Mimic vandalism“Hide in the human
psyche”14sifier, we adopt an iterative procedure to estimate the robustness (x0 ; f ). Specifically, at each iteration, f is linearizedaround the current point xi and the minimal perturbation of
the linearized classifier is computed as

DeepFoolTarg min kri k2 subject to f (xi ) + rf (xi ) ri = 0.
ri

(4)

;f
(x 0

Algorithm
for binary classifiers
0 iteration
The perturbation f (x)ri >ati of1 DeepFool
the algorithm
is com1: input: Image x, classifier f .
puted using the closedfform
solutionin(3),
andclassifiers
the next
Algorithm
1Eq.
DeepFool
2: output:
Perturbationr̂.for binary(x) > 0Initialize
x0Imagex,stops
0. when
1: input:x,i classifierf.
iterate xi+1 is updated. The3:4: algorithm
xi+1
2:
output:Perturbationr̂.
while sign(f (xi )) = sign(f (x0 )) do
3: Initialize
xf (x ) x, rf
i algorithm
0. ),
r (x)
f (x) < 0 changes signof the classifier. 5:The
forri DeepFool
(x
i
krf
(x
)k
4: while sign(f (x )) = sign(f (x )) do
6:xi+1
xi + r i ,
r
(x)
f
(x)
<
0
5:
r
rf
),
binary classifiers is summarized
ini Algorithm
1(xanda geoF
7:
i
+
1.
x
6:
x
x +r ,
8: endwhile
F illustration of the method
metricis
7: shown
i P
i + 1. in Figure 3.
x
9: returnr̂ while= i ri .
e 2: Adversarial examples for a linear binary classifier.
8: endP
Inpractice,theabove
algorithm
canoften
9: return r̂ =r . converge to a
Figure 2: Adversarial examples for a linear binary classifier.
onof thezero
en that thepointrobustness
f at point
x0 ,level(x0 ; fset
)2 , isF . In order to reach the other side
be seen fromthat therobustnessof f at point
x ,hyper(x ; f ) , isto the distancex
tothe
separating
affine0
oftoTtheclassification
boundary,equal
the distancefrom x to the separating
affine hyper- the final perturbation vector
F = {x : wIterativex + b = 0} optimization(Figure 2). The minimal
ofminimal
perturbations for linear classi ers
plane F = {x : w x + b = 0} (Figure 2). The
rbation toperturbation
changedecisionr̂ is themultipliedbycorresponds
a constanttoclassifier’schange the classifier’s
decisioncorresponds 1 + ⌘, with ⌘ ⌧ 1. In oure orthogonal
x0 ontoofFx. onto
It is Fgiven
to theprojection
orthogonalof
projection. It isbygiven byexperiments,
we
have
used
⌘ = 0.02.
losed-form
theformula:closed-form formula:
0

⇤

;f
(x 0

)

i

0

)

i+1

2

i 2
f (xi )
krf (xi )k22
i
i

0

i

0

i

‣

0

T

0

based oclassifie
Therefothe clasii⇤function
indicate
tained fof Rn .

i20where fk th clas
proposeize it to

0

3.1.
Af
(Moosavi-Dezfooli et al., DeepFool: A Simple and Accurate Method to Fool Deep Neural Networks, CVPR 2016)(3)

subject
to sign(f (x0 + r)) 6= sign(f (x0 ))subject to sign(f (x0 + r)) 6= sign(f (x0 ))

(3)

Let f

75
fi

r (x ) := arg min krk2

0 krk2⇤ (x0 ) := arg⇤min

Application to Transfer Learning
Transfer Learning via Fine-Tuning76Black-box Adversarial Reprogramming (BAR)‣ Transfer learning: from netuning to black-box settin‣ Cross domain and data limited transfer learning
Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models

Figure1. Schematic
overviewof our proposed
black-box
adversarial reprogramming(BAR) method.
(Y. Tsai et al.,Transfer
Learning
without
Knowing:
Reprogramming
Black-box
Machine Learning
Models with Scarce Data and Limited Resources, ICML 2020)gfi

white-box transfer learning methods and hence does not 77 the power of well-trained but access-limited ML models
address Question (ii).
for transfer learning. In particular, we focus on adversarial

Seyed-Mohsen Moosavi-Dezfooli∗ †Alhussein Fawzi∗ †seyed.moosavi@epfl.chhussein.fawzi@gmail.com

Universal Adversarial Perturbations
Omar Fawzi‡

Pascal Frossard†

omar.fawzi@ens-lyon.fr

pascal.frossard@epfl.ch

Abstract
Universal perturbationt
Given a state-of-the-art deep neural network classifier,
Data
sampl
we show
the existence of a universal (image-agnostic) andvery small perturbation vector that causes natural images
to be misclassified with high probability. We propose a sysModel
tematic algorithm for computing universal perturbations,and show that state-of-the-art deep neural networks are
Input
highlytransformation
vulnerable to such perturbations, albeit being quasiimperceptible to the human eye. We further empirically anEnsemble
methods
alyze these universal
perturbations and show, in particular,

‣
‣
‣
‣

that they generalize very well across neural networks. The
surprising existence of universal perturbations reveals important geometric correlations among the high-dimensional
decision boundary of classifiers. It further outlines poten∆v 2tial security breaches
withthe existence of single directionsv
in the input space that adversaries canRpossibly
exploit to3
1∆vnatural
break a classifier on mostimages.
1

1. Introduction

Face powder

Joystick

Chihuahua

Jay

GrilleLabradorThresher

Algorithm 1 Computation of universal perturbations

Labrador
Flagpole Data points X,
1: input:classifierk̂, desired "p nothe perturbation ξ, desired accuracy on perturbed
ples δ.
Tibetan mastiff
Tibetan mastiff
2: output: Universal perturbation vector v.
3: Initialize v ← 0.
4: whileLycaenid
Err(Xv ) ≤Brabancon
1 − δgriffon
do5:
for each datapoint xi ∈ X do
6:
if k̂(xi + v) = k̂(xi ) thenLabrador
Balloon7:
Compute the minimal perturbation
sends xi + v to the decision boundary:

Can we find a single small image perturbation that fools
a state-of-the-art deepneural network classifier on all natx1,2,3
ural images? We show in this paper the existence of suchquasi-imperceptible universal perturbationthat lead
Rvectors2
to misclassify natural images with high probability. Specifically, by adding such a quasi-imperceptible perturbation
to natural images, the label estimated by the deep neural network is changed with high probability (see Fig. 1).
R∆vi ← arg min !r!2 s.t. k̂(xi + v + r) '= k̂Such perturbations are 1dubbed universal, as they are imager
agnostic. The existence of these perturbations is problemFigure 1: When added to a natural image, a universal peratic when the classifier is deployed in real-world (and posFigure 2: Schematic
representationof thebeproposed
algo- turbation
image to beby the8: image causes theUpdatethemisclassified
perturbation:
sibly hostile) environments,etas al.,
they can
exploitedAdversarial
by ad(Moosavi-Dezfooli
Universal
Perturbations,
CVPR
2017)
rithm used toversaries
computeuniversal
perturbations.
In this il- deep neural network with high probability. Left images:
to breakthe classifier.
Indeed, the perturbation
Original natural images. The labels are shown on top of78
v ←perturbation.
Pp,ξ (v + Right∆vi ).
lustration, data ∗points
x
,
x
and
x
are
super-imposed,
and
1
2
3
The first two authors contributed equally to this work.
each arrow. Central image: Universal
o

†

s

e

Whiptail lizardsChihuahua

Border terrierBlack-box Attacks

Is That It?

Microsoft Azure (Language Services)Zero-query attac

Google Cloud Vision API

OutputInput !

‣ Random perturbatio‣ Difference of mean
‣ Transferability based attac

Parameters "Training

Deployment

Inference

Zero knowledge about modeland training data

k

n

s

k

k

79
n

Prediction

difference gradient‣ Finiteestimatio
reduced gradient‣ Query
estimation

Input

Query based attac

AI/ML
Attacking
AI/MLsystems
with
Limited
Access:Our
ZOO
AttackKey Techniquesystem• Now𝐹you might think your system is robust to adversarial examples….
Black-box system is also vulnerable to adversarial attac

‣
Input
AI/ML
: gradientestimationfromsystemoutputs
insteadofback-prop
Gradient
estimationfromsystemoutputs
insteadof back-prop‣
systemPrediction

𝐹

𝜕𝑓 𝐱
𝜕𝑥

𝑓 x

𝛽𝐞

𝑓 x

• Key technique: gradient estimation from𝑔system≔ outputs instead of back-prop

𝐹 x

𝛽

𝛽𝐞

2𝛽

𝜕𝑓 𝐱
𝜕𝑥

𝑓 x

𝛽𝐞

𝑓 x

cking AI/ML systems with Limited Access: Our ZOO Attack
2𝛽
𝑔 ≔

𝛽𝐞

𝛽𝐞

𝐹 Models,
x 𝛽𝐞P.-Y. Chen*, H. Zhang*, Y. Sharma, J. Yi, and C.-J. Hsieh, AI-Security 2017
sed Black-box Attacks to Deep Neural𝛽Networks without Training SubstituteIBM
Research AI with Limited Access: Our ZOO Attack
Attacking
AI/MLsystems
w you might think your system is robust to adversarial examples….

ZOO: Zeroth Order Optimization based Black-box Attacks to Deep Neural Networks without Training Substitute Models, P.-Y. Chen*, H. Zhang*, Y. Sharma, J. Yi, and C.-J. Hsieh, AI-Security 2017

IBM Research AI• Now you might think your system is robust to adversarial examples….

put

AI/ML
Input
system AI/ML
systemiction Prediction 𝐹
𝐹technique:gradient
estimationfrom system
outputs
insteadof back-prop• Keytechnique:gradient estimationfrom system
outputs
insteadof back-propk

80

Zero-Order Optimization
gradient using function value coordinate by‣ Estimate
coordinate(Chen et al., 2017)

Zeroth-Order (ZO) Optimization
SGD (first order)

ZO-SGD𝒙

Convergence rate 𝐸 ∇𝐹 𝒙

𝒙

𝑂 1/ 𝑇

Convergence rate 𝐸 ∇𝐹 𝒙

𝑂

𝑑/ 𝑇

[Duchi, et al., T-IT’1 ]d is # of variables

T is # of iterations

Question: Better gradient estimate & ZO
method with better convergence rate?
IBM Research AI(S. Ghadimi & G. Lan, Stochastic First- and Zeroth-Order Methods for Nonconvex Stochastic
Programming, SIAM J. Optim. 2013)81

•

Given d-dimensional vector x, we can•Finitedifference
gradient
estimation
gradientas
below
Query Based Attack•

difference gradient‣ Finiteestimatio‣

Given d-dimensional vector x, we can m
gradient as below• FGSMAn example of approximate FGS with finit
An example of approximate
with nite difference
xadv = x + ✏ · sign (FDx (`f (x, y), ))•An example of approximate FGS with fi

xadv

•
Query
reducedgradient
estimation= x + ✏ · sign (FD (` (x, y), ))xf• Random grouping
Similar attack success rate
• withPCA• Query reduced gradient estimation‣ white-box
attack••FD-xent and FD-logit are
overlapped

Random grouping
PCA

Effectiveness of various single step black-box attacks on MNIST. The y-axis represents thevariation in adversarial success as ✏ increases.

 

n

fi

82

Finite Differences method outperform other black-box attacks and achieves similar
attach success rate with the white-box attack

AutoZOOMThe Thirty-Third AAAI Conference on Artificial Intelligence (AAAI-19)‣ Scaled random full gradient estimation for ef cient quer

AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method
for Attacking Black-Box Neural Networks
1⇤

i)

q

Chun-Chen
Ting, Pin-Yu Chen, Sijia Liu, 1∂f(x)f(x + βu)
− f(x) 3 Tu, Paishun
43
5,6
Huan Zhang, ⋅ u,
Jinfeng
Yi,
Cho-Jui Hsieh,
Shin-Ming
Cheng
g=
=b⋅
u
is
a
unit-lenght
vector
ii)
ḡ
=
gj
1
University of Michigan, Ann Arbor, USA
∑
∂x
β
q
2
MIT-IBM Watson AI Lab, IBM Research
j=1
1⇤

2⇤

2

University of California, Los Angeles, USA
4
JD AI Research, Beijing, China
5
National Taiwan University of Science and Technology, Taiwan
6
Academia Sinica, Taiwan
3‣ Autoencoder for dimensional reduction of perturbations
AbstractyfiRecent studies have shown that adversarial examples in stateof-the-art image classifiers trained by deep neural networks(DNN) can be easily generated when the target model is transparent to an attacker, known as the white-box setting. However,when attacking a deployed machine learning service, one canonly acquire the input-output correspondences of the target
model; this is the so-called black-box attack setting. The major
drawback of existing black-box attacks is the need for excessive model queries, which may give a false sense of model
robustness due to inefficient query designs. To bridge this gap,
we propose a generic framework for query-efficient blackure 2: Illustration of attack dimension reduction
througha “decoder”
in AutoZOOM
forisimproving
inbox attacks.
Our framework,
AutoZOOM,
which
short for query efficiency
Figure 1:
AutoZOOM significantly reduces the number ock-box attacks. The decoder has two modes:(i)An
autoencoder(AE)
trainedonunlabelednaturalimagesthataredifferent
Autoencoder-based Zeroth Order Optimization Method, has
queries required to generate a successful adversarial Bagem the attacked images and training data; (ii)asimplebilinearimage
resizer(BiLIN)thatisappliedchannel-wisetoextrapolate
two novel building blocks towards efficient black-box attacks:
from the black-box Inception-v3 model.
-dimensional feature to the original image(i)dimension(width⇥ height).
Inestimation
the latter mode,notoadditional
training isimage
required.
anadaptive
randomgradient
strategybalance(Chun-Chen Tu et al., AutoZOOM: Autoencoder-Based Zeroth Order Optimization Method for Attacking Black-Box Neural Networks, AAAI-19)
query counts and distortion, and (ii) an autoencoder that iseither trained offline with unlabeled data or a bilinear resizing
h as (Chen et al. 2017; Nitin Bhagoji et al. 2018), arepost-success stage by performing finer gradient estimation.
also offer an ideal environment for testing malicious attempt
operation for attack acceleration.
Experimental 83
results suggest
query-efficient since they exploit coordinate-wise graEvenworse,
the risks can be amplified when these service
that, by applying AutoZOOM
a state-of-the-art
black-box
4. In the toexperiments,
we alsofind that AutoZOOM
withaSummary of Attack Methods
White-Box

Auto-PGD (Croce and Hein, 2020
Wasserstein Attack (Wong et al., 2020
Targeted Universal Adversarial Perturbations (Hirano
and Takemoto, 2019
Projected Gradient Descent (PGD) (Madry et al., 2017
Elastic Net (Chen et al., 2017
Universal Perturbation (Moosavi-Dezfooli et al. 2016
Feature Adversaries (Sabour et al. 2016
DeepFool [Moosavi-Dezfooli et al., CVPR 2016
L-BFGS [Szegedy et al. ICLR 2014]
FGSM [Goodfellow et al. ICLR 2015]

Poisoning Attack
Adversarial Backdoor
Embedding (Tan andShokri, 2019)
Backdoor Attack (Gu, et.
al., 2017)

Evasion Attack

Poisoning Attack on
Support Vector Machines(SVM) (Biggio et al.,2013)

ZO-SVRG [Liu et. al. NeurIPS 2018]
ZO-NES [Ilyas et. al. ICML 2018
AutoZoom [Chen et al. AAAI 2019
ZO-signSGD [Liu et. al. ICLR 2019
ZO-Natural Gradient Descent [Zhao et. al. AAAI 2019
ZO-ADMM [Zhao et. al. ICCL 2019]
ZO-ADAM [Chen et. al. NeurIPS 2019]
ZO hard-label attack [Cheng et. al. ICLR 2019]
Sign-OPT [Cheng et. al. ICLR 2020
Square Attack (Andriushchenko et al., 2020)

Clean Label Feature
Collision Attack (Shafahi,
Huang et. al., 2018)Black-Box

)

]

)

]

 

)

 

)

 

 

 

]

]

]

)

]

)

)

84

Software of Attacks‣ https://github.com/bethgelab/foolbox‣ https://github.com/IBM/adversarial-robustness-toolbox‣ https://github.com/tensor ow/cleverhans‣ https://github.com/Trusted-AI/adversarial-robustness-toolbox/
wiki/ART-Attacks
Ian Goodfellow

fl85

Adversarial Defense
be defensed by weight regularization, dropout and‣ Cannotmodel ensembl‣ Two typedefense: Find adversarial examples without‣ Passivemodifying the model, special case of Anomaly Detectio
defense: Training a model that is robust to‣ Proactive
adversarial examples

n

e

s

86

Passive Defense

87

Feature Squeezing‣ Goal: Detect adversarial examples‣ Feature Squeezer: coalesces similar samples into a single one(Xu et al. NDSS 2018)
Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks
 88Feature Squeezing‣ Approac‣ Hypothesis‣ Feature squeezing barely change legitimate inpu
‣ Destruct adversarial perturbations

t

h

89

Passive Defense
Randomization at Inference Phase‣ Randomization

https://arxiv.org/abs/1711.01991
90

eepCloak: Masking Deep Neural Network Model
Robustness
AgainstAdversarial
Samples
mask
layer
in
a
DNN
model
DeepCloak: Masking DNN• An adversarial sample x^′ can be deBined as:​x↑′ =x+Δx, |Δx|<ϵF(x)≠F(​x↑′ )• An adversarial sample must be similar to its seed sample.
• Adversarial samples can greatly reduce the effectiveness of deep

The algorithm to learn the mas

Ji Gao, Beilun Wang, Zeming Lin, Weilin Xu, Yanjun Qi
featuresin DNNs
make modelofvulnerabl
Departmentof Computer
Science,
University
Virginia‣ Motivation: Unnecessary
Example of the vulnerability:

he linear layer handling
Method:.
learning models.

• A Recent study [1] shows that extra unnecessary features extracted
by the machine classiBier are a vulnerability to adversarial samples.

Truth, e.g.,
by human eye

Machine Learning model(Extracted an extra feature)

Experimeep neuralnerable.

No retraining neede
Key: Insert a mask layer in a DNN model
On Res-net15
right before the linear layer handling
We’ve renamed our papey removing
DeepMask to DeepCloclassiBication.
Idea: Insert a mask layer in DNN model to remove unnecessary
Model:
features To use the mask: (())
ve unnecessary=*(+()))‣ask:

e model.

Model:(())=*(+()))

ated samples to fool DNN

d as:(Gao et al. DeepCloak: Masking Deep Neural Network Models for Robustness Against
Get 10% incr
Adversarial Samples, ICLR 2017 workshop)o its seed sample.
ehe effectiveness of deep

The algorithm to learn
the mask:
91

Proactive
Adversarial Training
AdversarialDefense:
training
1. Choose a set of perturbations: e.g., noise of small ℓ∞ norm:
2. For each example, find an adversarial example:

3. Train the model on
4. Repeat until convergence

Training on Adversarial Examples
Szegedy et al., 2014
Madry et al., 2017

100
Test misclassification rate

Train=Clean, Test=Clean
Train=Clean, Test=Adv
Adversarial Training and Robustness for Multiple Perturbations
Train=Adv, Test=Clean°1
10
Train=Adv, Test=Adv

10°2
0

50

100

150

200

250

300

Training time (epochs)

92(Goodfellow 2016)

Adversarial Machine Learning
Traditional ML:
optimization

Adversarial ML
game theory

Equilibrium

Minimum
One player
one cost

More than one player
more than one cost

,

:

,

93

Standard vs. Adversarial Training‣ Standard trainin
min E(x,y)⇠D L(✓, x, y)✓<latexit sha1_base64="nWKvdnzh9EApVTggbL3wZKrHzL8=">AAACQ3icbVDLSgMxFM34tr6qLt0Ei1ChlJkq6LL4AEEXClaFThkyaWpDk8yQ3BGHYT7Kj/AbxJ2u3YlbwfSBaPVA4Nxz7+HenDAW3IDrPjsTk1PTM7Nz84WFxaXlleLq2pWJEk1Zg0Yi0jchMUxwxRrAQbCbWDMiQ8Guw95hv399x7ThkbqENGYtSW4V73BKwEpB8dSXXAU+dBkQ7EsC3TDMjvMgK99X0m3fcDlUKRHZUZ5jv/JdnuW4PDRW8H0Fp9tBseRW3QHwX+KNSAmNcB4UX/12RBPJFFBBjGl6bgytjGjgVLC84CeGxYT2yC1rWqqIZKaVDT6d4y2rtHEn0vYpwAP1pyMj0phUhnayf7EZ7/XF/3rNBDr7rYyrOAGm6HBRJxEYItxPELe5ZhREagmhmttbMe0STSjYnH9tCWVuM/HGE/hLrmpVb6dau9gt1Q9G6cyhDbSJyshDe6iOTtA5aiCKHtATekGvzqPz5rw7H8PRCWfkWUe/4Hx+AeBXsVU=</latexit>‣ Adversarial example

Model
Parameter

Loss

Input

Label

max L(✓, x + , y) s.t. k kp  ✏<latexit sha1_base64="31c6EWNJxCfxGVtOs3RUTlGA4ew=">AAACVnicbVA9TxtBEF1fQvjIlyFlmhUoChHodEcKkNygUCQFBUgYkLyONbce4xV7H+zOIVvHtfyZ/I2U/AAoaWmhRuz5KAJkpJXevDdPM/uiTCtLQXDV8F69nnozPTM79/bd+w8fm/ML+zbNjcS2THVqDiOwqFWCbVKk8TAzCHGk8SA63qr0g1M0VqXJHo0z7MZwlKiBkkCO6jV/ixhGvUL0UROU3HU0lKCL7XJZ0BAJVvlopVZX+fgbFy3R4oJwRIX1yefO4vqzekKc9TIuNJ5wgZlVulqwFPjBpPhLED6Cpc3Wzz/nK18vdnrNa9FPZR5jQlKDtZ0wyKhbgCElNZZzIreYgTyGI+w4mECMtltMcij5F8f0+SA17iXEJ+y/jgJia8dx5Carf9rnWkX+T+vkNNjoFirJcsJE1osGueaU8ipU3lcGJemxAyCNcrdyOQQDklz0T7ZEcekyCZ8n8BLsr/nhd39t14Xzg9U1wz6zRbbMQrbONtkvtsPaTLK/7IbdsrvGZePem/Km61Gv8ej5xJ6U13wAIcS4sQ==</latexit>

Loss

Adversaria
Example

Tru
Label

Kee
Inperceptible‣ Adversarial training as aminimax problem
min E(x,y)⇠D✓<latexit sha1_base64="kNzSohNsKsYPbw6zb52PBj3BQtk=">AAACfnicbVHLatwwFJXdV5q+pukyXYgOLRMymdppoYFuQtpCF12k0EkCI2NkzfWMiCQb6bqMMd72H7PMvh8ReWYozeOC4HDOuQ+OslJJh1F0EYT37j94+Gjj8eaTp8+ev+i93DpxRWUFjEWhCnuWcQdKGhijRAVnpQWuMwWn2fmXTj/9DdbJwvzCuoRE85mRuRQcPZX2/jAtTcpwDsgp0xznWdZ8a9NmsBjWO8xJvSQFV83XtqXsM2UKcpz8Y3+0g1X3kC6GtN6hu92YRdqwKSjk7R3Gxe5K6+zMytkck7TXj0bRsuhtEK9Bn6zrOO1dsmkhKg0GheLOTeKoxKThFqVQ0G6yykHJxTmfwcRDwzW4pFnm1dK3npnSvLD+GaRL9v+Ohmvnap15Z3e9u6l15F3apML8IGmkKSsEI1aL8kpRLGgXPp1KCwJV7QEXVvpbqZhzywX6L7q2JdOtzyS+mcBtcLI/ij+M9n9+7B8erdPZINvkDRmQmHwih+Q7OSZjIsjfYCvYDl6HJHwX7oXvV9YwWPe8ItcqPLgCTFHB1A==</latexit>L(✓, x, y) + max L(✓, x + , y)Optimize Defense

Optimize Attack

s

g

l

 

p

e

94

Adversarial Training
as a minimax proble‣ Adversarial training
min E(x,y)⇠D✓<latexit sha1_base64="cOhyDbx+fbbZuuoHYO4bG2g2JTk=">AAACrnicbVFda9RAFJ3Er1o/utVHfRhchC1dQlIFC76UqiDYhwrutrATwmRysxk6k6QzN7Ihzd/yv/joPzHZLGI/LgwczjmXc+feuFTSou//dtx79x88fLT1ePvJ02fPd0a7L+a2qIyAmShUYc5jbkHJHGYoUcF5aYDrWMFZfPGp189+grGyyH9gXUKo+TKXqRQcOyoa/WJa5hHDDJBTpjlmcdx8aaNmsprWe8xKvSYFV83ntqXsI2UKUlzQf/RJOxnap3Q1pfUe3e+1VdSwBBTy9g7jan/QejszcplhSNllxRPKEFbYWA+9to+6GnzsKir73EvKoLRS9YOPfc9fF70Ngg0Yk02dRqM/LClEpSFHobi1i8AvMWy4QSkUtNusslByccGXsOhgzjXYsFnvt6VvOyahaWG6lyNds/93NFxbW+u4c/aftTe1nrxLW1SYHoaNzMsKIRdDUFopigXtj0UTaUCgqjvAhZHdrFRk3HCB3UmvpcS67XYS3NzAbTA/8IJ33sH39+Oj4812tsgr8oZMSEA+kCPylZySGRHOa+fY+eacuL47d0M3Gqyus+l5Sa6Vm/0FsmPUSA==</latexit>L(✓, x, y) + max L(✓, x + , y)Optimize Defense‣ Be simpli ed as
min E(x,y)⇠D✓<latexit sha1_base64="O6suxlOFtxfgGYjzqoC+Nh+znLc=">AAACknicbVFNb9QwEHVCgbZ8LR83LhYr0CJWaVKQQHAptEgceigS21ZaR5HjTHat2klqT9Cu0vyR/jOO/BOczQrRlpEsPb1582Y8k1ZKWgzDX55/a+P2nbubW9v37j94+Gjw+MmxLWsjYCJKVZrTlFtQsoAJSlRwWhngOlVwkp7td/mTn2CsLIsfuKwg1nxWyFwKjo5KBpdMyyJhOAfklGmO8zRtvrZJM1qMl6+ZlXpFCq6ag7al7BNlCnKcdtpF0rAMFPL2r+awHfVe48WbPjemzsbI2Rxjys5rnlGGsMDGBhi0nd1Fr2MXSdV5n1MGlZWqG24YBuEq6E0QrcGQrOMoGfxmWSlqDQUKxa2dRmGFccMNSqGg3Wa1hYqLMz6DqYMF12DjZrXDlr50TEbz0rhXIF2x/1Y0XFu71KlTdp+113Md+b/ctMb8Q9zIoqoRCtE3ymtFsaTdQWgmDQhUSwe4MNLNSsWcGy7Qne1Kl1S3bifR9Q3cBMe7QfQ22P3+brj3Zb2dTfKcvCAjEpH3ZI98I0dkQoRHvFfejhf6z/yP/md/v5f63rrmKbkS/uEfrcfJ/w==</latexit>

s.t. k kp  ✏Optimize AttackOuter Minimization

max L(✓, x + , y)s.t. k kp  ✏Inner Maximization

Active Learning or Data Augmentation or Regularization
m

fi

95

Adversarial Training
Inner maximization

Outer maximization

max L(✓, x + , y) s.t. k kp  ✏min E(x,y)⇠D L(✓, x + 0 , y)search (lower bound‣ Localon objective‣ Adversarial trainin<latexit sha1_base64="31c6EWNJxCfxGVtOs3RUTlGA4ew=">AAACVnicbVA9TxtBEF1fQvjIlyFlmhUoChHodEcKkNygUCQFBUgYkLyONbce4xV7H+zOIVvHtfyZ/I2U/AAoaWmhRuz5KAJkpJXevDdPM/uiTCtLQXDV8F69nnozPTM79/bd+w8fm/ML+zbNjcS2THVqDiOwqFWCbVKk8TAzCHGk8SA63qr0g1M0VqXJHo0z7MZwlKiBkkCO6jV/ixhGvUL0UROU3HU0lKCL7XJZ0BAJVvlopVZX+fgbFy3R4oJwRIX1yefO4vqzekKc9TIuNJ5wgZlVulqwFPjBpPhLED6Cpc3Wzz/nK18vdnrNa9FPZR5jQlKDtZ0wyKhbgCElNZZzIreYgTyGI+w4mECMtltMcij5F8f0+SA17iXEJ+y/jgJia8dx5Carf9rnWkX+T+vkNNjoFirJcsJE1osGueaU8ipU3lcGJemxAyCNcrdyOQQDklz0T7ZEcekyCZ8n8BLsr/nhd39t14Xzg9U1wz6zRbbMQrbONtkvtsPaTLK/7IbdsrvGZePem/Km61Gv8ej5xJ6U13wAIcS4sQ==</latexit><latexit sha1_base64="cnEpSj/JgvzCu/mHkLbuoAQRbYg=">AAACTHicbVBNSxxBEO3ZGKPmw01y9NK4hKxkWWZMIIFcJBrw4MGAq8LOMtT01rqN3T1Dd01wGOaP5Ufk7k2PevYWBHs/CFHzoOHVq3pU9UtzJR2F4XnQeLLwdPHZ0vLK8xcvX602X785dFlhBfZEpjJ7nIJDJQ32SJLC49wi6FThUXq6Pekf/UTrZGYOqMxxoOHEyJEUQF5KmgexliaJaYwEPNZA4zStvtdJ1T7rlBuxk3oqClDVTl3z+Cv/W+/V7Zmvw8/4h3iIiuB9h5cbPGm2wm44BX9MojlpsTn2k+ZlPMxEodGQUOBcPwpzGlRgSQqF9UpcOMxBnMIJ9j01oNENqunva/7OK0M+yqx/hvhU/ddRgXau1KmfnJzuHvYm4v96/YJGXwaVNHlBaMRs0ahQnDI+iZIPpUVBqvQEhJX+Vi7GYEGQD/zellTXPpPoYQKPyeFmN/rY3fzxqbX1bZ7OEltj66zNIvaZbbFdts96TLBf7IJdsevgd3AT/AluZ6ONYO55y+6hsXgH0Dqzlw==</latexit>✓

optimization‣ Combinatorial(exactly solve objective
relaxation (upper‣ Convex
bound on objective)‣ Provably rousting training

)

g

)

96

Adversarial Robustness is Not Free

ustnessFree‣ StateisOf Not
The Art in ` -Robustness

Optimization during training more dif cult and models need tobe large
1
Robustoptimization
as in [Madry,
Schmidt, Tsipras, Vladu, 2017]:
More
training
data
mightMakelov,
be require

Adversarial
Robustness
is
Not
Free
‣
training more difficult→ Optimization during training more difficultand models need to be larger

eed to be larger→ More training data might be required→ More training
data
might
be
required−"+"[Schmidt Santurkar Tsipras Talwar M 2018](Schmidt et al., Adversarially Robust Generalization Requires More Data, NeurIPS 2018)[Schmidt Santurkar Tsipras Talwar M 2018]‣ Might need to lose on “standard” measures of performance→ Might need to lose on “standard” measures of performance[Tsipras Santurkar Engstrom Turner M 2018] (Also see: [Bubeck Price Razenshteyn 2018])(Tsipras et al. 2018)

fi

d

r

97

But There Are (Unexpected) Bene ts
representation learned by robust model is more
‣ The
interpretabl‣ Align better to human perception

Loss gradient w.r.t. input(Tsipras et al. Robustness may be at odds with accuracy, NeurIPS 2018)

fi

e

98

Taxonomy of Adversarial ML

99

How to Evaluate Adversarial Robustness?
Game-based approac‣ Specify a set of players (attacks and defenses
the performance against each attacker-defender‣ Benchmark
pai‣ No guarantee on unseen threats and future attack
Veri cation-based approac‣ Attack-independent: does not use attacks for evaluatio
provide robustness certi cate: e.g., no attacks can alter‣ Can
the decision of the ML model if the attack strength is limite
veri cation is computationally impractical for large‣ Optimal
DNN
Zhang et al., Ef cient Neural Network Robustness Certi cation with General Activation Functions, NIPS 2018

d

n

s

)

fi

fi

h

h

fi

fi

fi

r

100

Veri cation: Lower Bounds on Robustness
Verification: lower bounds on robustness
Amount of
Perturbation

Vacuum Cleaner
label

Certified robustness
Lower bound on perturbation sothat any perturbations withingreen region cannot cause
misclassification

Vacuum Cleaner Attack
Shoe Shop Attack∆

Maximum Safe
Perturbation∆
Ostrich label

Shoe Shop
label

Lower
Bound

0

Decision boundary
Other Decision boundaries
IBM Research AI

fi

101

Decision boundaryEf cient Certi ed Bound with Activation Bounds
Efficient certified bound with activation bounds
Shoeshop[ 𝑙 ,𝑢 ]𝜀Trained CNN𝑥𝑥[ 𝑙 ,𝒖 ][ 𝑙 ,𝑢 ]Vacuum[ 𝑙 ,𝑢 ]𝜀[ 𝑙 ,𝑢 ][ 𝑙 ,𝑢 ][ 𝑙 ,𝒖 ]Ostrich[ 𝑙 ,𝑢 ][ 𝒍 ,𝑢 ]Image 𝑥

Input
Perturbation
Size 𝜀

Propagate
Bounds𝑙

Check ifrobust𝑢𝑙𝑢

IBM Research AI

Robustness Certi cate: Given a data input and a model, the top-1
prediction of the perturbed input will not be altered if the perturbation(e.g.
norm ball) is smaller than

𝜀

fi

fi

𝐿

fi

102

Timeline of Robustness Certi cation
Overview

DeepZ (Singh etal), Neurify (Wang etal) DeepPoly (Singh etal)

NeurIPS ‘1

POPL ‘1

AAAI

other teams

MIT-IBM
teams

ICLR ‘1

ICML ‘1

NeurIPS ‘1

CLEVER (Weng etal)

Fast-Lin (Weng etal)

CROWN (Zhang etal)

CNN-Cert

https://arxiv.org/abs/1801.10578 https://arxiv.org/abs/1804.09699 https://arxiv.org/abs/1811.00866 https://arxiv.org/abs/1811.12395

GeneralActivation

Robustness Estimation

MLP

ReLU

MLP

GeneralActivation

IBM Research AI

Robustness Certification

103
fi

CNN

CNN

GeneralActivation

Challenges‣ How to improve the state-of-the-art adversarial training method‣ Adversarial training is effective, but not scalable and ef cient‣ Tradeoff between accuracy and robustnes‣ Understand the nature of vulnerability of DNN‣ How to evaluate and certi cate model robustnes‣ Robustness to adaptive adversary, i.e. attack-agnostic defens‣ Need for human-like machine perception and understanding

s

e

fi

s

s

s

fi

104