Holistic Adversarial Robustness 

of AI Models

Pin-Yu Chen 

www.pinyuchen.com @pinyuchenTW

NIST AI Measurement and Evaluation Workshop

June 2021
The Deep Learning Revolution. What’s next?

IBM Research AI

Geoffrey Hinton

What’s

Next?

http://image-net.org/challenges/talks_2017/imagenet_ilsvrc2017_v1.0.pdf https://qz.com/1034972/the-data-that-changed-the-direction-of-ai-research-and-possibly-the-world/
AI revolution is coming, 

but Are We Prepared ?

❑ According to a recent Gartner report, 30% of

cyberattacks by 2022 will involve data

poisoning, model theft or adversarial examples.

❑ However, industry is underprepared. In a

survey of 28 organizations spanning small as

well as large organizations, 25 organizations did

not know how to secure their AI systems.
What is wrong with this AI model?

- This model is one of the BEST image classifier using neural networks

- Images and neural network models are NOT the only victims

IBM Research AI

EAD: Elastic-Net Attacks to Deep Neural Networks via Adversarial Examples, P.-Y. Chen*, Y. Sharma*, H. Zhang, J. Yi, and C-.J. Hsieh, AAAI 2018

The Great Adversarial Examples
Accuracy ≠ Adversarial Robustness

• Solely pursuing for high-accuracy AI model may get us in trouble...

IBM Research AI

Is Robustness the Cost of Accuracy? A Comprehensive Study on the Robustness of 18 Deep Image Classification Models, Dong Su*, Huan Zhang*, Hongge Chen, Jinfeng Yi, Pin-Yu Chen, and Yupeng Gao, ECCV 2018

Our benchmark 

on 18 ImageNet 

models reveals 

a tradeoff in 

accuracy and 

robustness Robustness

Accuracy
Why adversarial (worst-case) robustness matters?

➢ Prediction-evasive manipulation on a deployed AI model

1. Build trust in AI: address inconsistent perception and 

decision making between humans and machines & 

misinformation

2. Assess negative impacts in high-stakes, safety-critical tasks 

3. Understand limitation in current machine learning methods

4. Prevent loss in revenue and reputation

5. Ensure safe and responsible use in AI

IBM Research AI

Adversarial 

T-shirt
Holistic View of Adversarial Robustness

IBM Research AI

Data Model Inferenceg

Training Phase Deployment Phase

Attack Category / Attacker’s reach Data Model / Training Method Inference

Poisoning Attack [learning] X X*

Backdoor Attack [learning] X

Evasion Attack (Adversarial Example) [learning] X* X

Extraction Attack (Model Stealing, Membership inference) X

Model Injection [AI governance] X* X

*No access to model internal information in the black-box attack setting

AI/ML 

system

AI/ML 

system
Roadmap toward Holistic Adversarial Robustness

IBM Research AI

•In-house sensitivity and reliability tests for developed models

•Generate prediction-evasive examples (per user constraints)

•Customize to model deployment conditions (e.g. cloud APIs)

Attack

(Bug Finding)

•Detecting and mitigating potential adversarial threats

•Plug-and-play model patching for a given model

•Landscape exploration: model fix and cleaning

Defense

(Model Hardening)

•This model is certified to be attack-proof up to a certain level 

•Quantifiable metric for certified robustness

•AI standards, governance, and law regulation

Verification

(Model Certificate)

•Data augmentation

•Model reprogramming: data-efficient transfer learning

•Model watermarking

Applications to AI

(Model Boosting)

Data 

(Domain) 

- specific

Practical

Efficiency/Maximal 

utility/Compatibility

Model -

agnostic

Training Testing Monitoring

Penetration Testing
How to Define Levels of Robustness for AI?

• Lessons from autonomous driving systems 

IBM Research AI

https://blog.netapp.com/how-to-build-adata-pipeline-for-autonomous-driving/https://seekingalpha.com/article/4249591-techstocks-be-long-term-winners-in-autonomous-vehicles
My View of AI Robustness Levels and Evaluations 

Level 1 – Distribution Shifts

• Performance on non-adaptive (pre-generated) test sets

• Examples: Natural Corruption; Random Perturbation; Context Shifts

Level 2 – Single threat model

• Performance against optimized (worst-case) white-box adversarial 

examples based on one type of domain-specific data modifications

generated from a test dataset

• Examples: Gradient-based attacks using Lp norms

Level 3 – Multiple threat models

• Performance against white-box adversarial examples generated by a set 

of feasible threat models from a test dataset

• Examples: Ensemble attacks using Lp norms and semantic 

perturbations

Level 4 – Global (Universal) Robustness 

• Evaluation of global robustness (input-agnostic) instead of local 

robustness; Ultimate generalization (AGI); Fast adaptation 

• Examples: Unrestricted adversarial examples

IBM Research AI

1

st Party (model developer)

• Adaptive white-box attack

• Full system transparency

2

nd Party (model inspector)

• Non-adaptive white-box/gray-box attack

• Information obfuscation; Unknown implementation

3

rd Party (end user)

• Soft-label/hard-label/no-box black-box attack

• Target model is a black-box function with limited 

information feedback

Robustness Levels Robustness Evaluations
Making AI model Robust is truly ART

IBM Research AI