ViewFool: Evaluating the Robustness ofVisual Recognition to Adversarial Viewpoints
Yinpeng Dong1,3, Shouwei Ruan2, Hang Su1, Caixin Kang2, Xingxing Wei2, Jun Zhu1,3
1 Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua University
2 Institute of AI, Beihang University

3 RealAI

dongyinpeng@mail.tsinghua.edu.cn

OOD Generaliza,on is Hard

Image Translation andRotation (Engstrom etal., 2019)

Image Corruptions (Hendryckset al., 2019)2

Viewpoint Changes

Chair: 78.66%

Keyboard: 47.80%

Street sign: 99.55%

Traffic light: 97.94%

Board: 63.50%

Mouse: 37.44%

Cinema: 58.45%

Canoe: 41.01%

Goal: find adversarial viewpoints that lead to wrong predictions of visual recognition
models in the physical world.
3

Real-world Application

Autonomous driving
fails to recognize
trucks/cars in rare
viewpoints to cause
traffic accidents.

4Challengesn How to model real-world 3D objects with high-fidelity? — NeRF
¨ Simple pipeline: 1) training a NeRF; 2) optimize viewpoint parameters

based on NeRF; 3) verify the vulnerability from the adversarial viewpointn How to mitigate the reality gap between a real object and its neuralrepresentation?

NeRF
rendering

Realobject

n How to control the real camera pose to precisely match theadversarial viewpoint?

5ViewFool: Problem Formulationn Let ! ≔ [$, &, ', Δ) , Δ* , Δ+ ] denote the transformation parameters ofthe camera.
n Let - ≔ .(!) denote the rendered image.
3D Object#('%Camera&)

""&"′

*!

!

2D Image#&Camera
Rotation'!′'&Δ, , Δ. , Δ/%%

'CameraTranslation%

Rendering6ViewFool: Op,miza,on Problem
n Learning a distribution of adversarial viewpoints:max ($ & ) * + & , $(&)

n Since max ($ &$ &) * + & ,-+/⋅1 2 &≤ max ) * + & , - , we add an&entropic regularizer into the objective as
1 2 &= −($ & [log 2 & ]7ViewFool: Optimization Algorithmn A transforma7on of Gaussian distribu7on! = # ⋅ tanh )+ + ; ) ∼ . /, 12 3 , # =n Our problem becomes!456 − !489
!456 + !489,; =22max >? ); /,1@3 A B C # ⋅ tanh ) + + , D − E ⋅ log I(# ⋅ tanh ) + +)/,1

n Reparameteriza7on trick: ) = / + 1L, where L ∼ .(M, 3).
n Gradient calcula7on: adopt the search gradients

8Real Image fromNatural Viewpoint

Visualization of Adversarial Viewpoints

Real Image fromAdversarial Viewpoint

Rendered Image fromAdversarial Viewpoint

Granny Smith: 83.71% Studio Couch : 97.56%

Warplane: 94.42%

Notebook: 72.50%

Traffic Light: 99.91%

Forklift: 97.27%

Street Sign: 95.00%

Tennis Ball: 97.81%

Wallet: 90.24%

Hatchet: 81.23%

Scale: 93.82%

Mortarboard: 91.97% Rocking Chair: 58.98%

Spatula: 40.98%

Tennis Ball: 73.09%

Wallet: 93.13%

Hatchet: 31.01%

Scale: 45.23%

Mortarboard: 86.48% Folding Chair: 36.51%

Street Sign: 11.74%

9

Cross-model Transferability

High transferability between different models!

10

Real-world Experiments
Real Image fromNatural Viewpoint

Rendered Image fromAdversarial Viewpoint

Chair: 78.66%

Board: 79.53%

Board: 63.50%

Board: 57.94%

Board: 61.17%

Board: 53.80%

Keyboard: 47.80%

Matchstick: 25.07%

Mouse: 31.27%

Mouse: 25.88%

Mouse: 35.32%

Mouse: 37.44%

Street sign: 99.55%

Mailbox: 18.84%

Cinema: 58.45%

Gas pump : 36.63%

Unicycle: 15.79%

Flagpole: 31.98%

Traffic light: 97.94%

Syringe: 10.03%

Canoe: 41.01%

Stretcher: 50.01%

Missile: 8.09%

Solar dish: 31.23%

Real Image from Adversarial Viewpoint

11

ImageNet-V Benchmark
n Transformer-basedmodels have better
viewpoint robustness;n A larger model within thesame architecture family
tends to perform better;n Adversarial training andexisting data
augmentation techniques
do not obtain good results.
12

Thanks for listening

Code is available at:
h.ps://github.com/Heathcliﬀ-saku/ViewFool_