Attacking Optical Flow
Anurag Ranjan† Joel Janai†‡ Andreas Geiger†‡ Michael J. Black††
Max Planck Institute for Intelligent Systems‡
University of Tübingen{aranjan, jjanai, ageiger, black}@tue.mpg.de

Abstract
Deep neural nets achieve state-of-the-art performanceon the problem of optical flow estimation. Since opticalflow is used in several safety-critical applications like selfdriving cars, it is important to gain insights into the robustness of those techniques. Recently, it has been shown thatadversarial attacks easily fool deep neural networks to misclassify objects. The robustness of optical flow networks toadversarial attacks, however, has not been studied so far.
In this paper, we extend adversarial patch attacks to opticalflow networks and show that such attacks can compromise
their performance. We show that corrupting a small patchof less than 1% of the image size can significantly affect optical flow estimates. Our attacks lead to noisy flow estimatesthat extend significantly beyond the region of the attack, inmany cases even completely erasing the motion of objects
in the scene. While networks using an encoder-decoder architecture are very sensitive to these attacks, we found that
networks using a spatial pyramid architecture are less affected. We analyse the success and failure of attacking botharchitectures by visualizing their feature maps and comparing them to classical optical flow techniques which are robust to these attacks. We also demonstrate that such attacks
are practical by placing a printed pattern into real scenes.

1. Introduction
Optical flow refers to the apparent 2D motion of eachpixel in an image sequence. It is denoted by a vector
field (u, v) that corresponds to the displacement of eachpixel in the image plane. The classical formulation [12]seeks the optical flow (u, v) between two consecutive images I(x, y, t) and I(x + u, y + v, t + 1) in a sequence thatminimizes the brightness constancy (i.e. photometric) error
at each pixel, ρ(I(x, y, t) − I(x + u, y + v, t + 1)), for somerobust function ρ, subject to spatial coherence constraints
that regularize the solution [4].
The estimation of flow from pairs of images has a long

Figure 1. Overview. The first column shows the optical flow results using an encoder-decoder architecture, FlowNetC [8], a spatial pyramid architecture, SpyNet [20], and a classical method,LDOF [6]. In the second column, a small circular patch is added
to both frames at the same location and orientation (highlighted
by the red box for illustration purposes). SpyNet and LDOF arebarely affected by the patch. In contrast, FlowNetC is strongly
affected, even in regions far away from the patch.

history [12] and is used in many applications spanning
medicine, special effects, video analysis, action recognition,navigation, and robotics to name a few. A large number ofmethods [3] have approached the problem and steadily improved the results on current benchmarks [7, 10]. However, classical optical flow methods typically require extensive hand engineering [26] and computationally intensive optimization. Recent methods [8, 13, 20] have therefore approached optical flow estimation using deep neuralnetworks. These methods typically run in real time andproduce results that are competitive with, or even surpass,classical optimization-based approaches. Given the performance of recent optical flow networks, they could become
an important component in applications such as self-drivingcars. The safety issues surrounding many of these systems
implies that the robustness of the algorithms must be wellunderstood. To date, there has not been any study to mea-sure the robustness of these networks to adversarial attacks.
Adversarial attacks on neural networks have beenshown to work on image classification networks [18]and, consequently, it is reasonable to ask how such attacks can affect optical flow networks. Consider the
optical flow (u, v) between two frames of an image
pair (It , It+1 ) computed using a network F such that(u, v) = F (It , It+1 ).
A successful adversarial attack would cause large changes in estimated opticalflow for small, unnoticeable, perturbations in the images. Specifically, the adversary seeks a perturbed image
pair (I˜t , I˜t+1 ) such that kIt − I˜t k0 + kIt+1 − I˜t+1 k0 < ,
where  is a small constant, and the estimated optical flow on the perturbed images is significantly worse,||F (It , It+1 ) − F (I˜t , I˜t+1 )|| > E, with a large constant E.
We are particularly interested in perturbations under the `0norm, since the objective is to perturb a very small numberof pixels in the original image.
In general, the perturbations on the image can be definedin several ways [11, 18, 25, 29]. Here, we focus on applying perturbations by pasting a small patch on the image
motivated by Brown et al. [5], who use such an approachto carry out targeted adversarial attacks on image classification. The patch attack has real-world significance andwe show that adversarial patches can compromise opticalflow networks if an engineered patch is printed and placed
in real-world scenes.
Optical flow networks can be classified into two types ofarchitectures – the encoder-decoder [8, 13] and spatial pyramid networks [20, 28]. We contrast the robustness of adversarial attacks under these two architecture types. Similar
to [5], we independently and jointly learn patches to attack
all networks. The learned patches compromise the encoderdecoder architectures while spatial pyramid networks show
more robust behaviour as shown in Figure 1.
In the automotive scenario, cameras for autonomous
driving are often behind the windscreen. Patch attacks canpotentially be accomplished by placing the patch on thewindshield of the car or placing it in the scene (e.g. on a traffic sign or other vehicle). Note that when the patch has zero
motion w.r.t. the camera, classical optical flow algorithms
estimate zero optical flow over the patch (LDOF in Fig. 1).
However, this engineered patch, even if it has no motion,
can cause the optical flow predictions from encoder-decoderarchitectures to be wildly wrong (FlowNetC in Fig. 1). In
such a scenario, the patch affects the estimated optical flow
over large areas in the image that are far away from thepatch. The patch in Figure 1 is 0.53% the size of the image and is barely noticeable, yet it affects the flow in about
50% of the image region.
For the encoder-decoder networks, the patches not only
influence the optical flow estimates in the area of the patch,but their influence extends to remote areas in the image. In

spatial pyramid architectures, in contrast, the patch causes
at most small degradations in the area of the patch. Classical approaches [6, 23] are even more robust to adversarial
patches. We propose a Zero-Flow test (Section 5) to analyse the causes of patch attacks, where we visualize the feature maps of the networks while attacking a uniform random
noise image without motion. Thereby, we identify three major problems of optical flow architectures. 1) Flow networks
are not spatially invariant, leading to spatially varying feature maps even without any motion. 2) Spatial pyramid networks produce large errors at coarse resolutions but are able
to recover. 3) Deconvolution layers lead to strong amplification of activations and checkerboard artifacts.
Our contributions are as follows. We extend adversarial
patch attacks to optical flow neural networks. We learn adversarial patches and show that these attacks can considerably affect the performance of optical flow networks basedon an encoder-decoder architecture. We show that spatial pyramid architectures, along with classical optical flow
methods, are more robust to patch attacks. We show thatsuch attacks are easy to implement in practice by simply
printing the patch and placing it in the scene. We also analyse the feature maps of these networks under attack to provide insight into their behaviour under attack. Code is available at http://flowattack.is.tue.mpg.de/.

2. Related Work
Optical Flow. The classical version of the optical flow
problem involves solving for a flow field that minimizes
the brightness constancy loss. A survey of classical optical flow methods is available in [3, 27]. These classicalmethods solve a complex, non-convex, optimization problem and are often slow. Recent deep learning methods
replace the optimization process, and instead directly predict optical flow using convolutional networks [8, 13, 20].
FlowNet [8] is the first work to regress optical flow by learning an end-to-end deep neural network. FlowNet is based onan encoder-decoder architecture with skip connections. Although FlowNet is much faster than classical methods [26],it is not as accurate. More recently, Ilg et al. [13] proposed
FlowNet2, which achieves state-of-the-art performance on
optical flow benchmarks. FlowNet2 is constructed by stacking multiple FlowNets together and fusing the output witha network specialized on small motions.
In contrast, motivated by the classical coarse-to-finemethods, SpyNet [20] splits the matching problem into simpler subproblems using coarse-to-fine estimation on an image pyramid. PWC-Net [28] extends this idea with a correlation layer to learn optical flow prediction; this gives stateof-the-art results on optical flow benchmarks.
Still other methods approach this problem from the unsupervised learning perspective [17, 31, 15, 21] by using
a neural network to minimize the photometric error un-

der certain constraints. These methods use both encoderdecoder architectures [17, 31] and spatial pyramid architectures [15].
While deep networks show impressive results on benchmark datasets, their robustness is not yet well understood.
Robustness, however, is critical for applications such as autonomous driving. Therefore, we investigate the robustnessof representative approaches of each category in this paper.
In particular, we test FlowNet [8], FlowNet2 [13], SpyNet[20], PWC-Net [28], and Back2Future [15] for their robustness against adversarial attacks.
Adversarial Attacks. Adversarial attacks seek small
perturbations of the input causing large errors in the estimation by a deep neural network. Attacking neural networks
using adversarial examples is a popular way to examine thereliability of networks for image classification [11, 18, 29].
The key to all such attacks is that the change to the image
should be minor, yet have a large influence on the output.
Adversarial examples typically involve small perturbations
to the image that are not noticeable to the human eye. The
adversaries are shown to work even when a single pixel isperturbed in the image [25].
Although these attacks reveal limitations of deep networks, they can not be easily replicated in real-world settings. For instance, it is rather difficult to change a scene
such that one pixel captured by a camera is perturbed ina specific way to fool the network. However, recent work
[16] demonstrates that adversarial examples can also workwhen printed out and shown to the network under different
illumination conditions. Athalye et al. [2] show that adversarial examples can be 3D printed and are misclassified
by networks at different scales and orientations. Sharif etal. [24] construct adversarial glasses to fool facial recognition systems. Etimov et al. [9] show that stop signs can be
misclassified by placing various stickers on top of them.
Recently proposed patch attacks [5] place a small engineered patch in the scene to fool the network into making
wrong decisions. Patch attacks are interesting because they
can be easily replicated in the real world, and work at several scales and orientations of the patch. This makes deep
networks vulnerable in real world applications. Therefore,
we focus our investigation of the robustness of deep flow
networks on these kinds of attacks.
To our knowledge, there has been no work on attacking optical flow networks. Consequently, we explore howsuch networks can be attacked using adversarial patches andanalyse the potential causes of such vulnerabilities.

3. Approach
Adversarial attacks are carried out by optimizing for a
perturbation that forces a network to output the wrong labels
compared with ground truth labels. For example, if (u, v)
represents the ground truth labels for inputs (It , It+1 ), a

perturbed input (I˜t , I˜t+1 ) would produce incorrect labels(ũ, ṽ). However, there are no optical flow datasets that havedense optical flow ground truth labels for natural images.
Most of the optical flow datasets are synthetic [7, 8, 22].
SlowFlow [14] provides real world data but is limited insize. The KITTI dataset [10] has sparse ground truth labels
and the annotations are limited to 200 training examples. To
address the problem of limited ground truth labels, we use
the predictions of the optical flow network as pseudo groundtruth. We optimize for the perturbation that maximizes theangle between the predicted flow vectors obtained using theoriginal images and the perturbed images respectively. Using the predictions instead of ground truth has the advantagethat a patch can be optimized using any unlabelled video.
This makes it easier to attack optical flow systems even in
the absense of the ground truth.
Consider an optical flow network F that computes the
optical flow between two-frames of an image sequence(It , It+1 ) of resolution H × W . Consider a small patch
p of resolution h × w that is pasted onto the image to perturb it. Let δ ∈ T be a transformation that can be applied
to the patch. These transformations in T can be a combination of rotations and scaling. We define the perturbation
A(I, p, δ, l) on the image I, that applies the transformations
δ to the patch p and pastes it at a location l ∈ L in the image. We apply the same perturbation to both frames in thesequence I˜t = A(It , p, δ, l) and I˜t+1 = A(It+1 , p, δ, l).
This means that the optical flow between the perturbed
frames is zero in the region containing the patch. In thereal world, this would correspond to a patch being stationary with respect to the camera. In our experiments, weshow that patches obtained with this assumption generalize well to the realistic scenario of moving patches whereI˜t = A(It , p, δt , lt ) and I˜t+1 = A(It+1 , p, δt+1 , lt+1 ) with
δt 6= δt+1 and lt 6= lt+1 .
Our goal is to learn a patch p that acts as an adversary toan optical flow network, F , and is invariant to location l ortransformations δ of the patch. The resulting patch can be
optimized usingp̂ = argmin E(It ,It+1 )∼I,l∼L,δ∼T
p(u, v) · (ũ, ṽ)(1)k(u, v)k · k(ũ, ṽ)kwith(u, v) = F (It , It+1 )(2)(ũ, ṽ) = F (A(It , p, δ, l), A(It+1 , p, δ, l))(3)where l is sampled over all the locations L in the image, andI is a set of 2 frame sequences from a video.
Equation (1) computes the cosine of the angle between
the optical flow estimated by the network for normal andperturbed images. Therefore, minimizing the loss is equivalent to finding those adversarial examples that reverse thedirection of optical flow estimated by the network.

methods and encoder-decoder vs. spatial pyramid architectures, thus providing an overall picture of optical flow methods under adversarial patch attacks. We contrast these optical flow methods in Table 1. First, we evaluate the effectof White-box attacks, where we learn a patch of a specificsize for each network by optimizing over that particular network. We then show Black-box attacks by creating a universal patch using two different networks and testing all the
optical flow methods on this patch in a real world scenario.

tC
Ne
w
o
Fl

F

et2
N
low

4.1. White-box Attacks
y
Sp

t
Ne

t
Ne
C
PW

Ba

u
2F
ck

re
tu

25x25

51x51

102x102

153x153

Figure 2. Adversarial Patches. Obtained for different optical flow
networks. The size is enlarged for visualization purposes.
Table 1. Optical Flow Methods. Contrasting different opticalflow methods with respect to the Type, Supervision (Super.), Network Architecture (Net.) - Encoder-Decoder (ED) vs. Spatial
Pyramid (SP) - and Number of Parameters (#Params.). Numberof parameters is denoted in Millions (M).

FlowNetC [8]FlowNet2 [13]SpyNet [20]PWC-Net [28]Back2Future [15]LDOF [6]Epic Flow [23]

Type
CNN
CNN
CNN
CNN
CNN
Classic
Classic

Super.
Yes
Yes
Yes
Yes
No
-

Net.
ED
ED
SP
SP
SP
-

# Params
39 M
162 M
1.2 M
8.75 M12.2 M-

4. Experiments
We evaluate the robustness of five optical networks toadversarial patch attacks – FlowNetC [8], FlowNet2 [13],SpyNet [20], PWC-Net [28] and Back2Future [15]. We
also evaluate the robustness of two very different classicalmethods – LDOF [6] and EpicFlow [23] which take image derivatives (LDOF) and sparse matches (EpicFlow) tocompute optical flow. These methods cover deep networks
vs. classical methods, supervised vs. unsupervised learning

We perform White-box attacks on each of the networks
independently. We learn a circular patch that is invariant
to its location in the image, scale and orientation. The
scale augmentations of the patch are kept within ±5% of
the original size, and rotations are varied within ±10◦ . For
each network, we learn patches of four different sizes. The
patch size is kept under 5% the size of the images being
attacked. We optimize for each patch p using Eq. (1) on
unlabeled frames from the raw KITTI [10]. We choose the
KITTI dataset since it reflects a safety critical applicationand provides an annotated training set for evaluation. We
use the optical flow predictions from the networks as pseudo
ground truth labels to optimize for the patch that produces
the highest angular error. This allows us to leverage approximately 32000 frames instead of 200 annotated frames fromthe training dataset. For each of the networks, we use thepre-trained model that gives the best performance withoutfine tuning on KITTI dataset. We use Pytorch [1] as our
optimization framework and optimize for the patch using
stochastic gradient descent. Figure 2 shows the patches weobtain by optimizing over different optical flow networks
and different patch sizes.
Evaluation. To quantify the robustness of the networks to
our adversarial patch attacks, we measure their performanceon the training set of the KITTI 2015 optical flow benchmark using the original and perturbed images, respectively.
We measure the vulnerability of the optical flow networks
using average end point error (EPE) and relative degradation of EPE in the presence of the adversarial patch (Table 2). The errors are computed by placing the learned patch
p at a random location for each image in the training set. All
images are resized to 384 × 1280 and input to the networks.
We evaluate optical flow at the full image resolution.
As shown in Table 2, the performance of encoderdecoder architectures – FlowNetC and FlowNet2 degrades
significantly, even when using very small patches (0.1% of
the image size). The increase in error is about 400 − 500%
under attack. For the spatial pyramid based methods –SpyNet, PWC-Net and Back2Future, the degradation issmall. Back2Future, which is an unsupervised method andshares a similar architecture with PWC-Net, suffers less

Table 2. White-box Attacks. Effect of patch size (in pixel and percent of the image size) on different optical flow networks. We show
average End Point Error (EPE) on KITTI 2015 for each network with and without an attack. For attacks, we also show relative degradation% in the EPE.

Network
FlowNetC [8]FlowNet2 [13]SpyNet [20]PWC-Net [28]Back2Future [15]Unattacked
EPE
14.56
11.90
20.26
11.03
17.49

25x25 (0.1%)
EPE
Rel
29.07 +100%
17.04 +43%
20.59
+2%
11.37
+3%
18.04
+3%

Table 3. Black-box Attacks. Attacks on different optical flow
methods using a universal patch. Methods below the line were
not used for training the patch.

FlowNet2 [13]PWC-Net [28]FlowNetC [8]SpyNet [20]Back2Future [15]Epic Flow [23]LDOF [6]Unattacked
EPE
11.90
11.03
14.56
20.26
17.49
4.52
9.20

Attacked
EPE
Rel
36.13 +203 %
11.01
+0 %
86.12 +492 %
20.39
+1 %
17.44
+0 %
4.66
+3 %
9.17
+0 %

degradation than PWC-Net on the largest patch size.
In Figure 3, we illustrate that encoder-decoder architectures are extremely vulnerable to patch attacks irrespectiveof scene content, location and orientation of the patch. The
patch size in Figure 3 is 25 × 25 pixels, about 0.12% of thesize of the image being attacked. The figure shows that theattacks extend significantly beyond the region of the patch,causing a significant proportion of the optical flow vectors
to be degraded. We also observe that SpyNet is least affected by the attack. For PWC-Net and Back2Future, thedegradation is contained within the region of the patch. The
strength of the attack increases with the size of the patch.
Figure 4 shows attacks with a larger patch size of 153×153,
which is about 4.76% of the image size. Other examples
and patch sizes are shown in the supplementary material.

4.2. Black-box Attacks
In a real world scenario, the network used by a system
like an autonomous car will most likely not be accessible tooptimize the adversarial patch. Therefore, we also consider
a Black-box attack, which learns a “universal” patch to attack a group of networks. Since there are two architecture
types (Encoder-Decoder and Spatial Pyramid), we consider
one network from each type to learn a “universal” patch toattack all networks. We pick the networks – FlowNet2 [13]and PWC-Net [28] and jointly optimize for a patch that attacks both networks using Eq. 1. In this way, we obtain apatch that is capable of attacking both encoder-decoder andspatial pyramid architectures. The resulting patch can be

51x51 (0.5%)
EPE
Rel
40.27 +177%
24.42 +105%
21.00
+4%
11.50
+4%
18.24
+4%

102x102 (2.1%)
EPE
Rel
82.41 +466%
38.57 +224%
21.22
+5%
11.86
+7%
18.73
+7%

153x153 (4.8%)
EPE
Rel
95.32 +555%
59.58 +400%
21.00
+4%
12.52
+13%
18.43
+5%

seen in Figure 7 and the Sup. Mat. While we focus on two
networks for a proof of concept, the “universal” patch caneasily be jointly optimized using more networks to obtaineven more effective attacks.
The training of the universal patch is identical to theWhite-box case. However, we decided to make the evaluation of the Black-box attacks more realistic by using thecamera motion and disparity ground truth provided by the
KITTI Raw dataset to compute realistic motion of patches
assuming they are part of the static scene. We project the
random location of the patch from the first frame into the
3D scene and re-project it into the second frame considering the camera motion and disparity. We randomly pick adisparity between the maximal disparity of the whole scene
and minimal disparity in region of the patch to obtain a location in the free space between the camera and other objects
in the scene. Given the re-projection of points on the patch,
we then estimate a homography to transform the patch andwarp it using bilinear interpolation. For sequences without
ground truth camera poses we use zero motion like in the
White-box attacks. In this way, we simulate the situation
where a patch is attached to some static object in the scene.
In the supplementary material, we also show the effect of auniversal patch that has zero motion w.r.t the camera as in
the White-box attacks.
Evaluation. Table 3 shows the performance of all networks and classical optical flow methods from Table 1 in
the presence of the Black-box adversary. Consistent withWhite-box attacks, we observe that encoder-decoder architectures suffer significantly in the presence of the adversarial patch. Both, spatial pyramid architectures and classicalmethods, are more robust to patch attacks.
We observe that the attacks on encoder-decoder networks are most effective when the patch is placed in thethe center of the image. The patches influence image regions according to the receptive field of the convolutional
networks. Although the patches are small (∼ 0.5%) compared to the image size, they can affect up to 50% of theimage area as can be observed in Figure 5. We show morequalitative results in the supplementary material.

tC

e
wN
Flo

t2
Ne
w
o
Fl
yN
Sp

et

et
-N
C
PW
re
utu
F
k2

c
Ba

Attacked Reference

Unattacked Flow

Attacked Flow

Difference

Figure 3. White-box Attacks. 25x25 patch attacks on all networks.

wN
Flo
wN
Flo
e
yN
Sp

etC
et2

t

C-N
PW

et
re

utu

F
ck2
Ba

Attacked Reference

Unattacked Flow

Attacked Flow

Difference

Figure 4. White-box Attacks. 153x153 patch attacks on all networks.

4.3. Real World Attacks

5. Zero-Flow Test

In order to evaluate the effectiveness of adversarial
patches in the real world, we print the Black-box patch
at 1200 dpi, scaled to 8 times the original size in theimage. We then place the patch in the scene, record avideo sequence, and observe the effects on two networks- FlowNetC and FlowNet2. Figure 7 shows the effect ofthe printed patch on a deployed optical flow system running
FlowNetC. While the location of the patch in the scene wasconstant, we experiment with and without camera motion.
We observe that the real world attacks on FlowNetC work
equally well with and without camera motion, while attacks
on FlowNet2 worked better with camera motion. For thevideo demo of this experiment, please follow this link –http://flowattack.is.tue.mpg.de/.

To better understand the behaviour of neural networks inthe presence of an adversarial patch attack, we propose the
Zero-Flow test. We generate an image I by independently
sampling uniform random noise for each of the pixels. We
then paste the universal patch p on I, resulting in a per˜ We then replicate the image I˜ and input theturbed image I.
two frames to the network. Since the image is replicated forboth frames, the optical flow is zero over all the pixels.
˜ I)
˜ = 0
Assuming that the attack was ineffective, F (I,must hold. Furthermore, if perturbation is not applied to the
network inputs, F (I, I) = 0 must hold. For feature maps of
˜ I)
˜ = Fk (I, I) must hold. Here, Fk is thethe network, Fk (I,output from the k-th layer of the neural network. In Figure
6, we show feature map visualizations along with their average norm from two networks, FlowNetC and PWC-Net. For
each network we show the feature maps with and without

/
uts
p
n
I

wN
Flo

wN
Flo

yN
Sp

GT

etC

et2

et

C-N
PW

et

re
utu
F
2
ack

B

w
Flo
c
i
Ep
F

O
LD

Unattacked Flow

Attacked Flow

Difference

Figure 5. Black-box Attacks. Universal patch trained on FlowNet2 and PWC-Net used on all approaches pasted on a traffic sign withrealistic motion as described in Section 4.2.

the patch. Each feature map is normalized independently
for contrast and scaled to the same size.
Figure 6 illustrates that the encoder of FlowNetC(conv<3 1,4,5,6>) spatially propagates the patch information, as expected, and the feature activation encompasses the entire image towards the end of the encoder(conv6). While deconvolutions (deconv<4,5,6>)
reduce the spatial propagation of activations, they also leadto a strong amplification. Finally, the flow prediction layers (flow<5,4,3,2>) combine both problems by fusing the encoder and decoder responses. This leads to large
and widespread degradation of optical flow predictions for
the Zero-Flow test. Without the adversarial patch, we observe well distributed activations in the first convolutions aswell as low activations for all layers in general.
PWC-Net shows low activations in all correlation layers (corr<6,5,4,3,2>) irrespective of the adversarial patch. Interestingly, the flow decoder of lower levels(flow<6,5>) predict large flow in both cases – with andwithout patch. However, the flow prediction at finer levels of the pyramid (flow<3,2>) seems to compensate
for these errors. Similar to FlowNetC, the deconvolution
layers (upfeat<5,4,3>) lead to an amplification of activations, with and without the patch. In addition, they seemto cause checkerboard artifacts that have also been observedin other contexts [19, 30].
The Zero-Flow test reveals several problems with the optical flow networks. First, the networks (FlowNetC, PWCNet) are not spatially invariant. In the case of zero motion,
the activation maps of the network vary spatially across thewhole image even without the adversarial patch. Second,
the pyramid networks predict large motion in coarser levels flow<5,4> irrespective of the presence or absence ofthe patch. Finally, the deconvolution layers lead to an amplification of the responses and checkerboard artifacts. The
attacks exploit these weaknesses and amplify them, degrad-Inputcorr conv3 1 conv4 conv5 conv6 flow6 decon5 flow5 decon4 flow4 decon3 flow3 decon2 predict

Mean

0.0

0.0

0.0

0.0

0.0

0.1

0.1

0.1

0.1

Mean

0.1

6.6

50.3

98.3

196.5

464.9 1015.8 825.0 3035.6 717.8 2491.1 316.1
FlowNetC

668.2

78.6

Input

corr6

flow6 upfeat6 corr5

flow5 upfeat5 corr4

flow4 upfeat4 corr3

flow3 upfeat3 corr2

flow2

Mean

0.0

12.2

4.4

0.3

250.9

157.3

31.1

83.0

0.0

1.5

14.5

0.0

0.0

Mean

0.00

12.2

4.4

0.2

38.2

134.92 0.04
2.4
PWC-Net

62.6

0.01

0.1

13.8

0.00

0.02

0.1

0.0

0.2

0.2

0.1

0.2

Figure 6. Visualization of Feature Maps. Average norm of feature maps as we move deeper through the network. Each image isnormalized independently for contrast and scaled to the same size. Mean refers to average norm of the feature maps.

6. Discussion
We have shown that patch attacks generalize to state-ofthe-art optical flow networks and can considerably impact
the performance of optical flow systems that use deep neuralnetworks. The patch attacks are invariant to translation andsmall changes in scale and orientation of the patch. Learning patches for larger changes in scale and orientation iscomputationally costly. Nevertheless, it is possible to replicate the attacks with real printed patterns.

Figure 7. Real World Attacks on FlowNetC. Top: The presenceof a printed adversarial patch visible in the centre of the image
significantly degrades the optical flow predictions. Bottom: As thepatch is covered by the subject, the effect of the patch vanishes.

ing predictions. In the Supplementary Material, we show
additional visualizations of feature maps (from FlowNet2
and Back2Future).
Acknowledgement. We thank A. Punnakkal, C. Huang for paper revisions; and P. Ghosh for fruitful discussions.
Disclosure. MJB has received research gift funds from Intel, Nvidia,
Adobe, Facebook, and Amazon. While MJB is a part-time employee of
Amazon, his research was performed solely at MPI. He is also an investor
in Meshcapde GmbH.

Small adversarial patches cause large errors even inremote regions with encoder-decoder networks (FlowNet,
FlowNet2) while spatial pyramid networks (SpyNet, PWCNet, Back2Future) are not strongly affected. Classical approaches are also robust to these attacks. This indicates thatincorporating classical ideas such as image pyramids andcorrelations into architectures makes models more robust.
In our analysis, we observed the influence of such
patches on feature maps in encoder-decoder as well as
spatial pyramid networks. We also note that this resultsfrom inherent spatially variant properties of the optical flow
networks. This indicates that convolutions alone are notenough to enforce spatial invariance in these networks.
Finally, independent of the attacks, the Zero-Flow test
provides a novel approach to identify problems in deep flow
networks. It reveals the internal network behaviour which
may be useful to improve the robustness of flow networks.

References[1] https://pytorch.org. 4[2] Anish Athalye and Ilya Sutskever. Synthesizing robust adversarial examples. arXiv.org, 2017. 3[3] Simon Baker, Daniel Scharstein, JP Lewis, Stefan Roth,Michael J Black, and Richard Szeliski. A database and evaluation methodology for optical flow. International Journalof Computer Vision (IJCV), 92(1):1–31, 2011. 1, 2
[4] Michael J Black and P Anandan. The robust estimationof multiple motions: Parametric and piecewise-smooth flow
fields. Computer Vision and Image Understanding (CVIU),
63(1):75–104, 1996. 1[5] Tom B Brown, Dandelion Mané, Aurko Roy, Martı́n Abadi,
and Justin Gilmer. Adversarial patch. arXiv.org, 2017. 2, 3
[6] T. Brox and J. Malik. Large displacement optical flow: Descriptor matching in variational motion estimation. IEEE
Trans. on Pattern Analysis and Machine Intelligence (PAMI),
2011. 1, 2, 4, 5
[7] D. J. Butler, J. Wulff, G. B. Stanley, and M. J. Black. A naturalistic open source movie for optical flow evaluation. In A.
Fitzgibbon et al. (Eds.), editor, Proc. of the European Conf.
on Computer Vision (ECCV), Part IV, LNCS 7577, pages
611–625. Springer-Verlag, Oct. 2012. 1, 3
[8] Alexey Dosovitskiy, Philipp Fischery, Eddy Ilg, Caner Hazirbas, Vladimir Golkov, Patrick van der Smagt, Daniel Cremers, Thomas Brox, et al. Flownet: Learning optical flow
with convolutional networks. In Proc. of the IEEE International Conf. on Computer Vision (ICCV), pages 2758–2766.
IEEE, 2015. 1, 2, 3, 4, 5
[9] Ivan Evtimov, Kevin Eykholt, Earlence Fernandes, Tadayoshi Kohno, Bo Li, Atul Prakash, Amir Rahmati, andDawn Song. Robust physical-world attacks on deep learning models. arXiv.org, 1, 2017. 3[10] Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we
ready for autonomous driving? the KITTI vision benchmark
suite. In Proc. IEEE Conf. on Computer Vision and Pattern
Recognition (CVPR), 2012. 1, 3, 4
[11] Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy.
Explaining and harnessing adversarial examples. arXiv.org,
2014. 2, 3
[12] Berthold K Horn and Brian G Schunck. Determining opticalflow. Artificial Intelligence (AI), 17(1–3):185–203, 1981. 1[13] Eddy Ilg, Nikolaus Mayer, Tonmoy Saikia, Margret Keuper, Alexey Dosovitskiy, and Thomas Brox. Flownet 2.0:
Evolution of optical flow estimation with deep networks. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), volume 2, 2017. 1, 2, 3, 4, 5
[14] Joel Janai, Fatma Guney, Jonas Wulff, Michael J Black, andAndreas Geiger. Slow flow: Exploiting high-speed cameras for accurate and diverse optical flow reference data. In
Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 3597–3607, 2017. 3[15] Joel Janai, Fatma Güney, Anurag Ranjan, Michael Black,and Andreas Geiger. Unsupervised learning of multi-frame
optical flow with occlusions. In Proc. of the European Conf.
on Computer Vision (ECCV), 2018. 2, 3, 4, 5[16] Alexey Kurakin, Ian Goodfellow, and Samy Bengio. Adversarial examples in the physical world. arXiv.org, 2016. 3[17] Simon Meister, Junhwa Hur, and Stefan Roth. Unflow: Unsupervised learning of optical flow with a bidirectional census loss. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018. 2, 3
[18] Anh Nguyen, Jason Yosinski, and Jeff Clune. Deep neural networks are easily fooled: High confidence predictions
for unrecognizable images. In Proc. IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), pages 427–
436, 2015. 2, 3
[19] Augustus Odena, Vincent Dumoulin, and Chris Olah. Deconvolution and checkerboard artifacts. Distill, 2016. 7[20] Anurag Ranjan and Michael J. Black. Optical flow estimation using a spatial pyramid network. In Proc. IEEE Conf.
on Computer Vision and Pattern Recognition (CVPR), 2017.
1, 2, 3, 4, 5[21] Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim,
Deqing Sun, Jonas Wulff, and Michael J Black. Competitive
collaboration: Joint unsupervised learning of depth, camera
motion, optical flow and motion segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern
Recognition, pages 12240–12249, 2019. 2[22] Anurag Ranjan, Javier Romero, and Michael J Black. Learning human optical flow. Proc. of the British Machine Vision
Conf. (BMVC), 2018. 3[23] Jerome Revaud, Philippe Weinzaepfel, Zaid Harchaoui, andCordelia Schmid. EpicFlow: Edge-preserving interpolationof correspondences for optical flow. In Proc. IEEE Conf. on
Computer Vision and Pattern Recognition (CVPR), 2015. 2,4, 5[24] Mahmood Sharif, Sruti Bhagavatula, Lujo Bauer, andMichael K Reiter. Accessorize to a crime: Real and stealthy
attacks on state-of-the-art face recognition. In Proc. of the
ACM SIGSAC Conference on Computer and Communications Security, pages 1528–1540. ACM, 2016. 3[25] Jiawei Su, Danilo Vasconcellos Vargas, and Sakurai Kouichi.
One pixel attack for fooling deep neural networks. arXiv.org,
2017. 2, 3
[26] Deqing Sun, Stefan Roth, and Michael J Black. Secrets of
optical flow estimation and their principles. In Proc. IEEE
Conf. on Computer Vision and Pattern Recognition (CVPR),
pages 2432–2439. IEEE, 2010. 1, 2
[27] Deqing Sun, Stefan Roth, and Michael J Black. A quantitative analysis of current practices in optical flow estimation and the principles behind them. International Journal ofComputer Vision (IJCV), 106(2):115–137, 2014. 2[28] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.
Pwc-net: Cnns for optical flow using pyramid, warping, andcost volume. In Proc. IEEE Conf. on Computer Vision and
Pattern Recognition (CVPR), pages 8934–8943, 2018. 2, 3,
4, 5[29] Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan
Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. Intriguing properties of neural networks. arXiv.org, 2013. 2,
3[30] Zbigniew Wojna, Jasper R. R. Uijlings, Sergio Guadarrama,
Nathan Silberman, Liang-Chieh Chen, Alireza Fathi, andVittorio Ferrari. The devil is in the decoder: Classification,
regression and gans. In Proc. of the British Machine Vision
Conf. (BMVC), 2017. 7[31] Jason J Yu, Adam W Harley, and Konstantinos G Derpanis. Back to basics: Unsupervised learning of optical flow
via brightness constancy and motion smoothness. arXiv.org,
2016. 2, 3