Adversarial Attacks for Optical Flow-Based Action Recognition Classifiers

The success of deep learning research has catapulted deep models into production systems that our society is becoming increasingly dependent on , especially in the image and video domains . However , recent work has shown that these largely uninterpretable models exhibit glaring security vulnerabilities in the presence of an adversary . In this work , we develop a powerful untargeted adversarial attack for action recognition systems in both white-box and black-box settings . Action recognition models differ from image-classification models in that their inputs contain a temporal dimension , which we explicitly target in the attack . Drawing inspiration from image classifier attacks , we create new attacks which achieve state-of-the-art success rates on a two-stream classifier trained on the UCF-101 dataset . We find that our attacks can significantly degrade a model ’ s performance with sparsely and imperceptibly perturbed examples . We also demonstrate the transferability of our attacks to black-box action recognition systems . 1 INTRODUCTION . As machine learning ( ML ) for computer vision becomes more popular , and ML systems are integrated into production level technologies , the security of ML models becomes a serious concern . Until recently , researchers have been focused on pushing the boundaries of model accuracy , while not prioritizing model security as a first-order design constraint . Many previous works ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ; Moosavi-Dezfooli et al. , 2016 ; Papernot et al. , 2016b ; Szegedy et al. , 2013 ) have shown that adding imperceptible noise to an image can easily fool a CNN-based classifier , but no such work has been done in the video and action recognition domain . A unique property of the action recognition domain is the presence of a temporal dimension , which has been shown to be crucially important to the efficacy of current classifiers ( Sevilla-Lara et al. , 2017 ; Simonyan & Zisserman , 2014 ; Carreira & Zisserman , 2017 ) . It also provides a new axis to attack . In this work , we consider the two-stream architecture ( Simonyan & Zisserman , 2014 ) and its variants , which represent the market share of the current state-of-the-art action recognition models . We isolate the motion stream classifier of a two-stream model as our model to attack . Previous work has shown that the standalone motion stream , trained on stacks of optical flow fields , outperforms the standalone spatial stream , and is not significantly worse than the full model ’ s capability ( Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016b ; Varol et al. , 2018 ; Feichtenhofer et al. , 2017 ) . Therefore , we contend that if the temporal stream is compromised , the integrity of the entire classifier is compromised . Also , any image domain attack may be applied to the spatial classifier . The traditional variational optical flow calculation algorithms are non-differentiable , meaning that performing a known white-box image attack to directly perturb the video frames is not possible due to lack of gradient flow . To circumvent this , we consider the FlowNet 2.0 model , which is a convolutional neural network that estimates optical flow fields ( Ilg et al. , 2017 ) . If we consider a target model that is a combination of the FlowNet 2.0 model and the temporal stream CNN , we have an end-to-end differentiable model that takes a raw video and returns a classification . We leverage this composite model to obtain gradients of the temporal stream classification loss with respect to the spatial input video frames , which is precisely what is needed for an effective white-box attack . We then demonstrate the effectiveness of our white-box adversarial examples on black-box models trained with variational optical flow algorithms . The observed transferability of our attack on blackbox models greatly increases its practical usefulness in the real-world . Overall , our contributions are as follows : 1 . We create a white-box , untargeted attack for a two-stream action recognition classifier ; 2 . We show that the performance of action recognition systems can be completely degraded with sparsely and imperceptibly perturbed examples ; 3 . We create a black-box attack to produce examples that can transfer to other action recogni- tion systems with alternative optical flow and CNN algorithms ; 4 . We introduce the idea of salient video frames in the context of video classification . 2 RELATED WORK . Action Recognition . There are several general methods for modeling motion in action recognition classifiers . One approach is to use 3D-Convolution on stacked spatial frames , attempting to learn spatial features and temporal difference features concurrently ( Ji et al. , 2013 ; Karpathy et al. , 2014 ) . Another strategy involves training LSTM models on sequences of features extracted with convolutional layers of image CNNs ( Li et al. , 2018 ; Tsironi et al. , 2017 ) . Finally , a host of research efforts train CNN classifiers on optical flow displacement fields and spatial frames separately ( Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016b ; Wang et al. , 2016 ; Ng et al. , 2015 ; Feichtenhofer et al. , 2016a ; 2017 ; Carreira & Zisserman , 2017 ; Girdhar et al. , 2017 ; Sigurdsson et al. , 2017 ; Sevilla-Lara et al. , 2017 ; Varol et al. , 2018 ) . Here , we will focus on these optical flow-based systems , as they represent a significant portion of the most effective methods to date ( Carreira & Zisserman , 2017 ; Sevilla-Lara et al. , 2017 ) . The first design to use CNNs with optical flow is the two-stream model Simonyan & Zisserman ( 2014 ) . This model uses two CNNs : one trained on spatial frames and the other trained on stacks of sequential optical flow displacement fields . To produce a final prediction , each stream makes a classification independently , and the predictions are fused . Since this innovation , there have been many works that employ the idea of a two-stream architecture and use optical flow specifically to model motion . Feichtenhofer et al . ( 2016b ) propose convolutional network fusion , where spatial stream and temporal stream feature maps are combined before making a prediction . Feichtenhofer et al . ( 2016a ) and Feichtenhofer et al . ( 2017 ) describe spatiotemporal multiplier networks and spatio-temporal residual networks , which leverage deep residual networks for both streams by allowing the streams to share information via the residual connections . Finally , one of the top performing methods for action recognition is Carreira & Zisserman ( 2017 ) Two-Stream Inflated 3D ConvNet ( I3D ) architecture , which combines techniques from the two-stream model , 3D convolution , transfer learning , and the Inception model . Each of the aforementioned methods are considered state-of-the-art , and all involve a two-stream model where the motion stream operates on optical flow . This distinction serves as motivation for the framing of our attack model . Optical Flow . Optical flow generation consists of two broad categories : variational and deep learning-based . Two common variational techniques are Farneback ( Farnebäck , 2003 ) and TV-L1 ( Zach et al. , 2007 ; Snchez Prez et al. , 2013 ) . The Farneback algorithm estimates frame neighborhoods by quadratic polynomials using the polynomial expansion transform , and is optimized using a coarse-to-fine strategy . The TV-L1 algorithm is a more recent approach that works to minimize a function containing a data fidelity term using the L1 norm and a regularization term based on the total variation of the flow ( Zach et al. , 2007 ) . TV-L1 is more accurate , and shows increased robustness against illumination changes , noise , and occlusion . Recent works also show that deep convolutional neural networks trained in a supervised fashion can be a fast and effective way to estimate optical flow ( Fischer et al. , 2015 ; Ilg et al. , 2017 ) . Specifically , FlowNet2 has been shown to be as accurate as state-of-the-art variational optical flow estimation methods , while running significantly faster . Adversarial Attacks . Crafting adversarial examples has become an increasingly popular area of research recently , especially for image classifiers . Roughly speaking , the space is divided into two general categories : white-box and black-box . White-box attacks assume full knowledge of the model and parameters , and often use gradient information in the attack ( Goodfellow et al. , 2014 ; Carlini & Wagner , 2017 ; Moosavi-Dezfooli et al. , 2016 ; Papernot et al. , 2016b ; Szegedy et al. , 2013 ) . Black-box attacks on the other hand treat the model as an oracle and do not have intimate knowledge of the model architecture or specific parameters ( Papernot et al. , 2016a ) . Perhaps the most popular white-box attack method is the Fast Gradient Sign Method ( FGSM ) ( Goodfellow et al. , 2014 ) . FGSM uses the gradient of the loss w.r.t . the input to adjust the image in the direction that maximizes the loss . This results in an imperceptible noise field being added to the original image that significantly degrades the classification performance of the model . Since , attacks such as Carlini-Wagner Attack ( Carlini & Wagner , 2017 ) , Jacobian Saliency Map Attack ( JSMA ) ( Papernot et al. , 2016b ) , Deepfool ( Moosavi-Dezfooli et al. , 2016 ) , and Iterative Least Likely ( Kurakin et al. , 2016 ) , have leveraged the gradient information of the model in some fashion to create adversarial examples . Another important finding from these attacks is that an adversarial example computed from one model is often adversarial to other models . This principle is called transferability and has been studied extensively for image classification systems ( Papernot et al. , 2016a ; Tramèr et al. , 2017 ; Liu et al. , 2016 ) . 3 ATTACK METHODOLOGY . 3.1 MODEL UNDER ATTACK . In this work we define the Model Under Attack ( MUA ) as the isolated motion stream of the twostream architecture , where motion is modeled as optical flow fields . Fig . 1 shows a depiction of this general model configuration . Note , the input is a stack ( usually length 11 ) of video frames , optical flow is calculated internally , and the output is the classifier ’ s prediction . This MUA represents any model that uses optical flow stacks to model motion . Previous works ( Simonyan & Zisserman , 2014 ; Feichtenhofer et al. , 2016b ; Varol et al. , 2018 ; Feichtenhofer et al. , 2017 ) report that the standalone motion stream of a two-stream model outperforms the standalone spatial stream . On the UCF101 dataset , Simonyan & Zisserman ( 2014 ) show that the spatial stream of a two-stream model alone achieves 73.0 % accuracy , while the temporal stream alone achieves 83.7 % . Thus , if the motion stream can be fooled , the entire model is compromised . 3.2 ATTACK SETUP . We define the input frame stack as i , where i is a series of N discrete frames , i.e . i = [ i0 , i1 , i2 , . . . , iN−1 ] . The optical flow calculation that produces a single displacement field between two successive frames is Xn = Fn ( in , in+1 ) , where Xn is the horizontal and vertical displacement fields , and Fn is the optical flow function ( i.e . Farneback , TV-L1 , FlowNet2 ) that operates on frames in and in+1 . Since action recognition systems operate on stacks of optical flows , we define X as an optical flow stack , which is formed by concatenating the individual flow fields X0 , X1 , X2 , . . . , XN−2 while maintaining the respective ordering in time . For convenience , let F ( i ) = X represent an entire optical flow stack given a video frame stack ( i ) as input . The CNN classifier R , with loss function C , is a function of the optical flow stack , and R ( X ) ( or R ( F ( i ) ) ) is a softmax array of probabilities where the class prediction is argmax ( R ( X ) ) . The goal of the attack is to apply the least amount of noise ( ) , s.t . argmax ( R ( F ( i ) ) ) 6= argmax ( R ( F ( i + ) ) ) assuming the initial prediction is correct . In this case we define the perturbed video , i′ = i + , to be adversarial . The proposed white-box attack draws from the ideas of FGSM ( Goodfellow et al. , 2014 ) and iterative FGSM ( Kurakin et al. , 2016 ) . FGSM attacks adjust the input image using a scaled version of the sign of the gradient of the loss w.r.t . the input . In this context , the FGSM method only calculates the gradients through the classifier CNN , w.r.t . the optical flow stack ( ∂C/∂X ) . This means it does not provide the information necessary to adjust the input video frames ( i ) directly , as the optical flow calculation is performed internally within the model . For an action recognition attack , we must compute gradients through the classifier and the optical flow calculation . One problem when using variational algorithms such as Farneback and TV-L1 is that the optical flow calculation is nontrivially differentiable . Therefore , we use FlowNet2 , a convolutional neural network that estimates optical flow between successive frames . Using FlowNet2 , we can easily compute the gradients through the model ( Fn ) , and define the derivative of the output w.r.t . the inputs of FlowNet2 as ∂Xn/∂in and ∂Xn/∂in+1 . With chain rule and ∂C/∂X we can compute the gradient of the loss w.r.t each input image , for n in [ 0 , N ) as ∂C ∂in = ∂C ∂Xn ∂Xn ∂in n = 0 ∂C ∂Xn−1 ∂Xn−1 ∂in + ∂C∂Xn ∂Xn ∂in 0 < n < N − 1 ∂C ∂Xn−1 ∂Xn−1 ∂in n = N − 1 . ( 1 ) As a result of ( 1 ) , we obtain the gradient of the loss w.r.t . the input images themselves . Fig . 2 shows the diagram of the model used in the attack . The video frame stack is represented as separate images , there is one FlowNet2 model for each pair of sequential frames , and the individual optical flow displacement fields are concatenated before being input into the CNN classifier . The text shown in black represents the signals that are present in the forward pass and the text in blue pertains to the backward pass . The following sections are dedicated to describing each variant of our attack , and how they use the information computed in the backward pass to create adversarial examples .

Summary: This paper presents a framework for generating adversarial perturbations for videos. Specifically, the paper proceeds by using a standard image-based adversarial noise generation setup (such as the FGSM scheme), and applies it to the motion stream of a two-stream action recognition pipeline; this motion stream typically using optical flow images. As such flow generation is usually done offline and thus is not differentiable, the paper resorts to the recent FlowNet 2.0 scheme that uses an end-to-end learnable deep flow generation model. Three variants of the scheme are provided, (i) that perturbs all frames in a flow stack, (ii) that perturbs only a sparse set of frames as decided by the importance of a frame to action classification, and (iii) a variant of (ii) that recalculates the gradients for all frames if the ones selected in (ii) were not adversarial. Experiments are provided on UCF101 dataset and show promise. Analysis is presented on the transferability of  the learned noise to flow images generated via external means.