Adversarial Attacks for Optical Flow-Based Action Recognition Classifiers

Abstract: The success of deep learning research has catapulted deep models into production
      systems that our society is becoming increasingly dependent on, especially in the
      image and video domains. However, recent work has shown that these largely
      uninterpretable models exhibit glaring security vulnerabilities in the presence of
      an adversary. In this work, we develop a powerful untargeted adversarial attack
      for action recognition systems in both white-box and black-box settings. Action
      recognition models differ from image-classification models in that their inputs
      contain a temporal dimension, which we explicitly target in the attack. Drawing
      inspiration from image classifier attacks, we create new attacks which achieve
      state-of-the-art success rates on a two-stream classifier trained on the UCF-101
      dataset. We find that our attacks can significantly degrade a model’s performance
      with sparsely and imperceptibly perturbed examples. We also demonstrate the
      transferability of our attacks to black-box action recognition systems.

Reviews and author responses:

Dear Reviewer,

Thank you for your review and thoughtful comments. We would like to now address each of your observed weaknesses with some of our thoughts and explanations.

**Weakness 1:**  As we observed in the literature review stage, the problem of action recognition/video classification has not yet been “solved” and the community seems to be divided on techniques (CNN+LSTM, OpticalFlow+CNN, 3DConvolution) and datasets. This is of course in contrast to the image classification community which has largely accepted CNNs as the technique of choice, and the ImageNet dataset as the “gold standard” dataset. In this work, we tried to select one of the most popular classification techniques that has significant market share in the space of action recognition, being the two-stream model with optical flow. This model architecture brings with it several unique challenges that cannot be solved directly/trivially with image domain attack techniques. First, the optical flow stage is challenging to attack and requires a novel solution because most flow estimation techniques are nondifferentiable and may not be approximated with an identity function. Here we choose Flownet2 as a representative deep learning-based optical flow method purely because we can compute gradients through it. 
Since the volume of data is much larger, we are also faced with unique challenges regarding how to create sparse attacks and how to perturb a single frame when it has contributed to two paths through the model. Another novelty of the design is the idea of sparsely perturbing frames based on frame level saliency values. Unlike in image attacks where there is only one frame to consider, in videos we can choose to perturb a subset of frames, which is where our saliency metric comes into play. Ultimately, combining the FlowNet2 with the CNN was designed to be a reasonable motion stream configuration. We feel the novelty to be in the sparse attacking methodology based on the proposed idea of salient frames. Also, our initial results for attacking in a black-box setting shows promise, meaning that adversarial examples generated with the FlowNet2 model configuration transfer moderately well to models using other flow estimation techniques without requiring any additional effort over the white-box attack. We feel that further investigation of the FlowNet2 finetuning process to produce more transferable examples is a promising future direction. Furthermore, given the recent work in neural network acceleration and optimization, it is not unreasonable for a system to use FlowNet2 (or another deep learning method) as the optical flow step because it may be optimized end-to-end with the CNN model.

**Weakness 2:**  The intentions when using FlowNet2 were to have a representative deep learning based optical flow algorithm through which we can compute gradients. However, in our readings, we did not see mention of FlowNet2 being notoriously sensitive to input perturbations. Also, the goal when training FlowNet2 is not to make it robust to adversarial attack, rather to make it a good estimator of optical flow and to mimic the ground truth optical flow method (which in this case is TVL1). 
Also, given the surprising success of image domain attacks in producing nearly imperceptible perturbations and completely fooling state of the art models, it does not seem shocking that small perturbations may completely degrade the performance of a deep learning algorithm.



Dear Reviewer,

Thank you for your review and thoughtful comments. We would like to now address each of your observed weaknesses with some of our thoughts and explanations.

**Weakness 1:** We would contend that there are several non-trivial and novel aspects to attacking video classifiers, and specifically optical flow-based classifiers in this case. Since there are several methods for video classification, and one is not widely accepted as the “best”, we had to choose to attack one of the most popular methods. For optical flow-based methods, one of the core challenges for attacking is the optical flow calculation, and how to perturb through it. If we look to image-based attacks, one attack that broke most of the defenses in ICLR 2018 works by Backward Pass Differentiable Approximation [1]. The algorithm attempts to introduce a stand-in layer (only for the backward pass) for a layer of a system that is non-differentiable. This stand-in approximates gradients through the non-differentiable layers of image domain defenses. However, the non-differentiable layers in the tests did not fundamentally change the domain of the data. Rather, the layers applied filtering, or some randomization of pixels in local regions which largely keeps the global structure of the data. If we consider optical flow as a preprocessor, it is not trivial to extend BPDA here because the output data domain is fundamentally different and the stand-in layer would have to learn to approximate gradients through a very complex optical flow algorithm.  In this way, the task of extending image domain attacks to the optical flow video domain is non-trivial. The novelty of the method comes from the observation that no existing attack has been able to perturb the video frames of an optical flow based classifier, and there is no straightforward way to do this using only image based attacking methods. Another challenge is that a single frame of video contributes to two optical flow fields, and if not dealt with properly, the perturbation suggestions from both paths may be destructive to each other. Another challenge is the addition of the time dimension, and the realization that all frames do not have to be perturbed. This is in contrast to image domain attacks where we do not have to worry about what image to perturb because there is only one. This highlights another novelty in that existing-methods/image-attacks give no indication of how to create sparse perturbations in time on this classifier.

For the point about how the attack would work at test time, since the attack carries **white-box** assumptions, we assume that we would be able to calculate gradients through the model at test time and therefore know all of the weights and labels. We acknowledge that this is a major assumption, but it is a valid threat model in adversarial attack literature [2,3]. For the blackbox setup, we invoke the transferability property and create the adversarial example on our whitebox model where we have all of the information necessary to construct the example.


**Weakness 2:** As mentioned in the previous comment, there are several ways to do video classification/action recognition. OpticalFlow+CNN (i.e. two stream), CNN+LSTM, 3DConvolution, each of which is a fundamentally different architecture and has a unique way of modeling motion. Hence, each requires a different/innovative way of attacking. These different architectures also handle the input video in different ways. Some only consider one frame at a time, some may sample frames out of temporal ordering, and some transform the data into different domains. Thus, each deserves separate considerations. For the referenced papers, paper [a] has different goals than ours and their method is not easily comparable to ours. Their goal is to increase the robustness of the classifier by adding some perturbations to the CNN inputs. They are not so much concerned with image level perturbations because their goal is not an adversarial attack. Although their method is video-classifier agnostic, when using optical flow they only go so far as to perturb the optical flow, as that is the input to the classification stage. Thus, they provide no details on how to use optical flow perturbations to adjust video frames, nor how to create sparse perturbation in time to a video frame stack. Paper [b] uses a fundamentally different action recognition system (CNN + RNN) which presents a different set of challenges. These systems are automatically end-to-end differentiable and they only input single frames at a time. For this reason, extending image domain attacks to CNN+RNN systems is conceivably simpler. 
The fundamental challenge in our case is dealing with the optical flow step, which in most work involves a non-differentiable algorithm, and dealing with a large volume of input data at a single time step. This deserves special treatment and is somewhat unique to the video domain because image-based attacks often do not have to deal with such “roadblocks”. Another special treatment is creating sparse perturbations in time, of which image domain attacks do not have to contend with because there is no time dimension. Finally, the proposed attack is the first we are aware of that specifically targets optical flow-based systems which as mentioned has a unique set of challenges as compared to the other two methods of action recognition.

**Weakness 3:** Since this is an attack on video classifiers, the input data to the model is video frames. Therefore, for an attack on such classifiers we thought it would make the most sense to show perturbations to the video frames themselves. Otherwise, one may criticize the work for not actually being an attack on action-recognition/video-classifiers, as the flow calculation itself can be thought of as an internal preprocessing step before classification. We agree that perturbing the flow directly in new and unique ways is an interesting task that may be a work in itself. However, if one creates an adversarially perturbed optical flow directly that is dependent on feature wise perturbations of very fine granularity, they would not trivially be able to apply those perturbations to the frames and that would become a fundamental weakness of the work. As for the choice of the FlowNet2 model, it is regarded as one of the state-of-the-art deep learning optical flow methods and in this context is representative of any deep learning based optical flow algorithm with easily computed gradients. Our initial results in the black-box setting confirms that FlowNet2 is a promising general approach to optical flow estimation when fine-tuned using flow fields from other algorithms as ground truth.

**Weakness 4:** Since we treated the system as a traditional action recognition system, the optical flow fields are intermediate byproducts of the overall classifier that are never used or seen externally. Therefore, we did not see it fit to monitor the perturbations to the optical flow field, rather we focused on the image level perturbation. But, through experimentation we observed that the optical flow fields are visibly perturbed as a result of our image level perturbations.

[1] Athalye, Anish et al. “Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples.” ICML (2018).
[2] Goodfellow, Ian J. et al. “Explaining and Harnessing Adversarial Examples.” CoRRabs/1412.6572 (2014): n. pag.
[3] Kurakin, A., Goodfellow, I.J., Bengio, S., Dong, Y., Liao, F., Liang, M., Pang, T., Zhu, J., Hu, X., Xie, C., Wang, J., Zhang, Z., Ren, Z., Yuille, A.L., Huang, S., Zhao, Y., Zhao, Y., Han, Z., Long, J., Berdibekov, Y., Akiba, T., Tokui, S., & Abe, M. (2018). Adversarial Attacks and Defences Competition. CoRR, abs/1804.00097.


Dear Reviewer,

Thank you for your review and thoughtful comments. We would like to now address each of your observed weaknesses with some of our thoughts and explanations.

**Weakness 1:** As we observed in the literature review stage, the problem of action recognition/video classification has not yet been “solved” and the community seems to be divided on techniques (CNN+LSTM, OpticalFlow+CNN, 3DConvolution) and datasets (UCF,HMDB,Sports,etc.). This is of course in contrast to the image classification community which has largely accepted CNNs as the technique of choice, and the ImageNet dataset as the “gold standard” dataset. So, we tried to select one of the most popular classification techniques (two-stream with optical flow) and arguably the most popular dataset to benchmark on across the field that consistently shows up in almost all literature (UCF-101). In this assumed setting the motion stream was shown to be the more critical piece to attack. However, we agree that an attack of the spatial domain is also very important for an overall attack on a video classifier system. But, given that we are operating in a **white-box** setting, the idea we are running with is that we already know how to attack spatial frames, because we can use any existing attack from the set of image-domain attacks. Given that the image domain attacks have shown to degrade state of the art classifiers with nearly 100% success, there would arguably be little novelty in showing that a standard image domain attack works on an image classifier trained on UCF101 frames. We believe this is a feasible assumption because the two-stream architecture handles motion and spatial features separately, and therefore a white-box attack can target each separately.

**Weakness 2:** Since the attack uses the **sign** of the gradient rather than the magnitude, we were not concerned with the actual values of the gradients. This is because we always perturb with a fixed step size according to the sign only.

The paper addresses the problem of adversarial attack for an optical-flow-based action recognition in both the white-box setting (i.e., gradients are available) and the black-box settings (i.e., the gradients cannot be computed by the optical flow algorithm). A video is partitioned into fixed-size temporal intervals, a state-of-the-art deep optical flow estimator is applied on each pair of two consecutive frames within the interval, and perturbations are applied to each frame (or selected frames using saliency cues) within the interval to attack the recognition system using Fast Gradient Sign Method (FGSM). Experiments are carried out on the UCF-101 dataset, in the white-box and black-box settings, and show that the method is able to effectively attack the system.

Strengths:
- Considering the relatively unexplored problem of attacking action recognition in the temporal domain .
- Extending the Fast Gradient Sign Method (FGSM) to the temporal domain

Weaknesses:
 - Novelty seems incremental. What appears to be novel is the extension of FGSM from image classification to action recognition (i.e., Sec 3.2) -- specifically, the gradient computation through both the action classifier and the optical flow estimator.  The paper proposes a simple solution that incorporates an existing differential deep optical flow network (FlowNet2.0) into another existing differential action classification model (TwoStream Network). Combining two existing networks seems insufficient for the problem statement as explained next.

- Experiments seem unconvincing. Sec. 5.1 and 5.2 present effectiveness of the proposed attacking approach when using FlowNet2.0 as the optical flow estimator. However, the attacking effectiveness (i.e.,  accuracy drop) may be a consequence of using FlowNet2.0 which is known to be sensitive to additive perturbations.  FlowNet2.0-based action recognition is more sensitive to the perturbations than other methods like TVL1 and Far. 



Summary:
This paper presents a framework for generating adversarial perturbations for videos. Specifically, the paper proceeds by using a standard image-based adversarial noise generation setup (such as the FGSM scheme), and applies it to the motion stream of a two-stream action recognition pipeline; this motion stream typically using optical flow images. As such flow generation is usually done offline and thus is not differentiable, the paper resorts to the recent FlowNet 2.0 scheme that uses an end-to-end learnable deep flow generation model. Three variants of the scheme are provided, (i) that perturbs all frames in a flow stack, (ii) that perturbs only a sparse set of frames as decided by the importance of a frame to action classification, and (iii) a variant of (ii) that recalculates the gradients for all frames if the ones selected in (ii) were not adversarial. Experiments are provided on UCF101 dataset and show promise. Analysis is presented on the transferability of  the learned noise to flow images generated via external means.

Strengths:
1) The different variants of the scheme and sparse selection of the frames to be perturbed are interesting. 
2) The paper makes some interesting observations, namely that (i) only a single frame perturbation might be sufficient to make the video adversarial, and (ii) perturbations computed via FlowNet models are not transferable to those with flow computed via external software -- which is often the practice.

Weaknesses:
1) I think the main weakness of this paper is the lack of any surprise/significant novelty in the presented approach. The main idea follows the common trend in adversarial noise generation for image classification problems, except that the inputs are a stack of frames instead of  a single one; however, such a setting do not seem to bring along any non-trivial challenges. In the second contribution of this paper -- on the sparse selection of frames to attack, there is a lack of clarity in how one would do the iterative attacks at test time, given such sparse frame selection is done via computing the frame level saliency values via the classification loss, which depends on ground truth class labels, which are unavailable at test time. 

2) There are previous works that have attempted video level adversarial perturbation generation, which the paper do not cite or contrast to; such as a few below. Further, the literature survey fails to provide any compelling motivation as to why video perturbation generation is any difficult than image based noise generation -- it does not appear so from the subsequent text that this problem deserves any special treatment in the considered context.
[a] Learning Discriminative Video Representations Using Adversarial Perturbations, Wang and Cherian, ECCV 2018
[b] Sparse Adversarial Perturbations for Videos, Wei et al., arxiv, 2018

3) It is unclear why the paper chose to consider flow produced by a FlowNet model as their inputs for the attack? Why not consider the flow images directly? Of course, the optical flow algorithm may not be differentiable, but that is perhaps besides the point; the focus should be in perturbing flow, in whatever way it is generated. To that end, given that flow (on static camera images) can be sparse, it would be interesting to see how would a perturbation be generated that needs to operate on local regions (where motion happens). In my opinion, using a FlowNet model for flow generation trivializes the proposed algorithm.

4) It could have been interesting if the paper also provided some qualitative results of the optical flow images generated by FlowNet after adding perturbations to the input frames. Are these flow images also quasi-impercitable? 

Overall, I think the paper has some observations that may be slightly interesting; however, it lacks novelty and the analysis or presentation are unconvincing.

This paper proposes an effective attack technique for the widely used optical flow based classification models in white-box and black-box settings. The most interesting result is on the sparsity and frame salience, which could have a lot of applications. But the main idea is to transfer the standard attack techniques from image to video domain. I have some concerns as below. 

1. Page 3, "...while the temporal stream alone achieves 83.7%. Thus, if the motion stream can be fooled, the entire model is compromised."

This statement is not exactly correct. Motion stream is better on UCF101 and HMDB51 dataset, which are two medium scale action recognition dataset. On other large-scale datasets like Sports1M, Kinetics, ActivityNet, etc., motion stream performs much worse than spatial stream. Hence, the motivation of the paper is unclear. Especially for real-world applications, due to real-time requirement, people usually just use the spatial stream. Hence, the current flow attack setting has limited usage. It is very important to show attack results on spatial stream as well. 

2. For FlowNet2, authors use gradient of the loss wrt the input images. However, FlowNet2 is a very large network consisting of 5 FlowNets. I am curious what the gradients will look like after the long back-propagation. Can authors comment on this by drawing a figure of magnitude distribution of gradients in the very early layers? 


Due to limited novelty, I recommend an initial rating of 5. 





Meta-review: With scores of 5, 4 and 3 the paper is just too far away from the threshold for acceptance.

Decision: rejected-papers