Title: REPRESENTATION FLOW FOR ACTION RECOGNITION

Abstract: In this paper, we propose a convolutional layer inspired by optical flow algorithms to learn motion representations. Our representation flow layer is a fully-differentiable layer designed to capture the 'flow' of any representation channel within a convolutional neural network for action recognition. Its parameters for iterative flow optimization are learned in an end-to-end fashion together with the other model parameters, maximizing the action recognition performance. Furthermore, we newly introduce the concept of learning 'flow of flow' representations by stacking multiple representation flow layers. We conducted extensive experimental evaluations, confirming its advantages over previous recognition models using traditional optical flows in both computational speed and performance.

1. INTRODUCTION:

Activity recognition is an important problem in computer vision with many societal applications including surveillance, robot perception, smart environment/city, and more. Use of video convolutional neural networks (CNNs) have become the standard method for this task, as they can learn more optimal representations for the problem. Two-stream networks (Simonyan & Zisserman, 2014) , taking both RGB frames and optical flow as input, provide state-of-the-art results and have been extremely popular. 3-D spatio-temporal CNN models, e.g., (Carreira & Zisserman, 2017) , with XYT convolutions also found that such two-stream design (RGB + optical flow) increases their accuracy. Abstracting both appearance information and explicit motion flow benefits the recognition.

However, optical flow is expensive to compute. It often requires hundreds of optimization iterations every frame, and causes learning of two separate CNN streams (i.e., RGB-stream and flow-stream). This requires significant computation cost and a great increase in the number of model parameters to learn. Further, this means that the model needs to compute optical flow every frame even during inference and run two parallel CNNs, limiting its real-time applications.

There were previous works to learn representations capturing motion information without using optical flow as input, such as motion feature networks (Lee et al., 2018) and ActionFlowNet (Ng et al., 2018) . However, although they were more advantageous in terms of the number of model parameters and computation speed, they suffered from inferior performance compared to two-stream models on public datasets such as Kinetics (Kay et al., 2017) and HMDB (Kuehne et al., 2011) . We hypothesize that the iterative optimization performed by optical flow methods produces an important feature that other methods fail to capture.

In this paper, we propose a CNN layer inspired by optical flow algorithms to learn motion representations for action recognition without having to compute optical flow. Our representation flow layer is a fully-differentiable layer designed to capture 'flow' of any representation channels within the model. Its parameters for iterative flow optimization are learned together with other model parameters, maximizing the action recognition performance. This is also done without having/training multiple network streams, reducing the number of parameters in the model. Further, we newly introduce the concept of learning 'flow of flow' representations by stacking multiple representation flow layers. We conduct extensive action classification experimental evaluation of where to compute optical flow and various hyperparameters, learning parameters, and fusion techniques.

2. RELATED WORKS:

Capturing motion and temporal information has been studied for activity recognition. Early, handcrafted approaches such as dense trajectories (Wang et al., 2011) captured motion information by tracking points through time. Many algorithms have been developed to compute optical flow as a way to capture motion in video (Fortun et al., 2015) . Other works have explored learning the ordering of frames to summarize a video in a single 'dynamic image' used for activity recognition (Bilen et al., 2016) .

Convolutional neural networks (CNNs) have been applied to activity recognition. Initial approaches explored methods to combine temporal information based on pooling or temporal convolution (Ng et al., 2015; Karpathy et al., 2014) . Other works have explored using attention to capture sub-events of activities (Piergiovanni et al., 2017) . Two-stream networks have been very popular: they take input of a single RGB frame (captures appearance information) and a stack of optical flow frames (captures motion information). Often, the two network streams of the model are separately trained and the final predictions are averaged together (Simonyan & Zisserman, 2014) . There were other two-stream CNN works exploring different ways to 'fuse' or combine the motion CNN with the appearance CNN (Feichtenhofer et al., 2016b; a) . There were also large 3D XYT CNNs learning spatio-temporal patterns (Carreira & Zisserman, 2017; Xie et al., 2017) , enabled by large video datasets such as Kinetics (Kay et al., 2017) . However, these approaches still rely on optical flow input to maximize their accuracies.

Recently, there have been works on learning motion representations. Fan et al. (2018) implemented the TVL-1 method using deep learning libraries to increase its computational speed and allow for learning some parameters. The result was fed to a two-stream CNN for the recognition. Several works explored learning a CNN to predict optical flow, which also can be used for action recognition (Dosovitskiy et al., 2015; Hui et al., 2018; Gao et al., 2018; Sun et al., 2018; Ng et al., 2018) . Lee et al. (2018) shifted features from sequential frames to capture motion in a non-iterative fashion. Sun et al. (2018) proposed an optical flow guided feature (OFF) by computing the gradients of representations and temporal differences, but required RGB, optical flow and RGB differences to achieve state-of-the-art performance.

Unlike prior works, our proposed model with the representation flow layers relies only on the RGB input, learning much fewer parameters while correctly representing motion with the iterative optimization. It is significantly faster than the video CNNs requiring optical flow input, while still performing as good as or even better than the two-stream models. It clearly outperforms existing motion representation methods including (Fan et al., 2018) in both speed and accuracy, which we experimentally confirm.

3. APPROACH:

Our method is a fully-differentiable convolutional layer inspired by optical flow algorithms. Unlike traditional optical flow methods, all the parameters of our method can be learned end-to-end, maximizing action recognition performance. Furthermore, our layer is designed to compute the 'flow' of any representation channels, instead of limiting its input to be traditional RGB frames.

3.1. REVIEW OF OPTICAL FLOW METHODS:

Before describing our layer, we briefly review how optical flow is computed. Optical flow methods are based on the brightness consistency assumption. That is, given sequential images I 1 , I 2 , a point x, y in I 1 is located at x + ∆x, y + ∆y in I 2 , or I 1 (x, y) = I 2 (x + ∆x, y + ∆y). These methods assume small movements between frames, so this can be approximated with a Taylor series: I 2 = I 1 + δI δx ∆x + δI δy ∆y, where u = [∆x, ∆y]. These equations are solved for u to obtain the flow, but can only be approximated due to the two unknowns.

The standard, variational methods for approximating optical flow (e.g., Brox (Brox et al., 2004) and TVL-1 (Zach et al., 2007) methods) take sequential images I 1 , I 2 as input. Variational optical flow methods estimate the flow field, u, using an iterative optimization method. The tensor u ∈ R 2×W ×H is the x and y directional flow for every location in the image. Taking two sequential images as input, I 1 , I 2 , the methods first compute the gradient in both x and y directions: ∇I 2 . The initial flow is set to 0, u = 0. Then ρ, the residual, can be computed. For efficiency, the constant part of ρ, ρ c is pre-computed:

ρ c = I 2 -∇ x I 2 • u x -∇ y I 2 • u y -I 1 (1)

The iterative optimization is then performed, each updating u:

EQUATION

Here θ controls the weight of the TVL-1 regularization term, λ controls the smoothness of the output and τ controls the time-step. These hyperparameters are manually set. p is the dual vector fields, which are used to minimize the energy. The divergence of p, or backward difference, is computed as:

divergence(p) = p x,i,j -p x,i-1,j + p y,i,j -p y,i,j-1 (6)

where p x is the x direction and p y is the y direction, and p contains all the spatial locations in the image.

The goal is to minimize the total variational energy:

EQUATION

Approaches run this iterative optimization for multiple input scales, from small to large, and use the previous flow estimate u to warp I 2 at the larger scale, providing a coarse-to-fine optical flow estimation. These standard approaches require multiple scales and warpings to obtain a good flow estimate, taking thousands of iterations.

3.2. REPRESENTATION FLOW LAYER:

Inspired by the optical flow algorithm, we design a fully-differentiable, learnable, convolutional representation flow layer by extending the general algorithm outlined above. The main differences are that (i) we allow the layer to capture flow of any CNN feature map, and that (ii) we learn its parameters including θ, λ, and τ as well as the divergence weights. We also make several key changes to reduce computation time: (1) we only use a single scale, (2) we do not perform any warping, and

(3) we compute the flow on a CNN tensor with a smaller spatial size. Multiple scale and warping are computationally expensive, each requiring many iterations. By learning the flow parameters, we can eliminate the need for these additional steps. Our method is applied on lower resolution CNN feature maps, instead of the RGB input, and is trained in an end-to-end fashion. This not only benefits its speed, but also allows the model to learn a motion representation optimized for activity recognition.

We note that the brightness consistency assumption can similarly be applied to CNN feature maps. Instead of capturing pixel brightness, we capture feature value consistency. This same assumption holds because CNNs are designed to be spatially invariant; i.e., they produce roughly the same feature value for the same object as it moves.

Given the input F 1 , F 2 , a single channel from sequential CNN feature maps (or input image), we compute the gradient by convolving the input feature maps with the Sobel filter:

∇F 2x = 1 0 -1 2 0 -2 1 0 -1 * F 2 , ∇F 2y = 1 2 1 0 0 0 -1 -2 -1 * F 2 (8)

We set u = 0, p = 0 initially, each having width and height matching the input, then we can compute ρ c = F 2 -F 1 . Next, we run the iterative optimization for a fixed number of iterations, following Eqs. 2-5. To compute the divergence, we zero-pad p on the first column (x-direction) or row (y-direction) then convolve it with weights, w x , w y to compute Eq. 6:

divergence(p) = p x * w x + p y * w y (9)

where initially w x = [-1 1] and w y = -1 1 . Note that these parameters are differentiable and can be learned with backpropagation. We compute ∇u as

∇u x = 1 0 -1 2 0 -2 1 0 -1 * u x , ∇u y = 1 2 1 0 0 0 -1 -2 -1 * u y (10)

Algorithm 1 shows the process of our representation flow layer. Our flow layer with multiple iterations could also be interpreted as having a sequence of convolutional layers with each layer behavior dependent on its previous layer. Note that our method is fully differentiable and allows for the learning of all parameters, including (τ, λ, θ) and the divergence weights (w x , w y ).

Algorithm 1 Method for the representation flow layer

function REPRESENTATIONFLOW(F 1 , F 2 ) u = 0, p = 0

Compute image/feature map gradients (Eq. 8)

ρ c = F 2 -F 1 for n iterations do ρ = ρ c + ∇ x F 2 • u x + ∇ y F 2 • u y v =      u + λθ∇F 2 ρ < -λθ|∇F 2 | 2 u -λθ∇F 2 ρ > λθ|∇F 2 | 2 u -ρ ∇F2 |F2| 2 otherwise u = v + θ • divergence(p) p = p+ τ θ ∇u 1+ τ θ |∇u|

end for return u end function Computing Flow-of-Flow Standard optical flow algorithms compute the flow for two sequential images. An optical flow image contains information about the direction and magnitude of the motion. Applying the flow algorithm directly on two flow images means that we are tracking pixels/locations showing similar motion in two consecutive frames. In practice, this typically leads to a worse performance due to inconsistent optical flow results and non-rigid motion. On the other hand, our representation flow layer is trained for the data, and is able to suppress such inconsistency and better abstract/represent motion by having multiple regular convolutional layers between the flow layers. Fig. 4 illustrates such design, which we confirm its benefits in the experiment section. By stacking multiple representation flow layers, our model is able to capture longer temporal intervals and consider locations with motion consistency.

Representation Flow within a CNN CNN feature maps may have hundreds or thousands of channels and our representation flow layer computes the flow for each channel, which can take significant time and memory. To address this, we apply a convolutional layer to reduce the number of channels from C to C before the flow layer (note that C is still significantly more than traditional optical flow algorithms, which are only applied to a single channel, greyscale images). For numerical stability, we normalize this feature map to be in [0, 255], matching standard image values. We found that the CNN features were quite small on average (< 0.5) and the TVL-1 algorithm default hyperparameters are designed for standard images values in [0, 255], thus we found this normalization step important. Using the normalized feature, we compute the flow and stack the x and y flows, resulting in 2C channels. Finally, we apply another convolutional layer to convert from 2C channels to C channels. This is passed to the remaining CNN layers for the prediction. We average predictions from many frames to classify each video, as shown in Fig. 1 . 

EQUATION

)

where v is the video, CN N is the classification CNN and c represents which of the K classes v belongs. That is, the parameters in our flow layers are trained together with the other layers, so that it maximizes the final classification accuracy. Where to compute flow? To determine where in the network to compute the flow, we compare applying our flow layer on the RGB input, after the first conv. layer, and after the each of the 5 residual blocks. The results are shown in Table 1 . We find that computing the flow on the input provides poor performance, similar to the performance of the flow-only networks, but there is a significant jump after even 1 layer, suggesting that computing the flow of a feature is beneficial, capturing both the appearance and motion information. However, after 4 layers, the performance begins to decline as the spatial information is too abstracted/compressed (due to pooling and large spatial receptive field size), and sequential features become very similar, containing less motion information. Note that our HMDB performance in this table is quite low compared to state-of-the-art For the following experiments, unless otherwise noted, we apply the layer after the 3rd residual block.

4. EXPERIMENTS:

What to learn? As our method is fully differentiable, we can learn any of the parameters, such as the kernels used to compute image gradients, the kernels for the divergence computation and even τ, λ, θ. In Table 2 , we compare the effects of learning different parameters. We find that learning the Sobel kernel reduces performance, but learning the divergence and τ, λ, θ is beneficial.

How many iterations for flow?:

To confirm that the iterations are important and determine how many we need, we experiment with various numbers of iterations. We compare the number of iterations needed for both learning (divergence+τ, λ, θ) and not learning parameters. The flow is computed after 3 residual blocks. The results are shown in Table 3 . We find that learning provides better performance with fewer iterations (similar to the finding in (Fan et al., 2018) ), and that iteratively computing the feature is important. We use 10 or 20 iterations in the remaining experiments as they provide good perforamnce and are fast.

Two-stream fusion? Two-stream CNNs fusing both RGB and optical flow features has been heavily studied (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016b) . Based on these works, we compare various ways of fusing RGB and our flow representation, shown in Fig. 2 . We compare no fusion, late fusion (i.e., separate RGB and flow CNNs) and addition/multiplication/concatenation fusion. In Table 4 , we compare different fusion methods for different locations in the network. We find that fusing RGB information is very important "when computing flow directly from RGB input". However, it is not as beneficial when computing the optical flow of representations as the CNN has already abstracted much appearance information away. We found that concatenation of the RGB and flow features perform poorly compared to the others. We do not use two-stream fusion in any other experiments, as we found that computing the representation flow after the 3rd residual block provides sufficient performance.

Flow-of-flow:

We can stack our layer multiple times, computing the flow-of-flow (FoF). This has the advantage of combining more temporal information into a single feature. Our results are shown in Table 5 . Applying the TVL-1 algorithm twice gives quite poor performance, as optical flow features do not really satisfy the brightness consistency assumption, as they capture magnitude and direction of motion (shown in Fig. 3 ). Applying our representation flow layer twice performs significantly better than TVL-1 twice, but still worse than our baseline of not doing so. However, we can add a convolutional layer between the first and second flow layer, flow-conv-flow (FcF), (Fig. 4 ), allowing the model to better learn longer-term flow representations. We find this performs best, as this intermediate layer is able to smooth the flow and produce a better input for the representation flow layer. However, we find adding a third flow layer reduces performance as the motion representation becomes unreliable, due to the large spatial receptive field size. Flow of 3D CNN Feature Since 3D convolutions capture some temporal information, we test computing our flow representation on features from a 3D CNN. As 3D CNNs are expensive to train, we follow the method of Carreira & Zisserman (2017) to inflate a ResNet-18 pretrained on ImageNet to a 3D CNN for videos. We also compare to the (2+1)D method of spatial conv. followed by temporal conv from (Xie et al., 2017) , which produces a similar feature combining spatial and temporal information. We find our flow layer increases performance even with 3D and (2+1)D CNNs already capturing some temporal information: Tables 6 and 7 . These experiments used 10 iterations and learning the flow parameters. In this, FcF was not used.

We also compared to the OFF (Sun et al., 2018) using (2+1)D and 3D CNNs. We observe that this method does not result in meaningful performance increases using CNNs that capture temporal information, while our approach does.

Comparison to other motion representations We compare to existing CNN-based motion representations methods to confirm that our iterative method to compute a representation is important. For these experiments, when available, we used code provided by the authors and otherwise implemented the methods ourselves. To better compare to existing works, we used (16×) 224 × 224 images. MFNet (Lee et al., 2018) captures motion by spatially shifting CNN feature maps, then summing the results, TVNet (Fan et al., 2018) applies a convolutional optical flow method to RGB inputs, and ActionFlowNet (Ng et al., 2018) trains a CNN to jointly predict optical flow and activity classes. We also compare to OFF (Sun et al., 2018) using only RGB inputs. Note that the HMDB performance in (Sun et al., 2018) is only reported using RGB, RGB-diff, and optical flow inputs, here we compare to RGB-only inputs. Our method, which applies the iterative optical flow method on CNN feature maps, performs the best. (Lee et al., 2018) 52.5 56.8 TVNet (Fan et al., 2018) 39.4 57.5 RGB-OFF (Sun et al., 2018) 55.6 56.9 Ours 61.1 65.4 Computation time We compare our representation flow to state-of-the-art two-stream approaches in terms of run-time and number of parameters. All timings were measured using a single Pascal Titan X GPU, for a batch of videos with size 32 × 224 × 224. The flow/two-stream CNNs include the time to run the TVL-1 algorithm (OpenCV GPU version) to compute the optical flow. All CNNs were based on the ResNet-34 architecture. As also shown in Table 9 , our method is significantly faster than two-stream models relying on TVL-1 or other optical flow methods, while performing similarly or better. The number of parameters our model has is half of its two-stream competitors (e.g., 21M vs. 42M in the case of our 2D CNN).

Comparison to state-of-the-arts We also compared our action recognition accuracies with the state-of-the-arts on Kinetics and HMDB. For this, we train our models using 32 × 224 × 224 inputs with the full kinetics dataset, using 8 V100s. We used the 2D ResNet-50 as the architecture. Based on our experiments, we applied our representation flow layer after the 3rd residual block, learned the hyperparameters and divergence kernels, and used 20 iterations. We also compare our flow-of-flow model. Following Szegedy et al. (2016) , the evaluation is performed using a running average of the parameters over time. Our results, shown in Table 9 , confirm that this approach outperforms existing models using only RGB as inputs and is competitive against expensive two-stream networks. Our model performs the best among those not using optical flow inputs (i.e., among the models only taking ∼600ms per video). The models requiring optical flow were more than 10 times slower.

5. CONCLUSION:

We introduced a learnable representation flow layer inspired by optical flow algorithms. We experimentally compared various forms of our layer to confirm that the iterative optimization and learnable parameters are important. Our model outperformed existing methods in both speed and accuracy on standard datasets. We also introduced the concept of 'flow of flow' to compute longer-term motion representations and showed it benefited performance.

:



Figures and Tables:

Figure 1: Illustration of a video-CNN with our representation flow layer. The CNN computes intermediate feature maps, and sequential feature maps are used as input to the flow layer. The outputs of the flow layer are used for prediction.

Figure 2: Different approaches to fusing RGB and flow information. (a) No fusion (b) Late fusion (c) The circle represents elementwise addition/multiplication or concatenation.

Figure 3: (a) RGB (b) Flow (c) Flow-of-Flow.

Figure 4: Illustration of how our model computes the FoF.

Figure 5: Comparing the results of (b) TVL-1 and (c) our learned flow when applied to RGB images.

Implementation details We implemented our representation flow layer in PyTorch and will release our code and models upon publication. As training CNNs on videos is computationally expensive, we used a subset of the Kinetics dataset(Kay et al., 2017) with 100k videos from 150 classes: Tiny-Kinetics. This allowed testing many models more quickly, while still having sufficient data to train large CNNs. For most experiments, we used ResNet-34(He et al., 2016) with input of size 16 × 112 × 112 (i.e., 16 frames with spatial size of 112). As training video CNNs is computationally expensive, we used this smaller input, which reduces performance, but allowed us to use larger batch sizes and run many experiments more quickly. Our final models are trained on standard 224 × 224 images. See appendix A for specific training details.

Table content: <table/>

Computing the optical flow representation after various number of CNN layers. Results are video classification accuracy on our Tiny-Kinetics and LowRes-HMDB51 datasets using 100 iterations to compute the flow representation.

Table content: <table><tr><td/><td colspan="2">Tiny-Kinetics LowRes-HMDB</td></tr><tr><td>RGB CNN</td><td>55.2</td><td>35.5</td></tr><tr><td>Flow CNN</td><td>35.4</td><td>37.5</td></tr><tr><td>Two-Stream CNN</td><td>57.6</td><td>41.5</td></tr><tr><td>Flow Layer on RGB Input</td><td>37.4</td><td>40.5</td></tr><tr><td>After Block 1</td><td>52.4</td><td>42.6</td></tr><tr><td>After Block 2</td><td>57.4</td><td>44.5</td></tr><tr><td>After Block 3</td><td>59.4</td><td>45.4</td></tr><tr><td>After Block 4</td><td>52.1</td><td>43.5</td></tr><tr><td>After Block 5</td><td>50.3</td><td>42.2</td></tr><tr><td colspan="3">Table 2: Comparison of learning different parameters. The flow was computed after Block 3 using</td></tr><tr><td>100 iterations.</td><td/><td/></tr><tr><td/><td colspan="2">Tiny-Kinetics LowRes-HMDB</td></tr><tr><td>None (all fixed)</td><td>59.4</td><td>45.4</td></tr><tr><td>Sobel kernels</td><td>58.5</td><td>43.5</td></tr><tr><td>Divergence (w x , w y )</td><td>60.2</td><td>46.4</td></tr><tr><td>τ, λ, θ</td><td>59.9</td><td>46.2</td></tr><tr><td>All</td><td>59.2</td><td>46.2</td></tr><tr><td>Divergence + τ, λ, θ</td><td>60.7</td><td>46.8</td></tr></table>

Effect of the number of iterations on our Tiny-Kinetics dataset for learning and not learning.

Table content: <table><tr><td/><td colspan="2">Not learned Learned</td></tr><tr><td>1 iteration</td><td>46.7</td><td>49.5</td></tr><tr><td>5 iterations</td><td>51.3</td><td>55.4</td></tr><tr><td>10 iterations</td><td>52.4</td><td>59.4</td></tr><tr><td>20 iterations</td><td>53.6</td><td>60.7</td></tr><tr><td>50 iterations</td><td>59.2</td><td>60.9</td></tr><tr><td>100 iterations</td><td>59.4</td><td>60.7</td></tr></table>

Different fusion methods for flow computed at different locations in the network on our Tiny-Kinetics dataset using 10 iterations and learning flow parameters.

Table content: <table><tr><td/><td colspan="3">RGB 1 Block 3 Blocks</td></tr><tr><td>None</td><td>37.4</td><td>52.4</td><td>59.4</td></tr><tr><td>Late</td><td>61.3</td><td>60.4</td><td>61.5</td></tr><tr><td>Add</td><td>59.7</td><td>57.2</td><td>56.5</td></tr><tr><td>Multiply</td><td>58.3</td><td>58.1</td><td>57.8</td></tr><tr><td colspan="2">Layer + Multiply 60.1</td><td>61.7</td><td>61.7</td></tr><tr><td>Concat</td><td>42.4</td><td>48.5</td><td>47.6</td></tr></table>

Computing the FoF representation. TVL-1 twice provides poor performance, using two flow layers with a conv. in between provides the best performance. Experiments used 10 iterations and learning flow parameters.

Table content: <table><tr><td/><td>Tiny-Kinetics</td></tr><tr><td>TVL-1 twice</td><td>12.2</td></tr><tr><td>Single Flow Layer</td><td>59.4</td></tr><tr><td>Flow-of-Flow</td><td>47.2</td></tr><tr><td>Flow-Conv-Flow (FcF)</td><td>62.3</td></tr><tr><td>Flow-Conv-Flow-Conv-Flow</td><td>56.5</td></tr></table>

Flow using 3D ResNet-18.

Table content: <table><tr><td/><td/><td colspan="2">Table 7: Flow using (2+1)D ResNet-18.</td></tr><tr><td/><td>Tiny-Kinetics</td><td/><td>Tiny-Kinetics</td></tr><tr><td>RGB 3D ResNet-18</td><td>54.6</td><td>RGB (2+1)D ResNet-18</td><td>53.4</td></tr><tr><td>TVL-1 3D ResNet-18</td><td>37.6</td><td>TVL-1 (2+1)D ResNet-18</td><td>36.3</td></tr><tr><td>Two-Stream 3D ResNet</td><td>57.5</td><td>Two-Stream (2+1)D ResNet</td><td>55.6</td></tr><tr><td>RGB-Only OFF (Sun et al., 2018)</td><td>54.8</td><td>RGB-Only OFF (Sun et al., 2018)</td><td>53.7</td></tr><tr><td>Input (RGB)</td><td>38.5</td><td>Input (RGB)</td><td>39.2</td></tr><tr><td>After Block 1</td><td>58.4</td><td>After Block 1</td><td>57.3</td></tr><tr><td>After Block 3</td><td>59.7</td><td>After Block 3</td><td>60.7</td></tr></table>

Comparisons to other CNN-based motion representations, using 10 iterations and learning flow parameters. This is without FcF and two-stream fusion.

Table content: <table><tr><td/><td colspan="2">Tiny-Kinetics HMDB</td></tr><tr><td>ActionFlownet (Ng et al., 2018)</td><td>51.8</td><td>56.2</td></tr><tr><td>MFNet</td><td/><td/></tr></table>

Comparison to the state-of-the-art action classifications. 'HMDB(+Kin)' means that the model was pre-trained on Kinetics before training/testing with HMDB. Missing results are due to those papers not reporting that setting.

Table content: <table><tr><td>Kinetics HMDB HMDB(+Kin) Run-time (ms)</td></tr></table>

Peer review (iclr2019):
 Summary: This paper presents an end-to-end learnable deep network for action recognition that estimates "optical flow" within the neural network layers. The "flow" is quoted as it may not be the optical flow in traditional sense over image frames, but could be the "flow" over the intermediate channels of a layer from the two frames. The paper further extends this idea to compute the flow of flow. For the flow computation, the paper uses the standard TV-L1 scheme, but its steps are being done within the network and thus could be back propagated. Finally, the entire framework is trained on a loss defined over action classification. Experiments are presented on Kinetics and HMDB datasets, and show good performance. Strengths: I think the main strength of this paper is its competitive performances in Table 9 against RGB only methods. There are also various analysis presented on several architectural choices for the fusion with the flow layer. Weaknesses: 1) I think the main weakness of this paper is its lack of any significant novelty, or inadequate coverage of state of the art in deep learning based flow estimation. For example, there is prior work such as FlowNet or FlowNet2.0 that could compute optical flow in an end-to-end manner. This paper does not cite these prior works and in the context of which the novelty as claimed in the paper is not substantial. 2) Another important weakness is the lack of any convincing argument on why it is a good idea to compute/apply an optical flow algorithm on the intermediate feature maps of a network? How is the fundamental assumption of flow -- the brightness constancy -- applicable to such intermediate layers? 3) Further, while the idea of proposing the deep variant of flow was to avoid the computational expense of an otherwise out-of-the-box flow algorithm, the paper ultimately has to resort to several heuristic workarounds such as low-resolution of the inputs, or reducing the iterations of flow optimization, etc. to make the model practical. The flow estimation of the feature maps also require trimming down the number of channels in a layer. 4) It is surprising to see that even after such heuristic workarounds to compute an approximate flow, the final performance of the model is compelling; which needs more analysis to understand where precisely is the proposed scheme gaining in performance even with these approximations in comparison to an otherwise accurate external optical flow scheme? 5) Finally, given that the paper proposes to optimize the model against the action classification loss, it is unclear to me if the proposed flow layers are in fact learning anything related to flow, or why should they learn flow, or for that matter, if the proposed iterative scheme is even necessary? If learning flow is important, shouldn*t there be some intermediate objective that ensures that the flow layers do learn flow, using a suitable loss? For example, as in the FlowNet2.0 paper. Minor comments: 1) The section 3.1 on optical flow methods could be improved to be more intuitive. For example, it is unclear what .rho_c is capturing, and Eq.(2) seems to be just |I_2-I_1|. What precisely are the roles of .lambda, .theta etc? How is the second term .lambda |I_1-I_2| useful in (7) ? Isn*t it a constant? 2) More details and discussions need to be added to the comparisons against state of the art in Table 9. Why are the HMDB results missing from the middle column? Overall, I think the paper has some interesting results, however the method is unconvincing/unclear why it should work nor its novelty commendable.