Title: REPRESENTATION FLOW FOR ACTION RECOGNITION

Abstract: In this paper, we propose a convolutional layer inspired by optical flow algorithms to learn motion representations. Our representation flow layer is a fully-differentiable layer designed to capture the 'flow' of any representation channel within a convolutional neural network for action recognition. Its parameters for iterative flow optimization are learned in an end-to-end fashion together with the other model parameters, maximizing the action recognition performance. Furthermore, we newly introduce the concept of learning 'flow of flow' representations by stacking multiple representation flow layers. We conducted extensive experimental evaluations, confirming its advantages over previous recognition models using traditional optical flows in both computational speed and performance.

1. INTRODUCTION:

Activity recognition is an important problem in computer vision with many societal applications including surveillance, robot perception, smart environment/city, and more. Use of video convolutional neural networks (CNNs) have become the standard method for this task, as they can learn more optimal representations for the problem. Two-stream networks (Simonyan & Zisserman, 2014) , taking both RGB frames and optical flow as input, provide state-of-the-art results and have been extremely popular. 3-D spatio-temporal CNN models, e.g., (Carreira & Zisserman, 2017) , with XYT convolutions also found that such two-stream design (RGB + optical flow) increases their accuracy. Abstracting both appearance information and explicit motion flow benefits the recognition.

However, optical flow is expensive to compute. It often requires hundreds of optimization iterations every frame, and causes learning of two separate CNN streams (i.e., RGB-stream and flow-stream). This requires significant computation cost and a great increase in the number of model parameters to learn. Further, this means that the model needs to compute optical flow every frame even during inference and run two parallel CNNs, limiting its real-time applications.

There were previous works to learn representations capturing motion information without using optical flow as input, such as motion feature networks (Lee et al., 2018) and ActionFlowNet (Ng et al., 2018) . However, although they were more advantageous in terms of the number of model parameters and computation speed, they suffered from inferior performance compared to two-stream models on public datasets such as Kinetics (Kay et al., 2017) and HMDB (Kuehne et al., 2011) . We hypothesize that the iterative optimization performed by optical flow methods produces an important feature that other methods fail to capture.

In this paper, we propose a CNN layer inspired by optical flow algorithms to learn motion representations for action recognition without having to compute optical flow. Our representation flow layer is a fully-differentiable layer designed to capture 'flow' of any representation channels within the model. Its parameters for iterative flow optimization are learned together with other model parameters, maximizing the action recognition performance. This is also done without having/training multiple network streams, reducing the number of parameters in the model. Further, we newly introduce the concept of learning 'flow of flow' representations by stacking multiple representation flow layers. We conduct extensive action classification experimental evaluation of where to compute optical flow and various hyperparameters, learning parameters, and fusion techniques.

2. RELATED WORKS:

Capturing motion and temporal information has been studied for activity recognition. Early, handcrafted approaches such as dense trajectories (Wang et al., 2011) captured motion information by tracking points through time. Many algorithms have been developed to compute optical flow as a way to capture motion in video (Fortun et al., 2015) . Other works have explored learning the ordering of frames to summarize a video in a single 'dynamic image' used for activity recognition (Bilen et al., 2016) .

Convolutional neural networks (CNNs) have been applied to activity recognition. Initial approaches explored methods to combine temporal information based on pooling or temporal convolution (Ng et al., 2015; Karpathy et al., 2014) . Other works have explored using attention to capture sub-events of activities (Piergiovanni et al., 2017) . Two-stream networks have been very popular: they take input of a single RGB frame (captures appearance information) and a stack of optical flow frames (captures motion information). Often, the two network streams of the model are separately trained and the final predictions are averaged together (Simonyan & Zisserman, 2014) . There were other two-stream CNN works exploring different ways to 'fuse' or combine the motion CNN with the appearance CNN (Feichtenhofer et al., 2016b; a) . There were also large 3D XYT CNNs learning spatio-temporal patterns (Carreira & Zisserman, 2017; Xie et al., 2017) , enabled by large video datasets such as Kinetics (Kay et al., 2017) . However, these approaches still rely on optical flow input to maximize their accuracies.

Recently, there have been works on learning motion representations. Fan et al. (2018) implemented the TVL-1 method using deep learning libraries to increase its computational speed and allow for learning some parameters. The result was fed to a two-stream CNN for the recognition. Several works explored learning a CNN to predict optical flow, which also can be used for action recognition (Dosovitskiy et al., 2015; Hui et al., 2018; Gao et al., 2018; Sun et al., 2018; Ng et al., 2018) . Lee et al. (2018) shifted features from sequential frames to capture motion in a non-iterative fashion. Sun et al. (2018) proposed an optical flow guided feature (OFF) by computing the gradients of representations and temporal differences, but required RGB, optical flow and RGB differences to achieve state-of-the-art performance.

Unlike prior works, our proposed model with the representation flow layers relies only on the RGB input, learning much fewer parameters while correctly representing motion with the iterative optimization. It is significantly faster than the video CNNs requiring optical flow input, while still performing as good as or even better than the two-stream models. It clearly outperforms existing motion representation methods including (Fan et al., 2018) in both speed and accuracy, which we experimentally confirm.

3. APPROACH:

Our method is a fully-differentiable convolutional layer inspired by optical flow algorithms. Unlike traditional optical flow methods, all the parameters of our method can be learned end-to-end, maximizing action recognition performance. Furthermore, our layer is designed to compute the 'flow' of any representation channels, instead of limiting its input to be traditional RGB frames.

3.1. REVIEW OF OPTICAL FLOW METHODS:

Before describing our layer, we briefly review how optical flow is computed. Optical flow methods are based on the brightness consistency assumption. That is, given sequential images I 1 , I 2 , a point x, y in I 1 is located at x + ∆x, y + ∆y in I 2 , or I 1 (x, y) = I 2 (x + ∆x, y + ∆y). These methods assume small movements between frames, so this can be approximated with a Taylor series: I 2 = I 1 + δI δx ∆x + δI δy ∆y, where u = [∆x, ∆y]. These equations are solved for u to obtain the flow, but can only be approximated due to the two unknowns.

The standard, variational methods for approximating optical flow (e.g., Brox (Brox et al., 2004) and TVL-1 (Zach et al., 2007) methods) take sequential images I 1 , I 2 as input. Variational optical flow methods estimate the flow field, u, using an iterative optimization method. The tensor u ∈ R 2×W ×H is the x and y directional flow for every location in the image. Taking two sequential images as input, I 1 , I 2 , the methods first compute the gradient in both x and y directions: ∇I 2 . The initial flow is set to 0, u = 0. Then ρ, the residual, can be computed. For efficiency, the constant part of ρ, ρ c is pre-computed:

ρ c = I 2 -∇ x I 2 • u x -∇ y I 2 • u y -I 1 (1)

The iterative optimization is then performed, each updating u:

EQUATION

Here θ controls the weight of the TVL-1 regularization term, λ controls the smoothness of the output and τ controls the time-step. These hyperparameters are manually set. p is the dual vector fields, which are used to minimize the energy. The divergence of p, or backward difference, is computed as:

divergence(p) = p x,i,j -p x,i-1,j + p y,i,j -p y,i,j-1 (6)

where p x is the x direction and p y is the y direction, and p contains all the spatial locations in the image.

The goal is to minimize the total variational energy:

EQUATION

Approaches run this iterative optimization for multiple input scales, from small to large, and use the previous flow estimate u to warp I 2 at the larger scale, providing a coarse-to-fine optical flow estimation. These standard approaches require multiple scales and warpings to obtain a good flow estimate, taking thousands of iterations.

3.2. REPRESENTATION FLOW LAYER:

Inspired by the optical flow algorithm, we design a fully-differentiable, learnable, convolutional representation flow layer by extending the general algorithm outlined above. The main differences are that (i) we allow the layer to capture flow of any CNN feature map, and that (ii) we learn its parameters including θ, λ, and τ as well as the divergence weights. We also make several key changes to reduce computation time: (1) we only use a single scale, (2) we do not perform any warping, and

(3) we compute the flow on a CNN tensor with a smaller spatial size. Multiple scale and warping are computationally expensive, each requiring many iterations. By learning the flow parameters, we can eliminate the need for these additional steps. Our method is applied on lower resolution CNN feature maps, instead of the RGB input, and is trained in an end-to-end fashion. This not only benefits its speed, but also allows the model to learn a motion representation optimized for activity recognition.

We note that the brightness consistency assumption can similarly be applied to CNN feature maps. Instead of capturing pixel brightness, we capture feature value consistency. This same assumption holds because CNNs are designed to be spatially invariant; i.e., they produce roughly the same feature value for the same object as it moves.

Given the input F 1 , F 2 , a single channel from sequential CNN feature maps (or input image), we compute the gradient by convolving the input feature maps with the Sobel filter:

∇F 2x = 1 0 -1 2 0 -2 1 0 -1 * F 2 , ∇F 2y = 1 2 1 0 0 0 -1 -2 -1 * F 2 (8)

We set u = 0, p = 0 initially, each having width and height matching the input, then we can compute ρ c = F 2 -F 1 . Next, we run the iterative optimization for a fixed number of iterations, following Eqs. 2-5. To compute the divergence, we zero-pad p on the first column (x-direction) or row (y-direction) then convolve it with weights, w x , w y to compute Eq. 6:

divergence(p) = p x * w x + p y * w y (9)

where initially w x = [-1 1] and w y = -1 1 . Note that these parameters are differentiable and can be learned with backpropagation. We compute ∇u as

∇u x = 1 0 -1 2 0 -2 1 0 -1 * u x , ∇u y = 1 2 1 0 0 0 -1 -2 -1 * u y (10)

Algorithm 1 shows the process of our representation flow layer. Our flow layer with multiple iterations could also be interpreted as having a sequence of convolutional layers with each layer behavior dependent on its previous layer. Note that our method is fully differentiable and allows for the learning of all parameters, including (τ, λ, θ) and the divergence weights (w x , w y ).

Algorithm 1 Method for the representation flow layer

function REPRESENTATIONFLOW(F 1 , F 2 ) u = 0, p = 0

Compute image/feature map gradients (Eq. 8)

ρ c = F 2 -F 1 for n iterations do ρ = ρ c + ∇ x F 2 • u x + ∇ y F 2 • u y v =      u + λθ∇F 2 ρ < -λθ|∇F 2 | 2 u -λθ∇F 2 ρ > λθ|∇F 2 | 2 u -ρ ∇F2 |F2| 2 otherwise u = v + θ • divergence(p) p = p+ τ θ ∇u 1+ τ θ |∇u|

end for return u end function Computing Flow-of-Flow Standard optical flow algorithms compute the flow for two sequential images. An optical flow image contains information about the direction and magnitude of the motion. Applying the flow algorithm directly on two flow images means that we are tracking pixels/locations showing similar motion in two consecutive frames. In practice, this typically leads to a worse performance due to inconsistent optical flow results and non-rigid motion. On the other hand, our representation flow layer is trained for the data, and is able to suppress such inconsistency and better abstract/represent motion by having multiple regular convolutional layers between the flow layers. Fig. 4 illustrates such design, which we confirm its benefits in the experiment section. By stacking multiple representation flow layers, our model is able to capture longer temporal intervals and consider locations with motion consistency.

Representation Flow within a CNN CNN feature maps may have hundreds or thousands of channels and our representation flow layer computes the flow for each channel, which can take significant time and memory. To address this, we apply a convolutional layer to reduce the number of channels from C to C before the flow layer (note that C is still significantly more than traditional optical flow algorithms, which are only applied to a single channel, greyscale images). For numerical stability, we normalize this feature map to be in [0, 255], matching standard image values. We found that the CNN features were quite small on average (< 0.5) and the TVL-1 algorithm default hyperparameters are designed for standard images values in [0, 255], thus we found this normalization step important. Using the normalized feature, we compute the flow and stack the x and y flows, resulting in 2C channels. Finally, we apply another convolutional layer to convert from 2C channels to C channels. This is passed to the remaining CNN layers for the prediction. We average predictions from many frames to classify each video, as shown in Fig. 1 . 

EQUATION

)

where v is the video, CN N is the classification CNN and c represents which of the K classes v belongs. That is, the parameters in our flow layers are trained together with the other layers, so that it maximizes the final classification accuracy. Where to compute flow? To determine where in the network to compute the flow, we compare applying our flow layer on the RGB input, after the first conv. layer, and after the each of the 5 residual blocks. The results are shown in Table 1 . We find that computing the flow on the input provides poor performance, similar to the performance of the flow-only networks, but there is a significant jump after even 1 layer, suggesting that computing the flow of a feature is beneficial, capturing both the appearance and motion information. However, after 4 layers, the performance begins to decline as the spatial information is too abstracted/compressed (due to pooling and large spatial receptive field size), and sequential features become very similar, containing less motion information. Note that our HMDB performance in this table is quite low compared to state-of-the-art For the following experiments, unless otherwise noted, we apply the layer after the 3rd residual block.

4. EXPERIMENTS:

What to learn? As our method is fully differentiable, we can learn any of the parameters, such as the kernels used to compute image gradients, the kernels for the divergence computation and even τ, λ, θ. In Table 2 , we compare the effects of learning different parameters. We find that learning the Sobel kernel reduces performance, but learning the divergence and τ, λ, θ is beneficial.

How many iterations for flow?:

To confirm that the iterations are important and determine how many we need, we experiment with various numbers of iterations. We compare the number of iterations needed for both learning (divergence+τ, λ, θ) and not learning parameters. The flow is computed after 3 residual blocks. The results are shown in Table 3 . We find that learning provides better performance with fewer iterations (similar to the finding in (Fan et al., 2018) ), and that iteratively computing the feature is important. We use 10 or 20 iterations in the remaining experiments as they provide good perforamnce and are fast.

Two-stream fusion? Two-stream CNNs fusing both RGB and optical flow features has been heavily studied (Simonyan & Zisserman, 2014; Feichtenhofer et al., 2016b) . Based on these works, we compare various ways of fusing RGB and our flow representation, shown in Fig. 2 . We compare no fusion, late fusion (i.e., separate RGB and flow CNNs) and addition/multiplication/concatenation fusion. In Table 4 , we compare different fusion methods for different locations in the network. We find that fusing RGB information is very important "when computing flow directly from RGB input". However, it is not as beneficial when computing the optical flow of representations as the CNN has already abstracted much appearance information away. We found that concatenation of the RGB and flow features perform poorly compared to the others. We do not use two-stream fusion in any other experiments, as we found that computing the representation flow after the 3rd residual block provides sufficient performance.

Flow-of-flow:

We can stack our layer multiple times, computing the flow-of-flow (FoF). This has the advantage of combining more temporal information into a single feature. Our results are shown in Table 5 . Applying the TVL-1 algorithm twice gives quite poor performance, as optical flow features do not really satisfy the brightness consistency assumption, as they capture magnitude and direction of motion (shown in Fig. 3 ). Applying our representation flow layer twice performs significantly better than TVL-1 twice, but still worse than our baseline of not doing so. However, we can add a convolutional layer between the first and second flow layer, flow-conv-flow (FcF), (Fig. 4 ), allowing the model to better learn longer-term flow representations. We find this performs best, as this intermediate layer is able to smooth the flow and produce a better input for the representation flow layer. However, we find adding a third flow layer reduces performance as the motion representation becomes unreliable, due to the large spatial receptive field size. Flow of 3D CNN Feature Since 3D convolutions capture some temporal information, we test computing our flow representation on features from a 3D CNN. As 3D CNNs are expensive to train, we follow the method of Carreira & Zisserman (2017) to inflate a ResNet-18 pretrained on ImageNet to a 3D CNN for videos. We also compare to the (2+1)D method of spatial conv. followed by temporal conv from (Xie et al., 2017) , which produces a similar feature combining spatial and temporal information. We find our flow layer increases performance even with 3D and (2+1)D CNNs already capturing some temporal information: Tables 6 and 7 . These experiments used 10 iterations and learning the flow parameters. In this, FcF was not used.

We also compared to the OFF (Sun et al., 2018) using (2+1)D and 3D CNNs. We observe that this method does not result in meaningful performance increases using CNNs that capture temporal information, while our approach does.

Comparison to other motion representations We compare to existing CNN-based motion representations methods to confirm that our iterative method to compute a representation is important. For these experiments, when available, we used code provided by the authors and otherwise implemented the methods ourselves. To better compare to existing works, we used (16×) 224 × 224 images. MFNet (Lee et al., 2018) captures motion by spatially shifting CNN feature maps, then summing the results, TVNet (Fan et al., 2018) applies a convolutional optical flow method to RGB inputs, and ActionFlowNet (Ng et al., 2018) trains a CNN to jointly predict optical flow and activity classes. We also compare to OFF (Sun et al., 2018) using only RGB inputs. Note that the HMDB performance in (Sun et al., 2018) is only reported using RGB, RGB-diff, and optical flow inputs, here we compare to RGB-only inputs. Our method, which applies the iterative optical flow method on CNN feature maps, performs the best. (Lee et al., 2018) 52.5 56.8 TVNet (Fan et al., 2018) 39.4 57.5 RGB-OFF (Sun et al., 2018) 55.6 56.9 Ours 61.1 65.4 Computation time We compare our representation flow to state-of-the-art two-stream approaches in terms of run-time and number of parameters. All timings were measured using a single Pascal Titan X GPU, for a batch of videos with size 32 × 224 × 224. The flow/two-stream CNNs include the time to run the TVL-1 algorithm (OpenCV GPU version) to compute the optical flow. All CNNs were based on the ResNet-34 architecture. As also shown in Table 9 , our method is significantly faster than two-stream models relying on TVL-1 or other optical flow methods, while performing similarly or better. The number of parameters our model has is half of its two-stream competitors (e.g., 21M vs. 42M in the case of our 2D CNN).

Comparison to state-of-the-arts We also compared our action recognition accuracies with the state-of-the-arts on Kinetics and HMDB. For this, we train our models using 32 × 224 × 224 inputs with the full kinetics dataset, using 8 V100s. We used the 2D ResNet-50 as the architecture. Based on our experiments, we applied our representation flow layer after the 3rd residual block, learned the hyperparameters and divergence kernels, and used 20 iterations. We also compare our flow-of-flow model. Following Szegedy et al. (2016) , the evaluation is performed using a running average of the parameters over time. Our results, shown in Table 9 , confirm that this approach outperforms existing models using only RGB as inputs and is competitive against expensive two-stream networks. Our model performs the best among those not using optical flow inputs (i.e., among the models only taking ∼600ms per video). The models requiring optical flow were more than 10 times slower.

5. CONCLUSION:

We introduced a learnable representation flow layer inspired by optical flow algorithms. We experimentally compared various forms of our layer to confirm that the iterative optimization and learnable parameters are important. Our model outperformed existing methods in both speed and accuracy on standard datasets. We also introduced the concept of 'flow of flow' to compute longer-term motion representations and showed it benefited performance.

:



Figures and Tables:

Figure 1: Illustration of a video-CNN with our representation flow layer. The CNN computes intermediate feature maps, and sequential feature maps are used as input to the flow layer. The outputs of the flow layer are used for prediction.

Figure 2: Different approaches to fusing RGB and flow information. (a) No fusion (b) Late fusion (c) The circle represents elementwise addition/multiplication or concatenation.

Figure 3: (a) RGB (b) Flow (c) Flow-of-Flow.

Figure 4: Illustration of how our model computes the FoF.

Figure 5: Comparing the results of (b) TVL-1 and (c) our learned flow when applied to RGB images.

Implementation details We implemented our representation flow layer in PyTorch and will release our code and models upon publication. As training CNNs on videos is computationally expensive, we used a subset of the Kinetics dataset(Kay et al., 2017) with 100k videos from 150 classes: Tiny-Kinetics. This allowed testing many models more quickly, while still having sufficient data to train large CNNs. For most experiments, we used ResNet-34(He et al., 2016) with input of size 16 × 112 × 112 (i.e., 16 frames with spatial size of 112). As training video CNNs is computationally expensive, we used this smaller input, which reduces performance, but allowed us to use larger batch sizes and run many experiments more quickly. Our final models are trained on standard 224 × 224 images. See appendix A for specific training details.

Table content: <table/>

Computing the optical flow representation after various number of CNN layers. Results are video classification accuracy on our Tiny-Kinetics and LowRes-HMDB51 datasets using 100 iterations to compute the flow representation.

Table content: <table><tr><td/><td colspan="2">Tiny-Kinetics LowRes-HMDB</td></tr><tr><td>RGB CNN</td><td>55.2</td><td>35.5</td></tr><tr><td>Flow CNN</td><td>35.4</td><td>37.5</td></tr><tr><td>Two-Stream CNN</td><td>57.6</td><td>41.5</td></tr><tr><td>Flow Layer on RGB Input</td><td>37.4</td><td>40.5</td></tr><tr><td>After Block 1</td><td>52.4</td><td>42.6</td></tr><tr><td>After Block 2</td><td>57.4</td><td>44.5</td></tr><tr><td>After Block 3</td><td>59.4</td><td>45.4</td></tr><tr><td>After Block 4</td><td>52.1</td><td>43.5</td></tr><tr><td>After Block 5</td><td>50.3</td><td>42.2</td></tr><tr><td colspan="3">Table 2: Comparison of learning different parameters. The flow was computed after Block 3 using</td></tr><tr><td>100 iterations.</td><td/><td/></tr><tr><td/><td colspan="2">Tiny-Kinetics LowRes-HMDB</td></tr><tr><td>None (all fixed)</td><td>59.4</td><td>45.4</td></tr><tr><td>Sobel kernels</td><td>58.5</td><td>43.5</td></tr><tr><td>Divergence (w x , w y )</td><td>60.2</td><td>46.4</td></tr><tr><td>τ, λ, θ</td><td>59.9</td><td>46.2</td></tr><tr><td>All</td><td>59.2</td><td>46.2</td></tr><tr><td>Divergence + τ, λ, θ</td><td>60.7</td><td>46.8</td></tr></table>

Effect of the number of iterations on our Tiny-Kinetics dataset for learning and not learning.

Table content: <table><tr><td/><td colspan="2">Not learned Learned</td></tr><tr><td>1 iteration</td><td>46.7</td><td>49.5</td></tr><tr><td>5 iterations</td><td>51.3</td><td>55.4</td></tr><tr><td>10 iterations</td><td>52.4</td><td>59.4</td></tr><tr><td>20 iterations</td><td>53.6</td><td>60.7</td></tr><tr><td>50 iterations</td><td>59.2</td><td>60.9</td></tr><tr><td>100 iterations</td><td>59.4</td><td>60.7</td></tr></table>

Different fusion methods for flow computed at different locations in the network on our Tiny-Kinetics dataset using 10 iterations and learning flow parameters.

Table content: <table><tr><td/><td colspan="3">RGB 1 Block 3 Blocks</td></tr><tr><td>None</td><td>37.4</td><td>52.4</td><td>59.4</td></tr><tr><td>Late</td><td>61.3</td><td>60.4</td><td>61.5</td></tr><tr><td>Add</td><td>59.7</td><td>57.2</td><td>56.5</td></tr><tr><td>Multiply</td><td>58.3</td><td>58.1</td><td>57.8</td></tr><tr><td colspan="2">Layer + Multiply 60.1</td><td>61.7</td><td>61.7</td></tr><tr><td>Concat</td><td>42.4</td><td>48.5</td><td>47.6</td></tr></table>

Computing the FoF representation. TVL-1 twice provides poor performance, using two flow layers with a conv. in between provides the best performance. Experiments used 10 iterations and learning flow parameters.

Table content: <table><tr><td/><td>Tiny-Kinetics</td></tr><tr><td>TVL-1 twice</td><td>12.2</td></tr><tr><td>Single Flow Layer</td><td>59.4</td></tr><tr><td>Flow-of-Flow</td><td>47.2</td></tr><tr><td>Flow-Conv-Flow (FcF)</td><td>62.3</td></tr><tr><td>Flow-Conv-Flow-Conv-Flow</td><td>56.5</td></tr></table>

Flow using 3D ResNet-18.

Table content: <table><tr><td/><td/><td colspan="2">Table 7: Flow using (2+1)D ResNet-18.</td></tr><tr><td/><td>Tiny-Kinetics</td><td/><td>Tiny-Kinetics</td></tr><tr><td>RGB 3D ResNet-18</td><td>54.6</td><td>RGB (2+1)D ResNet-18</td><td>53.4</td></tr><tr><td>TVL-1 3D ResNet-18</td><td>37.6</td><td>TVL-1 (2+1)D ResNet-18</td><td>36.3</td></tr><tr><td>Two-Stream 3D ResNet</td><td>57.5</td><td>Two-Stream (2+1)D ResNet</td><td>55.6</td></tr><tr><td>RGB-Only OFF (Sun et al., 2018)</td><td>54.8</td><td>RGB-Only OFF (Sun et al., 2018)</td><td>53.7</td></tr><tr><td>Input (RGB)</td><td>38.5</td><td>Input (RGB)</td><td>39.2</td></tr><tr><td>After Block 1</td><td>58.4</td><td>After Block 1</td><td>57.3</td></tr><tr><td>After Block 3</td><td>59.7</td><td>After Block 3</td><td>60.7</td></tr></table>

Comparisons to other CNN-based motion representations, using 10 iterations and learning flow parameters. This is without FcF and two-stream fusion.

Table content: <table><tr><td/><td colspan="2">Tiny-Kinetics HMDB</td></tr><tr><td>ActionFlownet (Ng et al., 2018)</td><td>51.8</td><td>56.2</td></tr><tr><td>MFNet</td><td/><td/></tr></table>

Comparison to the state-of-the-art action classifications. 'HMDB(+Kin)' means that the model was pre-trained on Kinetics before training/testing with HMDB. Missing results are due to those papers not reporting that setting.

Table content: <table><tr><td>Kinetics HMDB HMDB(+Kin) Run-time (ms)</td></tr></table>

Peer review (iclr2019):
 This paper proposes a "representation flow layer" for action recognition, to learn hidden motion representation in a CNN end-to-end. The authors also come up with a concept termed "flow-of-flow". The paper is interesting, but I have several concerns. 1. The concept of *flow-of-flow* needs more justification. As shown in Table 5, flow-of-flow performs much worse than a single flow layer. After adding a conv layer in between, the performance gets better. I assume this is because the estimated flow are noisy, directly compute flow on them again making the result worse. It is the same as applying TVL1 twice. Adding the extra conv layer can help smoothing the feature map. So how about justify another configuration "flow-conv"? I just want to see if *flow-of-flow* makes sense. Actually, as pointed out by authors, flow-of-flow usually performs worse due to inconsistent optical flow results. I think the reason it gets a little bit higher accuracy here is because over-fitting. So one more experiment of only "flow-conv", will help making the paper solid. 2. The caption of figure 5 need more clarification. Is it TVL1 twice and FoF? or simply TVL1 and flow? Because subfig (b) and (c) do not align well. I am assuming this is computing the flow twice. 3. I have several concerns for Table 9. (1) The authors do not provide results on HMDB dataset (2nd column). Is it because the performance is inferior to R(2+1)D or simply have no time before the deadline? This result is important, so please complete the table. (2) I don*t see obvious advantage of the proposed approach over R(2+1)D and S3D. Especially for S3D, the performance is similar but S3D is faster. So the contribution of the paper is limited. (3) If we compare the proposed approach to I3D, I3D is still better. Besides, there are several recent work reporting better performance on these two datasets. For example, ARTNet (Appearance-and-Relation Networks for Video Classification, CVPR18) report a 78.7 score on Kinetics. (4) The authors report their performance for 2D CNN and 2+1D CNN, but what about 3D CNN? It is also important to show. 4. Since another goal of this paper is to improve efficiency, so I want to see a comparison, at least a discussion, to two previous literature. These two literature are the first work to propose to learn optical flow inside a CNN for action recognition. In these two literature, the authors incorporate FlowNet-like architectures to compute optical flow and feed them as input to the temporal stream. Because the FlowNet-like architecture can be quite shallow, their efficiency (fps) is very high, and the performance is on par with state-of-the-art. (1) Yi Zhu, et al, Hidden Two-Stream Convolutional Networks for Action Recognition (2) Laura Sevilla-Lara, et al, On the Integration of Optical Flow and Action Recognition In conclusion, (1) In terms of novelty, the flow layer is adapted from TVNet, and the FoF concept need more clarification. (2) In terms of performance, there is no obvious advantage over previous work like S3D and R(2+1)D. And the experiments right now seem not complete. Hence, I recommend a initial rating of 5.