# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 3 Proposed Approach
### Two-Stream Interaction  
In the original two-stream architecture, a video is represented as an RGB frame and a stack of optical flow frames, which are separately fed into the appearance stream and motion stream. Each of the two processing paths generates the spatial and temporal features on its own to obtain the softmax predictions which are averaged in the last step to calculate the final results of classification. Since this architecture extracts the appearance and motion features in parallel and has no interaction before the final fusion, it is incapable of modeling the subtle spatiotemporal cues, which are of significant importance for distinguishing videos that have similar appearances or motion patterns. Figure 4: Structure of the residual block, which has a shortcut from the input layer directly connected to later layers. In our work, a video was first represented as \(N\) small segments, where \(N\) also corresponds to the time steps of the BiLSTM decoder. For each segment, we sampled \(t\) RGB frames and the corresponding \(t\) optical flow frames. One stack of RGB frames and optical flow frames covering one local video segment were fed into the appearance and motion streams, respectively, which had the same CNN structure for modeling the correlations of appearance and motion features between the same abstract levels of two streams. Several types of cross-stream connection [39] can be introduced between the identical abstraction levels of the two pathways for an earlier interaction as depicted in Figure 5. However, the performances of structures with a simple direct connection, such as those shown in Figure 5a,b, are inferior to others, which is imputed to the huge changes in distribution directly induced by the other stream. As depicted in Figure 5c, the additive connections injected from the motion stream to the appearance stream can be formalized as:  
\[x_{l+1}^{a}=f\big{(}x_{l}^{a}\big{)}+F\big{(}x_{l}^{a}+f\big{(}x_{l}^{m}\big{)},\ W_{l}^{a}\big{)} \tag{2}\]  
Here, \(x_{l}^{a}\) and \(x_{l}^{m}\) stand for the inputs of the \(l\)th layer of the appearance and motion streams, respectively, \(x_{l+1}^{a}\) stands for the inputs of the \((l+1)\) th layer of appearance stream, and \(W_{l}^{a}\) denotes the weight matrix of this unit in the appearance stream. \(f(x)\) denotes ReLU activation function, and \(F(x)\) denotes the operators of convolutional layers and ReLU activation layers shown in above figures. Then, during the backpropagation, given the inputs of the \((l+1)\)th layer of motion stream \(x_{l+1}^{m}\), the gradient of the loss function \(L\) of the appearance stream \(\partial L/\partial x_{l}^{a}\) and motion stream \(\partial L/\partial x_{l}^{m}\) can be calculated through the chain rules as:  
\[\frac{\partial L}{\partial x_{l}^{a}}=\frac{\partial L}{\partial x_{l+1}^{a}} \frac{\partial x_{l+1}^{a}}{\partial x_{l}^{a}}=\frac{\partial L}{\partial x_{ l+1}^{a}}\big{(}\frac{\partial f(x_{l}^{a})}{\partial x_{l}^{a}}+\frac{\partial}{ \partial x_{l}^{a}}F(x_{l}^{a}+f(x_{l}^{m}),\ W_{l}^{a})\big{)} \tag{3}\]  
\[\frac{\partial L}{\partial x_{l}^{m}}=\frac{\partial L}{\partial x_{l+1}^{m}} \frac{\partial x_{l+1}^{m}}{\partial x_{l}^{m}}+\frac{\partial L}{\partial x_ {l+1}^{a}}\frac{\partial}{\partial x_{l}^{a}}F(x_{l}^{a}+f(x_{l}^{m}),W_{l}^{ a}) \tag{4}\]  
Similarly, as depicted in Figure 5d, applying the multiplicative motion gating to two streams can be formalized as:  
\[x_{l+1}^{a}=f(x_{l}^{a})+F(x_{l}^{a}\odot f(x_{l}^{m}),W_{l}) \tag{5}\]  
Here, \(\odot\) denotes element-wise multiplication. Correspondingly, the gradient of the loss function \(L\) of the appearance stream in the backpropagation is formulated as:  
Figure 5: Several types of cross-stream connections for capturing spatial and temporal features simultaneously. (**a**–**d**) The connections of direction from motion to the appearance stream; (**e**) the bidirectional connections.