# TBRNet: Two-Stream BiLSTM Residual Network for Video Action Recognition
## 4 Experiments
### Ablation Study
#### 4.2.2 Analysis of Cross-Stream Connections  
For the purpose of learning the correspondences between the appearance and motion signals from identical levels of abstraction in the two streams, we introduced cross-stream connections into the two-stream network. Generally, as shown in Figure 5, there are several possible variants of connections for interactions between pixels at the same position of the two processing pathways. We use the classification errors on UCF101 and HMDB51 to evaluate the performances of these different types of interaction; as shown in Table 2, a lower value of error represents a more appropriate model. In Table 2, \(\oplus\) denotes the additive operation, while \(\odot\) denotes the element-wise multiplicative operation. In addition, we use \(\leftarrow\) and \(\rightarrow\) to denote the direction of flow from the motion to the appearance stream and its opposite direction, respectively, and use \(\leftarrow\) to represent the bidirectional connection between the two streams. Firstly, we used the direct additive and multiplicative shortcut for cross-stream connections, as illustrated in Figure 5a,b. In the two-stream architecture equipped with the proposed Res-C3D, these kinds of structures allowed for the straightforward flow of the motion signals via the residual connections of each Res-C3D in both pathways, which, however, introduce large changes in input distribution into the appearance pathway. Those changes not only flow through the forward processing procedure to reach the deep layers but also are backpropagated to former layers after fusing with appearance features, thus resulting in the disturbance of the network capability of pattern encoding. \begin{table}
\begin{tabular}{c c c c} \hline \hline
**Approach** & & **UCF101** & **HMDB51** \\ \hline \multirow{2}{*}{Appearance stream} & C3D & 81.5 & 46.2 \\  & Res-C3D & 82.8 & 48.4 \\ \hline \multirow{2}{*}{Motion stream} & C3D & 85.9 & 53.3 \\  & Res-C3D & 86.1 & 57.0 \\ \hline \multirow{2}{*}{Two-stream} & C3D & 89.3 & 61.6 \\  & Res-C3D & 90.8 & 63.4 \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Comparison of classification accuracies (%) for the two-stream encoder constructed with C3D and the proposed Res-C3D on UCF101 and HMDB51. In other words, the straightforward shortcuts damaged the identity mapping of the original residual signal flow of the appearance Res-C3D, which increased the difficulty of optimization and increased the classification errors as listed in the first two rows of Table 2. We also note that the performance of the direct multiplicative shortcut connection is even worse compared with that of the additive shortcut connection. The explanation for this is that employing multiplicative operations on a straightforward connection amplifies the impacts of disturbing basic feature extraction on the two pathways. The results are similar when applying the connections from the appearance stream to the motion stream. Therefore, we continued the following experiments not based on the simple direct interactions but on the cross-stream residual connections illustrated in Figure 5c,d,e. Secondly, we compared the additive and multiplicative operations for cross-stream residual connections, which were introduced in Section 2. As we can see from Table 2, using multiplicative gating from the motion to the appearance stream led to errors of 7.53% and 36.69% on UCF101 and HMDB51, and additive gating achieved slightly worse results with errors of 9.04% and 39.91%. The multiplicative residual connections strengthen the information correspondences of the two pathways by modulating the gradient propagation with signals from both pathways. This type of reinforcement of two signals from each stream improves the ability of the two-stream encoder to learn spatiotemporal features. Thirdly, we evaluated the performances of employing different directions of cross-stream connections in our network.