## 3 The Proposed Encoder-Decoder Network
### Temporal Encoder
#### 3.2.1 Lstm  
Following [39], we adapt a 1-layer bidirectional LSTM (BLSTM), taking the vision feature vectors \(x_{t}\) as input and outputting a class probability \(y_{t}(\pi_{t})\), \(\pi_{t}\in\mathcal{A}\) for every frame \(t\). The BLSTM has 1024 cells in each direction. The overall network is trained together with CTC (see Section 3.4.2 for training detail). The output alphabet \(\mathcal{A}\) is therefore augmented with the CTC blank class label, and the decoding is performed with a beam search.  
\begin{table}
\begin{tabular}{c c c} \hline \hline
**Layer** & **I3D Res-50** & **Output Size** \\ \hline _res_1 & \(1\times 7\times 7\), 64 & \(L\times\frac{W}{2}\times\frac{H}{2}\) \\ _res_2 & \(\begin{bmatrix}1\times 1\times 1\), 64 \\ \(1\times 3\times 3\), 64 \\ \(1\times 1\times 1\), 256 \\ \end{bmatrix}\times 3\) & \(L\times\frac{W}{4}\times\frac{H}{4}\) \\ _res_3 & \(\begin{bmatrix}1\times 1\times 1\), 128 \\ \(1\times 3\times 1\), 128 \\ \(1\times 1\times 1\), 512 \\ \end{bmatrix}\times 4\) & \(L\times\frac{W}{8}\times\frac{H}{8}\) \\ _res_4 & \(\begin{bmatrix}1\times 1\times 1\), 256 \\ \(1\times 3\times 2\), 256 \\ \(1\times 1\times 1\), 024 \\ \end{bmatrix}\times 6\) & \(L\times\frac{W}{16}\times\frac{H}{16}\) \\ _res_5 & \(\begin{bmatrix}1\times 1\times 1\), 512 \\ \(1\times 3\times 3\), 512 \\ \(1\times 1\times 1\), 2048 \\ \end{bmatrix}\times 3\) & \(L\times\frac{W}{32}\times\frac{H}{8}\) \\ _fc_ & global pooling \(\rightarrow\) softmax & \(1\times 1\times C\) \\ \hline \hline \end{tabular}
\end{table}
Table 2: Network architecture of the I3D Res-50 backbone [37]. Residual blocks are shown in brackets, next to which is the number of repeated blocks in the stack. Each convolutional layer is followed by batch normalization and ReLU. Down-sampling is performed after each stack with a stride of 2 along dimensions of width and height _res_5 culminates with a global spatiotemporal pooling layer outputting a 512-dimensional feature vector, which is subsequently fed to a fully connected layer outputting final class probabilities through softmax activations. The dimension \(\mathcal{C}\) of the last fully connected layer is equal to the number of classes in the target dataset.  
Figure 5: Res-50 backbone, 2 nd, 3 rd, 4 th residual blocks are internally similar and are therefore omitted.  
\begin{table}
\begin{tabular}{c c c c} \hline \hline  & **BLSTM** & **Full Convolution** & **Transformer** \\ \hline Dimension & \(2\times 1024\) & \(15\times(Xepton\_Block+1536)\) & \((6+6)\times(512+2048+512)\) \\ \# Parameters & \(22\) M & \(35\) M & \(40\) M \\ Optimization & SGD, batch size \(=1\) & Adam, batch size \(=32\) & Adam, batch size \(=32\) \\ Momentum & \(0.9\) & \(-\) & \(-\) \\ Learning Rate & \(10^{-2}\to 10^{-4}\) & \(10^{-3}\to 10^{-4}\) & \(10^{-3}\to 10^{-4}\) \\ Dropout & \(0.3\) & \(0.8\) & \(0.1\) \\ Training Time/Batch & \(0.76\times 32\) & \(0.34\) s & \(0.41\) s \\ \({}^{*}\) Convergence (Iterations) & \(1.0\times 10^{6}\times 32\) & \(0.8\times 10^{6}\) & \(3.0\times 10^{6}\) \\ \({}^{*}\) Convergence (Clock Time) & \(9\) days & \(3.4\) days & \(14\) days \\ \hline \hline \end{tabular}  
* Statistics are collected on the Breakfast dataset [38] on a single GPU; however, convergence curves are generally consistent across the other datasets (Section 4.2) and are therefore omitted for presentation. The time to train the I3D Res-50 CNN (2 weeks) is also excluded from the statistics.  
\end{table}
Table 3: Structural and training details of the temporal backbones used in experiments.