# A Multi-Scale Video Longformer Network for Action Recognition
## 4 Experiments
### Hyperparameter Tuning  
We utilized the pretrained ViL weights on ImageNet-1K, provided by EsViT repository [41] as the initial weights. We extended them to accommodate the temporal dimension for video tasks through inflation. Since training MSVL on the HMDB51 dataset leads to severe overfitting issues, resulting in poor accuracy, we first trained MSVL on the larger UCF101 dataset for 100 epochs with a warm-up of 10 epochs. Our MSVL achieved a top-1 accuracy of 97.6% on the UCF101 dataset. In Figure 6, we have plotted the top-1 accuracy, top-5 accuracy, and loss curves on the validation set. Table 2 presents the prediction accuracy of MSVL across various categories in the UCF101 dataset. While the accuracy for most categories reached 100%, the category with the lowest accuracy is \"WalkingWithDog\", standing at only 80.62%. Upon reviewing the confusion matrix of MSVL on the UCF101 dataset (Appendix A), we identified that 12 samples categorized as \"WalkingWithDog\" were inaccurately identified as \"HorseRiding\". This directly contributes to the low accuracy observed in the \"WalkingWithDog\" category. Firstly, this misstep can be attributed to the limited samples in the \"WalkingWithDog\" category, as there are only 123 video files in this category in the UCF101 dataset, fewer than the average number of videos per category, which is 130.59. This limitation hinders MSVL's capacity to learn sufficient features for precise recognition. Secondly, appearance features appear to dominate over motion characteristics in distinguishing between \"WalkingWithDog\" and \"HorseRiding\". Since MSVL is sensitive to actions and does not prioritize the extraction of target features; while these two categories share similar actions in some samples, this could lead to confusion in MSVL due to the similarity in motion features. We utilized weights pretrained on the UCF101 dataset to train MSVL on the HMDB51 dataset. The training process spanned 100 epochs, with an initial warm-up period of 10 epochs, and the learning rate was set to 1 \(\times 10^{-4}\). Consequently, MSVL achieved a top-1 accuracy of 72.94% on the HMDB51 dataset and the distribution of accuracies across different categories is presented in Figure 7. It can be observed that, although a relatively high overall accuracy is achieved, there are still some actions that are prone to confusion. For example, it sometimes classifies \"walk\" as \"run\", or \"stand\" as \"jump\". One reason is that actions with a smaller field-of-view tend to receive less attention, causing MSVL to focus more on irrelevant information. Another factor is that the HMDB51 dataset consists of clips extracted from movies and, in some clips, background actors perform different actions from the main actors, introducing significant background interference. Figure 6: Training results of MSVL on the UCF101 dataset. To explore the effectiveness of the 3D Longformer and the optimal configuration for MSVL, we conducted hyperparameter tuning on MSVL with different parameters and network configurations using the HMDB51 dataset. Firstly, we investigated the impact of different positional encodings. Then, we tested the use of different numbers of 3D Longformer Attention Blocks and MlpBlocks. Finally, we investigated the role of global tokens in network feature extraction by determining whether they were passed to the next stage. We present the accuracy and loss of MSVL with four distinct network configurations in Figure 8. In the legend, \"Rpe\" denotes relative 3D position encoding, \"Ape\" denotes absolute 3D position encoding, \"1191\" signifies the specific network architecture employed, and \"Glo\" indicates the practice of passing global tokens to the next stage.