Related Work
We review some work that is closest to ours.

CNNs and Vision Transformers. CNNs work as the standard backbones throughout computer vision tasks for image and video. Various effective convolutional neural architectures have been raised to improve the precision and efficiency ( e.g. , VGG [ 45 ] , ResNet [ 24 ] and DenseNet [ 26 ] ). Although CNNs are still the primary models for computer vision, the Vision Transformers have already
shown their enormous potential.
Vision Transformer (ViT [ 15 ] ) directly applies the architecture of Transformer on image classification and gets encouraging performance. ViT and its variants [ 2 , 37 , 4 , 17 , 40 , 54 ] achieve outstanding results in both image and video processing in recent years.

Action Recognition / Classification. The research of action recognition has advanced with both new datasets and new models.
One of the largest modern benchmarks for action recognition is the Kinetics dataset [ 28 ] .
The Kinetics dataset proposes a large
benchmark with more categories and more videos ( e.g. , 400 categories 160,000 clips in [ 28 ] and 700 categories in [ 7 ] ) as
more challenging
benchmarks compared to previous datasets like UCF-101 [ 47 ] .
The Moments-in-Time [ 42 ] (MiT) dataset provides a million short video clips that cover 305 action categories.
Note that it is infeasible
for
Kinetics and MiT datasets to
cover
all the possible actions in all possible scales.
For example, surveillance actions [ 10 , 8 ] are missing in the two datasets.
Many new approaches [ 50 , 60 , 39 , 20 , 55 , 36 , 10 , 8 , 27 , 9 , 35 ] have been carried out on these datasets, of which the SlowFast network [ 20 ] and MViT [ 17 ] obtain promising
performance.
We can see that the trend of action recognition in the last two decades is to collect
larger
datasets ( e.g. , Kinetics)
and
build models with a larger capacity.

Multi-dataset Co-Training. Previously, multi-dataset co-training has been explored in the image domain such as detection [ 62 , 53 ] and segmentation [ 32 ] .
Several works [ 43 , 11 , 46 , 25 ] were proposed to combine multiple video datasets for training.
Larger datasets often deliver
better results. Combining multiple
datasets to boost data size, and improve the final performance [ 22 ] , and the simultaneous use of multiple datasets is
also likely to alleviate
the damaging impact of dataset bias. OmniSource [ 16 ] utilizes
web images as part of the training dataset to expand the diversity of the training data to reduce dataset bias.
VATT [ 1 ] uses
additional multi-modal data for self-supversied pretraining and finetunes on downstream datasets.
CoVeR [ 59 ] combines image and video training even during the finetuning stage and reports significant performance boost compared to single-dataset training.
PolyViT [ 38 ] further extends to training with image, video and audio datasets using different sampling procedures.
In this paper, we propose a simple yet effective way (no multi-stage training, no complex dataset-wise sampling and hyper-parameter tuning) for multi-action-dataset training, without the use of any image or additional data from other modality.

Video Domain Generalization (VDG). Our work is also related but different from video domain generalization [ 61 ] .
The key distinction is that our goal is to train a single model on multiple related tasks (multiple action datasets) such that the model performs well on the same set of tasks, whereas VDG aims to generalize a model to unseen out-of-distribution target domain [ 56 , 11 , 12 , 13 , 51 ] .
These models still suffer from problem of parameter-inefficiency, meaning that separate models are needed for different target datasets.