Introduction
Human vision can recognize video actions efficiently despite the variations of scenes and domains.
Convolutional neural networks (CNNs) [ 48 , 49 , 6 , 44 , 19 , 36 ] effectively
exploit
the power of modern computational devices and employ spatial-temporal filters to recognize actions, which
considerably outperform traditional models such as oriented filtering in space-time (HOG3D) [ 30 ] .
However, due to the high variations in space-time, the
state-of-the-art accuracy of action recognition is still far from being
satisfactory, compared with the success of 2D CNNs in image recognition [ 24 ] .

Recently, vision transformers such as
ViT [ 15 ] , and MViT [ 17 ] that are based on the self-attention [ 52 ] mechanism were
proposed to tackle the problems of image and video recognition,
and achieved impressive performance.
Instead of modeling pixels as
CNNs, transformers apply attentions on top of visual tokens.
The inductive bias of translation invariance
in CNNs makes it require less training data than attention-based transformers in general.
In contrast,
transformer has the advantage that it
can better leverage ‘big data’, leading to improved accuracy than CNNs.
We have witnessed
a rapid growth in video datasets [ 28 ] in recent years, which would make up for the shortcomings of data-hungry transformers.
The video data has not only grown in quantity from hundreds to millions of videos [ 42 ] , but also evolved from simple actions such as handshaking to complicated daily activities from the Kinetics-700 dataset [ 7 ] .
Meanwhile, transformers combined with low-level convolutional operations have been proposed [ 17 ] to further improve the efficiency and accuracy.

Due to the data-hungry nature of transformers, most transformer-based models for action recognition requires large-scale pre-training with image datasets such as ImageNet-21K [ 14 ] and JFT-3B [ 58 ] to achieve good performance.
This pre-training and fine-tuning training paradigm is time-consuming and it is not parameter-efficient, meaning that for each action dataset, a new model need to be trained end-to-end.
Different from large image datasets such as ImageNet-21K that covers a wide range of object classes, currently the most diverse action dataset, Kinetics-700, only contains 700 classes.
Each action dataset may be also limited to a certain topic or camera views.
For example, Moments-in-Time [ 42 ] only contains short actions that happen in three seconds and Something-Something-v2 [ 23 ] focuses on close-up camera view of person-object interactions.
These dataset biases might hinder models trained on a single dataset to generalize and be used in practical applications.
These challenges in action datasets make learning a general-purpose action model very difficult.
An ideal model should be able to cover a wide range of action classes meanwhile keeping the computation cost low.
However, simply combining all these datasets to train a joint model does not lead to good performance [ 38 ] .
In previous work [ 59 ] ,
the authors
have shown the benefit of training a joint model using multiple action datasets but their method requires large-scale image datasets such as ImageNet-21K [ 14 ] and JFT-3B [ 58 ] , which is not available to the research community.

In this paper,
we propose a general training paradigm for Multi -dataset Train ing of robust action recognition models, MultiTrain .
Our method is designed to learn robust and informative feature representations in a principled
approach, using the informative loss for regularization.
We do not assume the availability of large-scale image datasets pre-training (although one can certainly take advantage of that).
Since there are intrinsic relations between different classes across different action datasets (See Fig. 1 for examples of similar classes from two datasets), we propose a projection loss to mine such relations such that the whole network is trained to avoid over-fitting to certain dataset biases.
Finally, all proposed loss terms are weighted using learned parameters. Thus, no hyper-parameter tuning is needed.
Our empirical findings as shown in Table 1 indicate that our robust training method can consistently improve model backbone performance across multiple datasets.
We show that our model can achieve competitive results compared to state-of-the-art methods, even without large-scale image dataset pre-training, and with a lower computational cost.

The main contributions of this paper are thus three-fold:

To our knowledge,
this is the first work to introduce informative representation regularization into multi-dataset training for improving action recognition.

We propose an effective approach
to mine intrinsic class relations in multi-dataset training by introducing the projection loss.

Our method requires negligible computation
overhead during training and no additional computation during inference to the backbone network. Extensive experiments on
various
datasets suggest our method can
consistently
improve performance.