RiteeshGrandhi/sciclaimeval-shared-task huggingface dataset:
Method
Our method is built upon the backbone of
the improved Multi-scale Vision Transformers (MViTv2) [ 34 , 17 ] .
Note that
our approach works with any action recognition backbones.
Given videos from multiple datasets during training, the model backbone takes the video frames and produces feature embeddings for each video.
The same number of Multi-layer Perceptron (MLP) as the datasets are constructed as model heads to predict action classes for each dataset.
To facilitate robust cross-dataset training, we propose two loss terms, namely, the informative loss and projection loss .
The informative loss aims to maximize the embeddings’ representative power.
The projection loss, with the help of multiple cross-dataset projection layers, guides the model to learn intrinsic relations between classes of different dataset, hence the model heads can be trained jointly.
See Fig. 1 for an overview of our framework.
In this section, we first briefly describe the MViTv2 backbone design, and then present our proposed robust cross-dataset training paradigm.
The MViTv2 Backbone
Our model is based on the improved multi-scale vision transformers (MViTv2) [ 17 , 34 ] , which learns a hierarchy from dense (in space) and simple (in channels) to coarse and complex features.
The series of work of vision transformers [ 15 ] (ViTs) follows the basic self-attention architecture [ 52 ] originally proposed for machine translation.
The key component of the MViTv1 model [ 17 ] is the Multi Head Pooling Attention (MHPA), which pools the sequence of latent tensors to reduce the spatial or temporal dimension of the feature representations.
In MViTv2 [ 34 ] , a residual connection in MHPA for the pooled query tensor and a decomposed relative position embedding 1 1 1 We did not implement this part as the code were not available at the time of writing (March 2022). are added.
In this paper, we use 3D convolution as the pooling operation.
Please refer to supplemental material for a visualization of the MViTv2 block.
Each MViTv2 block consists of a multi-head pooling attention layer (MHPA) and a multi-layer perceptron (MLP), and the residual connections are built in each layer.
The feature of each MViTv2 block is computed by:

\begin{split}X_{1}&={\rm MHPA}(LN(X))+Pool(X)\ Block(X)&={\rm MLP}(LN(X_{1}))+X_{1},\end{split} (1)

where X is the input tensor to each block.
Multiple MViTv2 blocks are grouped into stages to reduce the spatial dimension while increase the channel dimension.
The full backbone architecture
is listed in supplementary material.

Classification head. For the action recognition problem, the model produces C-class classification logits by first averaging the feature tensor from the last stage along the spatial-temporal dimensions (we do not use the [CLASS] token in our transformer implementation), denoted as \mathbf{z}\in\mathbb{R}^{d} .
A linear classification layer is then applied on the averaged feature tensor to produce the final output, \mathbf{y}=\mathbf{W_{\rm out}}\mathbf{z}\in\mathbb{R}^{C} .

Multi-dataset training paradigm. In general, to facilitate multi-dataset training of K datasets, the same number of classification heads are appended to the feature embeddings.
The k -th dataset classification output is defined as \mathbf{Y}_{k}=h_{k}(\mathbf{Z};\mathbf{W}_{k})\in\mathbb{R}^{B\times C} ,
where h_{k} can
be a linear layer or a MLP and \mathbf{W}_{k} is the layer parameter.
Experiments
In this section, to demonstrate the efficacy of our training framework, we carry out experiments on five action recognition datasets, including Kinetics-400 [ 28 ] , Something-Something-v2 [ 23 ] , Moments-in-Time [ 42 ] , Activitynet [ 5 ] and Kinetics-700 [ 7 ] .
The action recognition task is defined to be a classification task given a trimmed video clip.
Unlike previous works [ 38 , 59 ] , we do not initialize our model using ImageNet [ 14 ] since it consumes more computation. Please refer to the supplementary material for detailed comparison between train from scratch recipe and from ImageNet.
In the experiments, we aim to showcase
that our method can achieve significant performance improvement with minimal computation overhead compared to baselins.
Experimental Setup
Datasets. We evaluate our method on five datasets.
Kinetics-400 [ 28 ] (K400) consists of about 240K training videos and 20K validation videos in 400 human action classes.
The videos are about 10 seconds long.
Kinetics-700 [ 7 ] (K700) extends the action classes to 700 with 545K training and 35K validation videos.
The Something-Something-v2 (SSv2) [ 23 ] dataset contains person-object interactions, which emphasizes temporal modeling.
SSv2 includes 168K videos for training and 24K videos for evaluation on 174 action classes.
The Moments-in-Time (MiT) dataset is one of the largest action dataset with 727K training and 30k validation videos.
MiT videos are mostly short 3-second clips.
The ActivityNet dataset [ 5 ] (ActNet) originally contains untrimmed videos with temporal annotations of 200 action classes.
We cut the annotated segments of the videos into 10-second long clips and split the dataset into 107K training and 16K testing.
Following previous works [ 20 , 59 ] , we follow the standard dataset split and report top-1/top-5 classification accuracy on the test split for all datasets.
We conduct two sets of experiments, namely, “K400, MiT, SSv2, ActNet”, and “K700, MiT, SSv2, ActNet”.

Implementation. Our backbone model utilizes MViTv2 as described in Section 3.1 .
Our models are trained from scratch with random initialization, without using any pre-training (same as in [ 20 ] and different from previous works [ 59 , 38 ] that require large-scale image dataset pre-training like ImageNet-21K [ 14 ] or JFT-3B [ 58 ] ).
We follow standard dataset splits as previous works [ 34 , 20 , 54 ] .
See more details in the supplementary material.

Baselines. PolyViT [ 38 ] utilizes multi-task learning on image, video and audio datasets to improve vision transformer performance. The backbone they used are based on ViT-ViViT [ 2 ] .
Similarly, VATT [ 1 ] utilizes additional multi-modal data for self-supversied pretraining and finetunes on downstream datasets. The backbone network is based on ViT [ 15 ] .
CoVER [ 59 ] is a recently proposed co-training method that includes training with images and videos simultaneously. Their model backbone is based on TimeSFormer [ 4 ] .
We also compare our method with other recent models trained using large-scale image datasets.
See Table 1 and Table 2 for the full list.
Main Results
We summarize our method’s performance in Table 1 and Table 2 .
We train our model jointly on MiT, SSv2, ActNet and two versions of the Kinetics datasets.

We first compare our method with the original MViTv2 backbone in Table 1 .
“MViTv2 w/ abs. pos.” means MViTv2 model with absolute positional embedding, which is taken from Table A.6 of MViTv2 paper [ 34 ] and it is (almost) the same as our model implementation.
We can not achieve the same accuracy with the same recipe as MViTv2, which may be due to differences in the Kinetics dataset (missing some videos, etc. See supplementary material for full dataset statistics).
PolyViT [ 38 ] is trained jointly with multiple image, audio and video datasets.
We list the larger ones.
We train our baseline model on the training set of each dataset to investigate the baseline performance.
As we see, after adding robust joint training proposed in this paper, performance on each dataset has increased by 2.1%, 3.1%, 1.9% and 5.9% on K400, MiT, SSv2, ActivityNet, respectively in terms of top-1 accuracy.
Note that our method achieves such improvement withtout large-scale image pre-training and additional inference computational cost.

We then compare our method with state-of-the-art on these datasets.
We train a higher resolution model with larger spatial inputs (312p) and achieves better performance compared to recent multi-dataset training methods, CoVER [ 59 ] and PolyVit [ 38 ] , on Kinetics-400, and significantly better on MiT and SSv2, as shown in Table 1 .
Note that our model does not use any image training datasets, and our model computation cost is only a fraction of the baselines.
We also show that our performance boost does not come from the additional training dataset of ActivityNet in Table 3 .

Our method also achieves competitive results compared to state-of-the-art models trained with large-scale image dataset (ImageNet-21K [ 14 ] ).
Compared to a recent method, MTV-B [ 54 ] , our method is able to achieve significantly better top-1 accuracy across Kinetics-400, MiT, SSv2 by 0.8%, 1.4%, 0.8%, respectively, at half of the computation cost and without large-scale pre-training.
Note that our model is parameter-efficient, while multiple MTV-B models need to be trained and tested on these datasets separately.
Our method can achieve better performance with a deeper base backbone or larger resolution inputs but we have not tested due to limitation of computation resources.

We then compare our method on the Kinetics-700, MiT, SSv2 and ActivityNet training with baselines.
Our parameter-efficient model can achieve better performance than MTV-B [ 54 ] at one-fifth of the computation cost.
With a larger resolution model at 312p, we achieves significantly better performance than the baseline across Kinetics-400, MiT, SSv2 by 2.2%, 4.9%, 3.4%, respectively.
Discussion
By multi-dataset training transformers on various datasets, we obtain competitive results on multiple action datasets, without large-scale image datasets pre-training.
Our method, MultiTrain , is parameter-efficient and does not require hyper-parameter tuning.
Current limitations of our experiments are that we have not tried co-training with image datasets such as ImageNet-21K [ 14 ] .
Hence we do not know how much performance gain that would entail.
We plan to explore this in future work.
In addition, we have not tried training larger model with FLOPs on par with state-of-the-art or other backbone architectures ( e.g. , CNNs) due to limitation of our computational resources.
Hence we are not sure how our algorithm would behave with these models.
We have not explicitly explored how temporal modeling could benefit from multi-action-dataset training, which we leave for future work.
Although our model is trained on multiple datasets, potential dataset biases can still cause negative societal impact in real-world deployment, as the datasets we have do not fully represent all aspects of human actions.
Conclusion
In this paper, we present MultiTrain , a robust multi-dataset training approach that maximizes information content of representation and learns intrinsic relations between individual datasets.
Our method can train parameter-efficient models that perform well across multiple datasets.
Acknowledgement
This work was in part supported by Foshan HKUST Projects (FSUST21-FYTRI01A, FSUST21-FYTRI02A).
C. Shen’s participation was in part supported by a major grant from Zhejiang Provincial Government.
Checklist
For all authors…

Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? [Yes]

Did you describe the limitations of your work? [Yes] See Section 4.4

Did you discuss any potential negative societal impacts of your work? [Yes] See Section 4.4

Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]

If you are including theoretical results…

Did you state the full set of assumptions of all theoretical results? [N/A]

Did you include complete proofs of all theoretical results? [N/A]

If you ran experiments…

Did you include the code, data, and instructions needed to reproduce the main experimental results (either in the supplemental material or as a URL)? [No] We plan to open-source our code upon paper acceptance. We have provided detailed implementation instructions in the main text and in the supplemental material.

Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4 and the supplemental material.

Did you report error bars (e.g., with respect to the random seed after running experiments multiple times)? [No] We follow standard data splits and evaluation protocol as previous works. See Section 4 .

Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See supplemental material.

If you are using existing assets (e.g., code, data, models) or curating/releasing new assets…

If your work uses existing assets, did you cite the creators? [Yes]

Did you mention the license of the assets? [No] We used open-source (Apache-2.0 licensed) MViTv2 code and standard public datasets

Did you include any new assets either in the supplemental material or as a URL? [No]

Did you discuss whether and how consent was obtained from people whose data you’re using/curating? [No]

Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 4.4

If you used crowdsourcing or conducted research with human subjects…

Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]

Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, if applicable? [N/A]

Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
Figure 1: Overview of our multi-dataset training framework. We propose to utilize the intrinsic relations between classes across different action datasets. As we can see, the two video examples from Kinetics and Moments-in-Time datasets, respectively, show that samples from these two classes can be used to train both classification heads.
The videos from multiple action datasets are input to the MViTv2 (Section 3.1 ) backbone and the model is trained jointly.
The informative loss is applied to maximize the information content of the embedding from the backbone and the projection loss is applied to learn the intrinsic relations (Section 3.2 ).; Overview of our multi-dataset training framework. We propose to utilize the intrinsic relations between classes across different action datasets. As we can see, the two video examples from Kinetics and Moments-in-Time datasets, respectively, show that samples from these two classes can be used to train both classification heads.
The videos from multiple action datasets are input to the MViTv2 (Section 3.1 ) backbone and the model is trained jointly.
The informative loss is applied to maximize the information content of the embedding from the backbone and the projection loss is applied to learn the intrinsic relations (Section 3.2 ).
Comparison with state-of-the-art on Kinetics-400, Moments-in-Time, Something-something-v2 and ActivityNet.
We divide the baselines into two groups based on whether they are parameter-efficient.
We report top-1/top-5 accuracy for each dataset.
The bold numbers and underlined are ranked first and second, respectively.
The FLOPs computation is for a single video clip input.
Training data:
(a) ImageNet-21K;
(b) ImageNet-1K;
(c) AudioSet [ 21 ] ;
(d) HowTo100M [ 41 ] ;
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Comparison with state-of-the-art on Kinetics-700, Moments-in-Time, Something-something-v2 and ActivityNet.
The bold numbers and underlined are ranked first and second, respectively.
See text and caption in Table 1 for details.
Training data:
(a) ImageNet-21K;
(c) AudioSet [ 21 ] ;
(d) HowTo100M [ 41 ] ;
(e) Kinetics-700;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Ablation experiments. We investigate the effectiveness of each component of our method as well as compare to vanilla multi-dataset training method.
“Vanilla” means using cross entropy (CE) loss in training.
“w/o informative los” means using CE and projection loss.
The numbers are top-1/top-5 accuracy, respectively.
Training data:
(e) Kinetics-400;
(f) SSv2;
(g) MiT;
(h) ActivityNet.
Visualization of projection weights between different action datasets.
We output the top-5 final cross-dataset projection layer weights of the K400-312p model.
For all cross-dataset pairs, please refer to supplementary materials.