Ablation Experiments
In this section, we perform ablation studies on the K400 set.
To understand how action models can benefit from our training method,
we explore the following questions (results are shown in
Table 3 ):

Does our proposed robust loss help? We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss.
The vanilla model is trained from a K400 checkpoint as ours.
As shown in Table 3 , we try training the vanilla model with both the same training schedule as ours and a 4x longer schedule.
As we see, there is a significant gap between the overall performance of the vanilla model and ours, validating the efficacy of our proposed method.
Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi-dataset training is unstable.
In terms of performance on ActivityNet, we observe that both training methods achieve good results, which might be because ActivityNet classes are highly overlapped with Kinetics-400 (65 out of 200).

How important is the informative loss? We then experiment with removing the informative loss (Section 3.2 ) during multi-dataset training.
It seems that the feature embedding of the model collapse and the model is not trained at all.
We further investigate why “w/o informative Loss” completely fails but “Vanilla” seems to work by running an experiment of "w/o informative Loss & w/o projection add", which means we remove the projected logits addition in Eq. 5 and directly compute classification loss on the projected logits.
Therefore we can consider this run as adding additional projection branches to the vanilla architecture. The results are slightly better than "Vanilla" on K400 and much better on MiT/SSv2. It indicates that adding projected logits to the original branch without informative loss would prevent the model from converging (the total loss does not go down).

How important is the projection loss? We then experiment with removing the projection heads (Section 3.2 ) during multi-dataset training.
The model is trained with the original cross-entropy loss and the informative loss.
As shown in Table 3 , the performance on MiT and SSv2 suffers by a large margin, indicating that the projection design helps boost training by better utilizing multi-dataset information.

What does the cross-dataset projection layer learn? We analyze the cross-dataset projection weights of the K400/312p model and list top 5 concepts for each pair of datasets in Table 4 .
We make two observations. First, the top projections are visually similar actions, which confirms our intuition that there are intrinsic relations in the datasets that the model can mine to improve performance. For example, “bending metal” in K400 and “bending” in MIT, “parkour” in K400 and “Capoeira” in Activitynet. Interestingly, “Wiping something off of something” in SSv2 and “cleaning windows” in K400.
Second, the action with the same name may not have the highest weights. In “mit to kinetics”, the “sneezing” action ranks 5th in the projection weights, suggesting that there might be discrepancies of the same concept in different datasets.
These observations are interesting and one may compare the learned weights with textual semantic relations (like those in ConceptNet). We leave this to future work.

Does the additional ActivityNet data help? In previous methods like CoVER and PolyViT, the ActivityNet dataset has not been used.
In this experiment, we investigate the important of the ActivityNet dataset by removing it from the training set.
From Table 3 , we can see that the performance across all datasets drop by a small margin, indicating our superior results compared to CoVER (see Table 1 and Table 2 ) come from the proposed robust training paradigm rather than the additional data.