M ultiTrain: Robust Multi-dataset Training
Our training process fully leverages different action recognition datasets by enforcing an informative loss to maximize the expressiveness of the feature embedding and a projection loss for each dataset that mines the intrinsic relations between classes across other datasets.
We then use uncertainty to weight different loss terms without the need for any hyper-parameters.

Informative loss. Inspired by the recently proposed VICReg [ 3 ] and Barlow Twins [ 57 ] methods for self-supervised learning in image recognition, we propose to utilize an informative loss function with two terms, variance and covariance , to maximize the expressiveness of each variable of the embedding.
This loss is applied to each mini-batch, without the need for batch-wise nor feature-wise normalization.
Given the feature embeddings of the mini-batch, \mathbf{Z}\in\mathbb{R}^{B\times d} , an expander (implemented as a two-layer MLP) maps the representations into an embedding space for the informative loss to be computed, denoted as \mathbf{Z^{\prime}}\in\mathbb{R}^{B\times d} .
The variance loss is computed using a hinge function and the standard deviation of each dimension of the embeddings by:

\mathcal{L}^{v}=\frac{1}{d}\sum^{d}_{j=1}max\Bigl{(}0,1-\sqrt{\frac{\sum(\mathbf{Z^{\prime}}_{ij}-\bar{\mathbf{Z^{\prime}}}_{:j})}{d-1}+\epsilon}\Bigr{)}, (2)

where : is a tensor slicing operation that extracts all elements from a dimension, and \bar{\mathbf{Z^{\prime}}}_{:j} is the mean over the mini-batch for j -th dimension. \epsilon is a small scalar preventing numerical instabilities.
With random sampling videos across multiple datasets for each batch, this criterion encourages the variance of each dimension in the embedding to be close to 1 , preventing embedding collapse [ 57 ] .
The covariance loss c(\mathbf{Z^{\prime}}) is defined as:

\begin{split}C(\mathbf{Z^{\prime}})&=\frac{1}{n-1}\sum^{n}_{i=1}(\mathbf{Z^{\prime}}_{i}-\bar{\mathbf{Z^{\prime}}})(\mathbf{Z^{\prime}}_{i}-\bar{\mathbf{Z^{\prime}}})^{T}\;,\;where\;\bar{\mathbf{Z^{\prime}}}=\frac{1}{n}\sum^{n}_{i=1}\bar{\mathbf{Z^{\prime}}}_{i}\ \mathcal{L}^{c}&=\frac{1}{d}\sum_{i eq j}[C(\mathbf{Z^{\prime}})]^{2}_{i,j}\end{split} (3)

Inspired by VICReg [ 3 ] and Barlow Twins [ 57 ] , we first compute the covariance matrix of the feature embeddings in the batch, C(\mathbf{Z^{\prime}}) , and then define the covariance term \mathcal{L}^{c} as the sum of the squared off-diagonal coefficients of C(\mathbf{Z^{\prime}}) , scaled by a factor of 1/d .

Projection Loss. In previous works [ 59 , 38 ] , the intrinsic relations between classes from across different datasets have been mostly ignored during training.
We believe that samples in one dataset can be utilized to train the classification head of other datasets. As shown in Fig. 1 , the “Clean and jerk” video sample from Kinetics can be considered as a positive sample for “Weightlifting” in Moments-in-Time as well (but not vice versa).
Based on this intuition, we propose to add a directed projection layer for each pair of datasets for the model to learn such intrinsic relations.
One can also initialize the projection using prior knowledge but it is out-of-scope for this paper.
Given the output from the k -th dataset classification output, the projected classification output is defined as:

\mathbf{Y^{\prime}}_{k}=\mathbf{Y}_{k}+\sum^{K-1}_{i eq k}\mathbf{W}^{proj}_{ik}\mathbf{Y}_{i}\in\mathbb{R}^{C_{k}}, (4)

where C_{k} is the number of classes for the k -th dataset and \mathbf{W}^{proj}_{ik} is the learned directed class projection weights from i -th to k -th dataset.
In this paper we only consider a linear projection function.
We then use the ground truth labels of the k -th dataset to compute standarad cross-entropy loss:

\mathcal{L}_{k}=-\sum^{C_{k}}_{c=1}\hat{\mathbf{Y}}_{k,c}\log(\mathbf{Y^{\prime}}_{k,c}), (5)

where \hat{\mathbf{Y}}_{k,c} is the ground truth label for the c -th class from the k -th dataset.

Training. We jointly optimize the informative loss and the projection loss during multi-dataset training.
To avoid tuning loss weights of different terms, we borrow the weighting scheme from multi-task learning [ 29 ] and define the overall objective function as:

\mathcal{L}(\sigma)=\mathcal{L}^{v}+\mathcal{L}^{c}+\sum^{K}_{k=1}\frac{1}{2\sigma_{k}^{2}}\mathcal{L}_{k}+\log\sigma_{k}, (6)

where \sigma is a vector of learnable parameters of size K (the number of datasets) for each projection loss term. This avoids the need to manually tune loss weights for different datasets.