Multi-dataset Training of Transformers for Robust Action Recognition
Authors: Junwei Liang
Enwei Zhang
Jun Zhang
Chunhua Shen
Venue: NeurIPS 2022 Accept (2022)
Keywords: Action Recognition
Vision Transformers
Multi-task learning
Multi-dataset learning
Robust Representation

Abstract: We study the task of robust feature representations, aiming to generalize well on multiple datasets for action recognition. We build our method on Transformers for its efficacy. Although we have witnessed great progress for video action recognition in the past decade, it remains challenging yet valuable how to train a single model that can perform well across multiple datasets. Here, we propose a novel multi-dataset training paradigm, MultiTrain, with the design of two new loss terms, namely informative loss and projection loss, aiming to
learn robust representations for action recognition. In particular, the informative loss maximizes the expressiveness of the feature embedding while the projection loss for each dataset mines the intrinsic relations between classes across datasets. We verify the effectiveness of our method on five challenging datasets, Kinetics-
400, Kinetics-700, Moments-in-Time, Activitynet and Something-something-v2 datasets. Extensive experimental results show that our method can consistently improve state-of-the-art performance. Code and models are released.

Multi-dataset Training of Transformers for Robust Action Recognition
Junwei Liang', ; Enwei Zhang", Jun Jun Zhang", Chunhua Shen 1 AI Thrust, Hong Kong University of Science and Technology (Guangzhou)
Tencent Youtu Lab
*hejiang University
Email: junweiliang @ hwwst-gz...d..co. miyozhang @ @tencent.com, bobbyjzhang @tenceet.com. com, chunhua @ me.com
Abstract
We study the task of robust feature representations, aiming to generalize well on multiple datasets for action recognition. We build our method on Transformers for its efficacy. Although we have witnessed great progress for video action recognition in the past decade, remains challenging yet valuable how train single model that can perform well across multiple datasets. Here, we propose a novel multi- dataset training paradigm, Multi Train, with the design of two new loss terms, namely informative loss and projection loss, aiming to learn robust representations for action recognition. particular, the informative loss maximizes the expressive- ness of the feature embedding while the projection loss for each dataset mines the intrinsic relations between classes across datasets. We verify the effectiveness of our method on five challenging datasets, Kinetics-40n Kinetics-700 Moments-in Time, Activitynet and show that our method can consistently improve state-of-the-art performance.. Code and models are available at wthhtps: //:gtt/tbbbt com/ homm/JJnneeLLLaangahtttaTaaaaaa
datasets. Extensive experimental results
Introduction
Human vision can recognize video actions efficiently despite the variations of scenes and domains. Convolutional neural networks (CNNs) [88
49
6 44 [9]]6]] effectively exploit the power of modern
computational devices and employ spatial temporal filters to recognize actions, which considerably outperform traditional models such oriented filtering in space-time (HOG3D) [30 However, due the high variations in space-time, the state-of-the-art accuracy of action recognition still far from being satisfactory, compared with the success of 2D CNNs in image recognition 244
Recently, vision transformers such as ViT (15), and MVIT [17 that are based on the self-attention [52 mechanism were proposed to tackle the problems of image and video recognition, and achieved impressive performance. Instead of modeling pixels as CNNs, transformers apply attentions on top of visual tokens. The inductive bias of translation invariance in CNNs makes it require less training data than attention-basee based transformers in general. contrast, transformer has the advantage that it can better leverage 'big data', leading to improved accuracy than CNNs. We have witnessed rapid growth video datasets (28] in recent years, which would make up for the shortcomings of data-hungry transformers. The video data has not only grown in quantity from hundreds to millions of videos [42), but also evolved from simple actions such handshaking to complicated daily activities
from the Kinetics-700 dataset [7]. Meanwhile, transformers combined with low-level convolutional operations have been proposed to further improve the efficiency and accuracy.
Coorrepponiing author. This work was partially done when JL was with Tencent.
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Video frames
Mutt-scale Vision Transformers
Embedding
P
Prediction
Dataset
Dataeet Head
Kinatica- Clean and jark
Detisel ]
Expander
Datazel Head
Projection
Prediction
Projection
Momonts In- time
Weighttting
Informative loes
Dataset Head
Croas- Data-et Projection
Datasot K Prediction
Figure 1: Overview of multi-dataset training framework. We propose utilize the intrinsic relations between classes across different action datasets. As we can see, the two video examples from Kinetics and Moments-in- Time datasets, respectively, show that samples from these two classes can be used to train both classification heads. The videos from multiple action datasets are input to the MViTv2 (Section[3.] (3T] backbone and the model is trained jointly. The informative loss is applied maximize the information content of the embedding from the backbone and the projection loss applied learn the intrinsic relations (Section 3.2)
Due to the data-hungry nature of transformers, most transformer- bassed models for action recognition requires large-scale pre-training with image datasets such as ImageNet-21KK [14] and JFT-3B [58] to achieve good performance. This pre-training and fine-tuning training paradigm is time-consuming and it is not parameter-efficient meaning that for each action dataset, a new model need to be trained end-to-end. Different from large image datasets such as ImageNet-21K that covers wide range of object classes, currently the most diverse action dataset, Kinetics-700, only contains 700 classes. Each action dataset may be also limited a certain topic or camera views. For example, Moments-in- Time [42] only contains short actions that happen in three seconds and Soarthinn-SSmething--- focuses on close-up camera view of person-object interactions. These dataset biases might hinder models trained on a single dataset to generalize and be used in practical applications. These challenges in action datasets make learning general-purpose purpose action model very difficult. An ideal model should be able to cover wide range of action classes meanwhile keeping the computation cost low. However, simply combining all these datasets to train a joint model does not lead to good performance [38] In previous work [59]. the authors have shown the benefit of training a joint model using multiple action datasets but their method requires large-scale image datasets such as ImageNet-211 [14 and
(2
JFT-3B (58) which is not available to the research community.
In this paper, propose a general training paradigm for Multi-dataset Training of robust action recognition models, MultiTrain. Our method is designed to learn robust and informative feature representations principled approach, using the informative loss for regularization. We do not assume the availability of large-scale image datasets pre-training (although one can certainly take advantage of that). Since there are intrinsic relations between different classes across different action datasets (See Fig. [ for examples of similar classes from two datasets), we propose projection loss to mine such relations such that the whole network trained to avoid over-fitting to certain dataset biases. Finally, all proposed loss terms are weighted using learned parameters. Thus, no syper-parameter tuning is needed. Our empirical findings as shown in Table I indicate that our robust training method can consistently improve model backbone performance across multiple datasets. We show that our model can achieve competitive results compared to state-of-the-art methods, even without large-scale image dataset pre-training, and with lower computational cost.
The main contributions of this paper are thus three-fold:
To our knowledge, this the first work to introduce informative representation regularization into multi-dataset training for improving action recognition.
We propose an effective approach to mine intrinsic class relations in multi-dataset training by introducing the projection loss.
Our method requires negligible computation overhead during training and no additional computation during inference to the backbone network. Extensive experiments on various datasets suggest our method can consistently improve performance.

---

2
Related Work
We review some work that is closest to ours.
CNNs and Vision Transformers. CNNs work the standard backhones throughout computer vision tasks for image and video. Various effective convolutional neural architectures have been raised to improve the precision and efficiency (e-g. VGG [4], ResNet (24] and DenseNet (26)). Although CNNs are still the primary models for computer vision, the Vision Transformers have already shown their enormous potential. Vision Transformer (ViT (T5) directly applies the architecture of
Transformer on image classification and gets encouraging performance. ViT and its variants 22 37
17 40
achieve outstanding results in both image and video processing in recent years.
Action Recognition/ Classificattion. The research of action recognition has advanced with both new datasets and new models. One of the largest modern benchmarks for action recognition is the Kinetics dataset (28). The Kinetics dataset proposes a large benchmark with more categories and
more videos (e.g., 400 categories 160,000 clips in (28] and 700 categories in 77 more challenging benchmarks compared to previous datasets like UCF-101 [47]. The Moments-in- Time [42 (MiT)
dataset provides million short video clips that cover 305 action categories. Note that is infeasible for Kinetics and MiT datasets to cover all the possible actions in all possible scales. For example.
surveillance actions 10
g
are missing in the two datasets. Many new approaches 500 60
39
20
55 36 TO, 8 27
9 35
have been carried out on these datasets, of which the SlowFast network (20
and MVIT obtain promising performance. We can see that the trend of action recognition in the last two decades is to collect larger datasets (e.g., Kinetics) and build models with a larger capacity.
Multi-dataset Co- Co-Training. Previously, multi-dataset co-training has been explored in the image domain such as detection (62 53 and segmentation [32). Several works [4]
46 25 were
proposed to combine multiple video datasets for training. Larger datasets often deliver better results. Combining multiple datasets to boost data size, and improve the final performance [22]. and the simultaneous use of multiple datasets is also likely to alleviate the damaging impact of dataset bias. OmniSource [16] utilizes web images as part of the training dataset to expand the diversity of the training data to reduce dataset bias. VATT M uses additional multi-modal data for self-supversied pretraining and finetunes on downstream datasets. CoVeR [59 combines image and video training even during the finetuning stage and reports significant performance boost compared to single-dataset training. Poly ViT (38) further extends to training with image, video and audio datasets using different sampling procedures.
this paper, we propose simple yet effective way (no multi-stage training, no complex dataset-wise sampling and Hyper-paramete tuning) for multi-action-datan training, without the use of any image or additional data from other modality.
Video Domain Generalization (VDG). Our work is also related but different from video domain generalization 661. The key distinction is that our goal is to train single model on multiple related tasks (multiple action datasets) such that the model performs well on the same set of tasks, whereas
VDG aims to generalize a model unseen -iitrii-iititttionn target domain [56] 13 51) These models still suffer from problem of parameter- ineffeeieenffffiinnn...... meaning that separate models are needed for different target datasets.
3 Method
Our method is built upon the backbone of the improved Multi-scale Vision Transformers (MViTv2) [34] [7). Note that our approach works with any action recognition backbones. Given videos from multiple datasets during training, the model backbone takes the video frames and pro- duces feature embeddings for each video. The same number of Multi-layer Perceptron (MLP) as the datasets are constructed as model heads to predict action classes for each dataset. To facilitate robust cross-dataset training, we propose two loss terms, namely, the informative loss and projection loss. The informative loss aims to maximize the embeddings' representative power. The projection loss, with the help of multiple cross-datass projection layers, guides the model to learn intrinsic relations between classes of different dataset, hence the model heads can be trained jointly. See Fig. for an overview of our framework. In this section, we first briefly describe the MViTv2 backbone design, and then present proposed robust cross-dataset training paradigm.
3

---

3.1 The MViTv2 Backbone
Our model is based on the improved multi-scale vision transformers (MViTv2) 17 34 which learns a hierarchy from dense (in space) and simple (in channels) to coarse and complex features. The
series of work of vision transformers [15] (ViTs) follows the basic self-attention architecture [52]
originally proposed for machine translation. The key component of the MViTv] model (17] is the Multi Head Pooling Attention (MHPA), which pools the sequence of latent tensors reduce the spatial or temporal dimension of the feature representations. In MVīTv2 [34], a residual connection MHPA for the pooled query tensor and a decomposed relative position embedding 3 are added. In this paper, we use 3D convolution as the pooling operation. Please refer to supplemental material for visualization of the MVVTv2 block. Each MViTv2 block consists of multi-head pooling attention layer (MHPA) and multi-layer perceptron (MLP), and the residual connections are built in each layer. The feature of each MViTv2 block is computed by:
Xi (H)) + (Xol(X)
Block(X) = MLP(LN (X1)) X1,
where X is the input tensor each block. Multiple MViTv2 blocks grouped into stages to reduce the spatial dimension while increase the channel dimension. The full backbone architecture listed in supplementary material.
Classification head. For the action recognition problem, the model produces C-class classification logits by first averaging the feature tensor from the last stage along the spatial-tempont dimensions (we do not use the [CLASS] token in our transformer implementation). denoted as z E R A linear classification layer is then applied on the averaged feature tensor to produce the final output, y = Wont 2 E RC.
Multi-dataset training paradigm. In general, to facilitate multi-dataset training of K datasets, the same number of classification heads are appended to the feature embeddings. The k-th dataset classification output is defined Y (Z; Wa) E BBxC, where h4 can be a linear layer MLP and Wa & is the layer parameter.
3.2 MultiTrain: Robust Multi-dataset Training
Our training process fully leverages different action recognition datasets by enforcing an informative loss maximize the expressiveness of the feature embedding and projection loss for each dataset that mines the intrinsic relations between classes across other datasets. We then use uncertainty to weight different loss terms without the need for any super-parameters.
Informative loss. Inspired by the recently proposed VICReg ( and Barlow Twins 57 methods for self-supervised learning in image recognition, we propose utilize an informative loss function with two terms, variance and covariance, to maximize the expressiveness of each variable of the embedding. This loss is applied to each mini-batch, without the need for batch-wise nor feature- wise normalization. Given the feature embeddings of the mini-batch, Z E RR Bxd an expander (implemented as two-layer MLP) maps the representations into an embedding space for the informative loss to be computed, denoted as Z' E RR Rxd The variance loss is computed using hinge function and the standard deviation of each dimension of the embeddings by:
Ez''s - Z' +0). d 1-11
L Σ max 0, 10,1-1
where is a tensor slicing operation that extracts all elements from a dimension, and Z is the mean over the mini-batch for j-th dimension. is small scalar preventing numerical instabilities. With random sampling videos across multiple datasets for each batch, this criterion encourages the variance of each dimension in the embedding be close to preventing embedding collapse [57
We did not implement this part the code not available at the time of writing (March 2022).

---

The covariance loss c(Z') is defined as:
1
71
1 12 Σ 2' 71 1=1
C(Z') n
Z' - -Z) -Z)", where
1=1
(3)
_ - Σ iF1
LIC(Z')
Inspired by VICReg 3 and Barlow Twins [57]. we first compute the covariance matrix of the feature embeddings in the batch, C(Z'), and then define the covariance term L the sum of the squared off-diagonal coefficients of C(Z'), scaled by factor of 1/d.
Projection Loss. In previous works 55 38]. the intrinsic relations between classes from across different datasets have been mostly igmored during training. We believe that samples in one dataset can be utilized to train the classification head of other datasets. As shown in Fig. 1 the "Clean and jerk" video sample from Kinetics can be considered as a positive sample for "Weightlifting" in Moments-in- Time as well (but not vice versa). Based on this intuition, propose to add directed projection layer for each pair of datasets for the model to learn such intrinsic relations. One can also initialize the projection using prior knowledge but it is out-of-scope for this paper. Given the output from the k-th dataset classification output, the projected classification output is defined as:
K-1
Y' = Y, + Σ Werojy, Y, E RC
(4)
where Cg is the number of classes for the k-th dataset and it is the learned directed class projection weights from i-th to k-tth dataset. In this paper we only consider linear projection function. We then use the ground truth labels of the k-th dataset to compute standarad cross-entropy loss:
LE = Σ
Yr.c log( (okk,)
(5)
where YR.C f.c is the ground truth label for the e-th class from the k-th dataset.
Training. We jointly optimize the informative loss and the projection loss during multi-dataset training. To avoid tuning loss weights of different terms, we borrow the weighting scheme from multi-task learning (29 and define the overall objective function as:
K Σ k=1
L(o) = ["
20261 L*
+ log dk:
(
20%
where a is a vector of learnable parameters of size K (the number of datasets) for each projection loss term. This avoids the need manually tune loss weights for different datasets.
4
Experiments
In this section, to demonstrate the efficacy of our training framework, carry out experiments on five action recognition datasets, including Kinetics-400 (28),
(23), Moments- in-Time 42 Activitynet [5) and Kinetics-700 77. The action recognition task is defined to be classification task given a trimmed video clip. Unlike previous works [8 59], we not initialize our model using ImageNet [14] since it consumes more computation. Please refer to the supplementary material for detailed comparison between train from scratch recipe and from ImageNet. In the experiments, aim to showcase that method can achieve significant performance improvement with minimal computation overhead compared to baselins.
4.1 Experimental Setup
Datasets. We evaluate our method on five datasets. Kinetics-400 28 (K400) consists of about 240K training videos and 20K validation videos in 400 human action classes. The videos are about 10 seconds long. Kinetics-700 7 (K700) extends the action classes to 700 with 545K training and

---

Table I: Comparison with state-of-the-art Kinetics-400, Moments-in-Tine Something Comeelting-something--- something--2 and ActivityNet. We divide the baselines into two groups based on whether they are parameler-effiientt report 1op-1/top-S accuracy for each dataset. The bold numbers and underlined are ranked first and second, respectively, The FLOPs computation is for a single video clip input. Training data: (a) ImageNet-22 21 (b) ImageNet- IK; (c) AudioSet [21]: (d) HowTo100M [41): (e) Kinetics 400; (1) SSv (g) MIT, (h) ActivityNet.
Method Pre-/Training Data
gFLOPs
K400
MiT
SSv2
ActNet
ViVIT 2
+(a)
3992
81.3 / 94.7
38.5
64.1
65.9 89.9
VidTr 33
+(a)
392
80.5 94.6
63.0/-
Tireerrrmer
+(a)
2380
80.7 / 94.7
62.4/-
X3D-XXL [8
+(a)
194
80.4 / 94.6
MoViNet
Scratch
386
81.5 / 95.3
40.2
64.1 / 88.8
MVĪT-B 17
Scratch
455
81.2 / 95.1
67.7 99.99
MTV-B (320p) 54
+(a)
1116
82.4 / 95.2
41.7 69.7
68.5 99.4
Video Swin
+(a)
2107
84.9 / 96.7
69.6
MV 34
+(a)
2828
86.1 / 97.0
73.3/ /3.1 94.1
MVīTv2 w/ abs. pos.
Scratch
225
80.4
Ours-baseline
Scratch
224
79.8 / 93.9
38.6 67.5
67.0
81.5 99.1
VATT (c),d), Downstream
2483
82.1 /95.5
41.1 67.7
CoVER 59
(f))))
2380
83.1 /-
41.3/-
64.2/-
(b),e).(f)
PolyVIT
[Audio] [Image]
3992
82.4 / 95.0
38.6 65.5
MultiTrain
(c).()(())))))
224
81.9 95.2
41.7 / 71.0
68.9/91.6
87.4/97.3
MultiTrain (312p)
(c),(f).(g).h))
614
83.2 / 96.4
43.1 / 71.9
69.3 /2.11
92.1
88.2/97.
35K validation videos. The mamething-Somethingg (SSv2) 23 dataset contains person-object interactions, which emphasizes temporal modeling. SSv2 includes 168K videos for training and 24K videos for evaluation on 174 action classes. The Moments-in-Time (MiT) dataset is one of the largest action dataset with 727K training and 30k validation videos. MiT videos are mostly short 3-second clips. The AetivityNet dataset (5) (ActNet) originally contains untrimmed videos with temporal annotations of 200 action classes. We cut the annotated segments of the videos into 10-second long clips and split the dataset into 107K training and 16K testing. Following previous works [20] 55), we follow the standard dataset split and report top- 1/top-5 classification accuracy on the test split for all datasets. We conduct two sets of experiments, namely, "K400, MiT, SSv2, ActNet", and "K700, MiT, SSv2, ActNet".
Implementation. Our backbone model utilizes MV as deseribed in Section 3.1 Our models
are trained from scratch with random initialization, without using any pre-training (same as in (20 and different from previous works (59. 38 that require large-scale image dataset pre-training like
ImageNet-211 K 14] or JFT-3B (88).. We follow standard dataset splits as previous works (3
See more details in the supplementary material.
Baselines. VIT (38 utilizes multi-task learning on image, video and audio datasets to improve vision transformer performance. The backbone they used are based ViT-ViVIT [2]. Similarly, VATT utilizes additional multi-modal data for self-supversied pretraining and finetunes on downstream datasets. The backbone network is based VIT [15]. COVER [59] is a recently proposed co- to-ttrinnngg method that includes training with images and videos simultaneouslyy Their model backbone is based on SFomerr [4. We also compare method with other recent models trained using large-scale image datasets. See Table and Table for the full list.
4.2
Main Results
We summarize our method's performance Table and Table 2 We train our model jointly on MiT, SSv2, ActNet and two versions of the Kinetics datasets.

---

Table 2: Comparison with state-of-the-art on Kinetics-700, Moments-in-Tinee soameellinn-ommehingg---- and Aeti The bold numbers and underlined are ranked first and second, respectively. See text and caption
Table 1 for details. Training data: (a) ImageNet-21K; (c) AudioSet [21] (d) How To100M [1]; (e) Kinetics-700;
(1) SSV2; (g) MiT; (h) ActivityNet.
Method
Training Data
gLLPPs
K700
MIT
SSv2
ActNet
VidTr [33
+(a)
392
70.8/-
63.0/-
MoViNel
Scratch
386
72.3/-
MTV-B (320p)
+(a)
1116
75.2/91.7
41.7 / 69.7
68.5/90.4
MV
+(a)
2828
79.4/94.9
73.3 /9.11
Ours baselia
Scratch
224
74.1
38.6/67.5
67.0 90.7
81.5 15..1
VATT O+
(c),d), Downstream
2483
72.7
41.1/67.7
COVER (59
(a).())(((())
2380
74.9/-
41.51-
64.7 /-
MultiTrain
(e).(f)(g))(h)
224
79.8/
42.2 72..3
69.1 /9.2
88.1 / 97.2
MultiTrain (312p)
(e).(O.(g)))h))
614
76.3/ 93.5
43.5 73.0
70.4 93.1
89.1 / 18.1
We first compare our method with the original MViTv2 backbone in Table "MViTv2 w/ abs. pos."
means VV model with absolute positional embedding, which is taken from Table A.6 of MViTv2 paper [34] and it is (almost) the same as our model implementation. We can not achieve the same accuracy with the same recipe as MVīTv2. which may be due differences in the Kinetics dataset (missing some videos, etc. See supplementary material for full dataset statistics). Poly (38] is trained jointly with multiple image, audio and video datasets. We list the larger ones. We train our baseline model on the training set of each dataset to investigate the baseline performance. As we see, after adding robust joint training proposed in this paper, performance on each dataset has increased by 2.1 %3 3.1 1,1,, 1.9% and 5.9% on K400, MiT, SSv2, ActivityNet, respectively in terms of top-1 accuracy. Note that our method achieves such improvement withtout large-scale image pre-training and additional inference computational cost.
We then compare our method with state-of-the-at on these datasets. We train higher resolution model with larger spatial inputs (3 12)) and achieves better performance compared to recent multi- dataset training methods, COVER (59 and PolyVit Vit (38], on Kinetics-400, and significantly better on MiT and SSv2, as shown in Table I Note that our model does not use any image training datasets, and our model computation cost only fraction of the baselines. We also show that our performance boost does not come from the additional training dataset of ActivityNet in Table
Our method also achieves competitive results compared state-of-the-art models trained with large- scale image dataset (ImageNet-21K (14). Compared to recent method, MTV-B [54], our method is able to achieve significantly better top-1 accuracy across Kinetics-400, MiT, SSv2 by 0.8%, 1.4%, 0.8%, respectively, at half of the computation cost and without large-scale pre-training. Note that our model is parameter-efficient while multiple MTV-B models need be trained and tested on these datasets separately. Our method can achieve better performance with a deeper base backbone or larger resolution inputs but we have not tested due to limitation of computation resources.
We then compare our method on the Kinetics-700, MiT, SSv2 and ActivityNet training with baselines. Our parameter-efffcient model can achieve better performance than MTV-B [54] at nifth of the computation cost. With a larger resolution model at 312p, we achieves significantly better performance than the baseline across Kinetics-400, MiT, SSv2 by 2.2%, 4.9%, 3.4%, respectively.
4.3 Ablation Experiments
In this section, we perform ablation studies on the K400 set. To understand how action models can benefit from our training method, we explore the following questions (results are shown in Table B):
Does our proposed robust loss help? We compare our model training with vanilla multi-dataset training, where multiple classification heads are attached to the same backbone and the model is trained simply with cross-entropy loss. The vanilla model is trained from a K400 checkpoint as ours. As shown in Table we try training the vanilla model with both the same training schedule as ours and 4x longer schedule. As see, there is significant gap between the overall performance of the
7

---

Table 3: Ablation experiments. We investigate the effectiveness of each component of our method as well compare to vanilla multi-dataset training method. "Vanilla" means using cross entropy (CE) loss in training. "woo informative los" means using CE and projection loss. The numbers are top-1/top-S accuracy, respectively. Training data: (e) Kinetics-400; (1) SSv2; (g) MIT; (h) ActivityNet.
Method Training Data
K400
MiT
SSv2
ActNet
MultiTrain
(t).(O.())(h)
/952
41.7 71.0
68.9 91.6
87.4 97.3
Vanilla (50 ep)
(e)((), (g),(h)
80.1/ 94.0
33.4 / 60.1
60.8 89.0
86.5 97.1
Vanilla (200 ep)
(e),(), (g).(h)
80.6/ 94.7
35.1 / 63.9
56.8/ 85.3
86.3 97.2
w/o Informative Loss
(e).(D.())(h)
13.5 33.4
7.3 19.9
9.7 / 28.5
24.8/ / 54.3
w/o Informative Loss
()))()(g)(h))
80.4 94.5
38.7 / 68.9
62.6 89.8
86.5 97.4
& w/o Projection Add
w/o Projection Loss
()).(D(g))h))
80.6 94.8
39.9 69.2
61.5 88.0
86.9 / 97.5
w/o ActNet
(e).(f.(g)
81.4/95.0
41.3 70.5
68.7 /91.3
Table Visualization of projection weights between different action datasets. We output the top-5 final cross-dataset projection layer weights of the K400-312p model. For all cross-dataset pairs, please refer to supplementary materials.
Cross-dataset Pair
Top-5 Action Pairs
[C (('eending metal', "bending'). ). ('riding elephant", "skating").
K400 to MiT
Cppushing wheelchair' 'swinging') ) ('tossing coin', 'tattooing') ) (' ('eleaning toilet', 'plunging') )
[Cparkour', 'Cupocira'), ) ("running on treadmill', "Walking the dog").
K400 to ActNet
('whistling'. 'Snowboarding" boarding"), ) ('water sliding'. 'Kayaking') ) (rriding mule', 'Canoeing') )]
(("Wiiin something off of something'. "cleaning windows"). ("Throwing something in the air and catching it', 'dining"..
SSv2 to K400
("Pretending or trying and failing to twist something', 'playing poker'), ("Turnin the camera upwards while filming something'. : 'playing poker'), ("Pulling two ends of something so that separates into two pieces'. 'dining)) ]
[C (Cgambling'. 'bookbinding". ). ('uutographing'. 'cating ice cream'), ("tearing'. 'ripping paper'), ) ('hitchhiking'. "throwing ball'), ('sneezing', 'sneezing') )
MiT to K400
vanilla model and ours, validating the efficacy of our proposed method. Also, longer training schedule does not lead to better performance on some datasets, including SSv2, suggesting vanilla multi- dataset training is unstable. In terms of performance on ActivityNet, we observe that both training methods achieve good results, which might be because ActivityNet classes are highly overlapped with Kinetics-400 (65 out of 200).
How important is the informative loss? We then experiment with removing the informative loss (Section 3.2) during multi-datase training. It seems that the feature embedding of the model collapse and the model is not trained at all. We further investigate why "w/o informative Loss" completely fails but "Vanilla" seems to work by rumning an experiment of "w/o informative Loss & w/o projection add", which means we remove the projected logits addition in Eq. 5 and directly consider this
compute classification loss the projected logits. Therefore additional projection branches to the vanilla architecture. The results are slightly better than "Vanilla" on K400 and much better on MiT/SSv2. It indicates that adding projected logits to the original branch without informative loss would prevent the model from converging (the total loss does not go down).
as adding
How important is the projection loss? We then experiment with removing the projection heads (Section (.2) during multi datase training. The model is trained with the original cross-entropy loss and the informative loss. As shown in Table the performance MiT and SSv2 suffers by large margin, indicating that the projection design helps boost training by better utilizing multi-dataset information.
What does the cross-dataset projection layer learn? We analyze the cross-dataset projection weights of the K400/312p model and list top 5 concepts for each pair of datasets in Table We make
8

---

two observations. First, the top projections are visually similar actions, which confirms our intuition that there are intrinsic relations in the datasets that the model can mine improve performance, For example, "bending metal" in K400 and "bending" in MIT; "parkour" in K400 and "Capoeira" in Activitynet. Interestingly, "Wiping something off of something" in SSv2 and "cleaning windows" in K400. Second, the action with the same name may not have the highest weights. In "mit to kinetics", the "sneezing" action ranks 5th the projection weights, suggesting that there might be discrepancies of the same concept in different datasets. These observations are interesting and one may compare the learned weights with textual semantic relations (like those in ConceptNet). We leave this to future work.
Does the additional ActivityNet data help? In previous methods like CoVER and PolyViT. the ActivityNet dataset has not been used. In this experiment, we investigate the important of the ActivityNet dataset by removing it from the training set. From Table 3 we can see that the performance across all datasets drop by a small margin, indicating our superior results compared to CoVER (see Table T Table ( come from the proposed robust training paradigm rather than the additional data.
4.4 Discussion
By multi-dataset training transformers on various datasets, we obtain competitive results on multi- ple action datasets, without large-scale image datasets pre-training. Our method, multiTrain, is parameter-effiiiet and does not require hyper-parameter tuning. Current limitations of our experi- ments are that we have not tried co-training with image datasets such as ImageNet-2IK [14). Hence we do not know how much performance gain that would entail. We plan explore this future work. In addition, we have not tried training larger model with FLOPs par with state-of-the-art or other backbone architectures (e.g., CNNs) due to limitation our computational resources. Hence we are not sure how our algorithm would behave with these models. We have not explicitly explored how temporal modeling could benefit from multi-action-de work. Although our model is trained on multiple datasets, potential dataset biases can still cause negative socictal impact in worldworld deployment, as the datasets we have do not fully represent all aspects of human actions.
training, which we leave for future
5
Conclusion
In this paper, we present MultiTrain, robust multi-dataset training approach that maximizes information content of representation and learns intrinsic relations between individual datasets. Our method can train parameter-efficient models that perform well across multiple datasets.
6 Acknowledgement
This work was in part supported by Foshan HKUST Projects (FSUST2I-FYTRIOIA, FSUST21- FYTRI02A). C. Shen's participation was in part supported by major grant from Zhejiang Provincial Government.
References
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang. Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. Advances in Neural Information Processing Systems, 34, 2021.
[2] Anurag Amab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 6836-6846, 2021.
[3] Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Varianc-invarrnne--vvriin regular- ization for self-supervised learning. arXiv preprint arXiv:2105.04900 2021.
[4] Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding, arXiv preprint arXiv:2102.05500 2(3):4, 2021.

---

[5] Fabian Caba Heilbron, Victor Escorcia, Bernard Ghanem, and Juan Carlos Niebles. Activitynet: large-scale video benchmark for human activity understanding. In Proc. IEEE Conf. Computer Vision and Puttern Recognition, 2015.
[6] João Carreira and Andrew Zisserman. Quo vadis, action recognition? A new model and the kinetics dataset. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2017.
[7] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics- 700 human action dataset. arXiv preprint arXiv:1907.0698) 2019.
[8] Xiaojun Chang, Wenhe Liu, Po- Yao Huang, Changlin Li, Fengda Zhu, Mingfei Han, Mingjic Li, Mengyuan Ma, Siyi Hu, Guoliang Kang, Junwei Liang, et al. Mmvg-inf-etrol@e trecvid 2019: Activities in extended video. In TRECVID, 2019.
[9] Jia Chen, Junwei Liang, Jiang Liu, Shizhe Chen, Chenqiang Gao, Qin Jin, and Alexander Hauptmann. Informedia@@ trecvid 2017. 2017.
[10] Jia Chen, Jiang Liu, Junwei Liang, Ting- Yao Hu, Wei Ke, Wayner Barrios, Dong Huang, and Alexander Hauptmann. Minding the gaps in a video action analysis pipeline. In 2019 IEEE Winter Applications of Computer Vision Workshops (WACVW), pages 41 41-46. IEEE, 2019.
[11] Min-Hung Chen, Zsolt Kira, Ghassan AlRegib, Jaekwon Yoo, Ruxin Chen, and Jian Zheng. Temporal attentive alignment for large-scale video domain adaptation. In Proc. IEEE Int. Conf. Computer Vision, pages 6321-6330, 2019.
[12] Jinwoo Choi, Gaurav Sharma, Samuel Schulter, and Jia-Bin Huang. Shuffle and attend: Video domain adaptation. In European Conference on Computer Vision, pages 678-698--899899 Springer, 2020.
[13] Victor G Turrisi da Costa, Giacomo Zara, Paolo Rota, Thiago Oliveira-Santos. Nicu Sebe, Vittorio Murino, and Elisa Ricci. Dual-head contrastive domain adaptation for video action recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1 1181-11990 2022.
[14] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2009.
[15] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations, 2020.
[16] Haodong Duan, Yue Zhao, Yuanjun Xiong, Wentao Liu, and Dahua Lin. Omni-sourced webly- supervised learning for video recognition. In European Conference Computer Vision, pages 670-688. Springer, 2020.
[17] Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. In Proc. IEEE Int. Conf. Computer Vision, pages 6824-6835, 2021.
[18] Christoph Feichtenhofer. X3d:
Expanding architectures for efficient video recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 203-213, 2020.
[19] Christoph Feichtenhofer, Axel Pinz, and Richard P. Wildes. Spatiotemporal residual networks for video action recognition. In Daniel D. Lee, Masashi Sugiyama, Ulrike von Luxburg, Isabelle Guyon, and Roman Garnett, editors, Proc. Advances in Neural Information Processing Systems, 2016.
[20] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proc. Int. Conf. Computer Vision, 2019.
10

---

[21] Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and numan-labelens dataset for audio events. In 2017 IEEE international conference on acoustics, speech and signal processing (ICASSP), pages 776-780, IEEE, 2017.
[22] Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly-supervised pre-training for video action recognition. In Proc. IEEE Conf. Computer Vision and Pultern Recognition, pages 12046-12055, 2019.
[23] Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne Westphal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The "something something" video database for learning and evaluating visual common sense. In Proc. IEEE Int. Conf. Computer Vision, pages 5842-5850, 2017.
[24] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2016.
[25] Ryota Hinami, Junwei Liang, Shin'icht Satoh, and Alexander Hauptmann. Multimodal co-traiming for selecting good examples from webly labeled video. arXiv preprint 1rXiv:1804.06099 2018.
[26] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[27] Po-Yao Huang, Junwei Liang, Vaibhav Vaibhav, Xiaojun Chang, and Alexander Hauptmann. Informedia@ @ trecvid 2018: Ad-hoe video search with discrete and continuous representations. In TRECVID, 2018.
[28] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: meee:1005099999 2017.
[29] Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semanties. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 7482-7491, 2018.
[30] Alexander Kläser, Marcin Marszalek, and Cordelia Schmid. A spatio-temporal descriptor based on 3d-gradients. Mark Everingham, Chris Needham, and Roberto Fraile, editors, Proc. British Conf. Computer Vision, 2008
[31] Dan Kondratyuk, Liangzhe Yuan, Yandong Li, Li Zhang, Mingxing Tan, Matthew Brown, and Boqing Gong. Movinets: Mobile video networks for efficient video recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16020-16030, 2021.
[32] John Lambert, Zhuang Liu, Ozan Sener, James Hays, and Vladlen Koltun. Mseg: A composite dataset for donti- semantic segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2020.
[33] Xinyu Li, Yanyi Zhang, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe, Vidtr: Video transformer without convolutions. arXiv e-prints, pages arXiv-2104, 2021.
[34] Yanghao Li, Chao- Yuan Wu, Haoqi Fan, Karttikeya Mangalam, Bo Xiong, Jitendra Malik, and Christoph Feichtenhofer. Improved multiscale vision transformers for classification and detection. arXiv preprint arXiv:2112.01524 2021.
[35] Junwei Liang, Jia Chen, Poyao Huang, Xuanchong Li, Lu Jiang, Zhenzhong Lan, Pingho Pan, Hehe Fan, Qin Jin, Jiande Sun, et al. Informedia@ trecvid 2016. In TRECVID, 2016.
[36] Junwei Liang, Liangliang Cao, Xuehan Xiong, Ting Yu, and Alexander Hauptmann. Spatial-temporal alignment network for action recognition and detection. arXiv preprint arXiv:2012.02426. 2020.
11

---

[37] Junwei Liang, He Zhu, Enwei Zhang, and Jun Zhang. Stargazer: A transformer-bas based driver ac- tion detection system for intelligent transportation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3160-3167, 2022.
[38] Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2111. 129993, 2021.
[39] Ji Lin, Chuang Gan, and Song Han. TSM: Temporal shift module for efficient video understand- ing. In Proc. Int. Conf. Computer Vision, 2019.
[40] Ze Liu, Jia Ning, Yue Cao, Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu, Video swin transformer. arXiv preprint arXiv:2106. 13230, 2021.
[41] Antoine Miech, Dimitri Zhukov, Jean-Baptiite Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto 100m: Learning text-video embedding by watching hundred million narrated video clips. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2630-2640, 2019.
[42] Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan. Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Trans. Pattern Analysis and Machine Intelligence, 42(():502-500 2019.
[43] Jonathan Munro and Dima Damen. Multi-modal domain adaptation for fine-grained action recognition. In Proc. IEEE Conf. Computer Vision and Puttern Recognition, pages 122-132, 2020.
[44] Gunnar A Sigurdsson, Santosh Divvala, Ali Farhadi, and Abhinav Gupta. Asynchronous temporal fields for action recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 585-594, 2017.
[45] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:1409, 1556, 2014.
[46] Xiaolin Song, Sicheng Zhao, Jingyu Yang. Huanjing Yue, Pengfei Xu, Runbo Hu, and Hua Chai. Spatio-temporal contrastive domain adaptation for action recognition. In Proc. IEEE Conference on Computer Vision and Pattern Recognition, pages 9787-9795, June 2021.
[47] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv: /2 12.0402, 2012.
[48] Graham W. Taylor, Rob Fergus, Yann LeCun, and Christoph Bregler. Convolutional learning of spatio-temporal features. In Proc. Eur. Conf. Computer Vision, 2010.
[49] Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spatiotemporal features with 3d convolutional networks. In Proc. Int. Conf. Computer Vision, 2015.
[50] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri. A closer look at spatiotemporal convolutions for action recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2018
[51] Eric Tzeng, Judy Hoffman, Kate Saenko, and Trevor Darrell. Adversarial discriminative domain adaptation. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages 7167-7176. 2017.
[52] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Proc. Advances in Neural Information Processing Systems, 2017.
[53] Xudong Wang. Zhaowei Cai, Dashan Gao, and Nuno Vasconcelos. Towards universal object detection by domain attention. Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pages 7289-7298, 2019.
12

---

[54] Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. arXiv preprint wrXiv:2201.042855 2022.
[55] Ceyuan Yang, Yinghao Xu, Jianping Shi, Bo Dai, and Bolei Zhou. Temporal pyramid network for action recognition. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, 2020.
[56] Zhiyu Yao, Yunbo Wang, Jianmin Wang, Philip Yu, and Mingsheng Long, Videodg: generalizing temporal relations in videos to novel domains. IEEE Trans. Pattern Analysis and Machine Intelligence, 2021.
[57] Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In International Conference on Machine Learn- ing, pages 12310-12320. PMLR, 2021.
[58] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transform- ers. arXiv preprint, Abs/2106.04560, 2021.
[59] Bowen Zhang, Jiahui Yu, Christopher Fifty, Wei Han, Andrew M Dai, Ruoming Pang, and Fei Sha. Co-training transformer with videos and images improves action recognition. arXiv preprint arXiv:2 2021.
[60] Yue Zhao, Yuanjun Xiong, and Dahua Lin. Trajectory convolution for action recognition. In Proc. Advances Neural Information Processing Systems, 2018
[61] Kaiyang Zhou, Ziwei Liu, Yu Qiao, Tao Xiang, and Chen Change Loy. Domain generalization: A survey, IEEE Trans. Pattern Analysis and Machine Intelligence, 2022.
[62] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7571-7580) 2022.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] See Section 4.4
(c) Did you discuss any potential negative societal impacts of your work? [Yes] See Section 4.4
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments...
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either the supplemental material or as a URL)? [No] We plan to
open-source our code upon paper acceptance. We have provided detailed implementa- tion instructions in the main text and in the supplemental material.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they chosen)? [Yes] See Section 4 and the supplemental material.
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We follow standard data splits and evaluation protocol as previous works. See Section
(d) Did you include the total amount of compute and the type of resources used (c.g., type of GPUs, internal cluster, cloud provider)? [Yes] See supplemental material.
4. If you using existing assets (c.g., code, data, models) or curating/releasing assets...
13

---

(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [No] We used open-source (Apache-2.0
licensed) MVTTv2 code and standard public datasets
(c) Did you include any new assets either in the supplemental material or as URL? [No]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [No]
(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [Yes] See Section 4.4
5. If you used crowdsourcing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links Institutional Review Board (IRB) approvals. if applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "jOkuFm850-z", "reviewer": "Reviewer_YQNQ", "summary": "This paper is the first work to propose an informative representation of regularization into cross-dataset action recognition. This work makes full use of existing visual transformer backbones. This method is dedicated to learning robust and informative representations. By combining the projection loss, this work can effectively mine intrinsic class relations. Experiments on different datasets prove the method can achieve better performance and produce state-of-the-art results. ", "questions": "- The table caption should be above the column.\n- The cross-dataset action recognition has yielded many forms such as Video Domain Adaptation and Video Domain Generalization. The cross-dataset in this paper focuses more on multi-domain learning. The concept of cross-dataset in this paper may lead to misleading and need to be clarified. \n- This paper should add the multi-domain baselines for a fair comparison with the existing multi-domain methods.\n- The author should clarify the novelty of this paper as described in Strengths And Weaknesses.\n\n\n======\nI have read the author's comments. Most of my concerns are clarified. I will increase my score.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n- This work seems to be the first work to introduce informative representation regularization into cross-dataset training for action recognition. It explores how to learn robust representations among multiple video domains. \n- To my knowledge, this work is the first work to bring the informative loss and projection loss into cross-dataset action recognition. This self-supervised loss can bring performance gains. \n-  This method may be suitable for any action recognition model. \n\nWeakness:\n- This work seems to be the combination of existing video backbones, projection loss, and informative loss. Cross-dataset action recognition is a challenging problem due to the temporal information inner the video sequence. This paper did not consider more on the temporal information in the sequence. The design of the projection loss and the informative loss did not consider the temporal dynamics. The projection loss and the informative loss should be carefully designed for this specific cross-dataset action recognition task not directly used. \n- This work has no contributions to both the backbone and loss. The novelty of this work should be clarified.\n- The problem this work studies is a multi-domain problem. However, this work did not compare with multi-domain methods. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "a7SyyN7UZKN", "reviewer": "Reviewer_U8Wa", "summary": "This paper proposes a new co-training paradigm **CrossRoad** for video representation learning. \nIt consists of two novel loss terms, namely informative loss and projection loss. \nThe informative loss encourages the variance of each dimension in the embedding to be large.\nThe projection loss maps predictions from other dataset heads to the current dataset class and uses ground-truth action label to compute the standard cross-entropy loss. \nExperiment results show that the two auxiliary losses are helpful in co-training. ", "questions": "1. Why removing the informative loss leads to a complete failure? Would you please provide more insights or analysis? \n2. In experiments, with vanilla co-training, the K400 performance improved just a little bit (only 0.3% and is the same as MViTv2 w/o rel entry in Table 1), while performance on other datasets drops drastically. That finding is, however, quite different from the findings in CoVER. In CoVER, with vanilla joint-training, the performance on all datasets improves. What do you think could be the reason? Is it due to different pre-training or different architectures adopted? Please provide more results (like experiments with IN-21K pretraining) to support you conclusions. ", "limitations": "Yes.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strength\n1. This work achieves strong recognition results via co-training multiple datasets with a relatively light transformer backbone MViTv2, the improvement across multiple datasets is around 2% to 4%. \n2. The efficacy of two novel components are validated via ablation study.\n\n## Weakness\n1. Large-scale image pre-training is a common practice among video transformers. However, the related results are not provided in this paper. \n2. Though the authors have the ablation study for two aux losses, there exists few analyses (See questions). \n3. Some experimental findings in this paper is quite different from findings in CoVER, but no explanation is provided. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "OFMh8_NpsF_", "reviewer": "Reviewer_iVPd", "summary": "The paper proposes a method to train video transformer on multiple video datasets. Instead of simply applying multiple cross-entropy loss over, the authors proposed to manipulate on embeddings encoded by an improved multiscale vision transformer to capture the intrinsic relations between classes across different action datasets. Specifically, they first adopt the informative loss from Barlow-Twins to maximize variance and covariance across embedding channels. Second, they propose to perform directed project from one dataset's classification head to another, to learn the label relation across datasets. The results show that the full pipeline is superior to vanilla training over multiple datasets and achieve state-of-the-art result.", "questions": "1. \"Note that our model does not use any image training datasets, and our model computation cost is only a fraction of the baselines\" (L247-248). It is true that pre-training on large image datasets. However, training video model from scratch typically means more epochs for convergence. (e.g. MViTv2 uses 200 epochs). Since training image model is often cheaper (1/10 than video model), I would see more discussion to justify this argument.\n\n2. MViTv2 without Rel-PE. If I am not mistaken, MViTv2 uses relative positional embedding by default (See their Figure 2). Therefore I do not fully understand the statement \"The “MViTv2 w/o rel” indicates the model without the relative positional embedding in the original paper\" (L234-235).\n\n\n==== \nPost-rebuttal revision:\nMy questions have been addressed. A 34-90% improvement over image-based pretraining at the cost of 1-2% drop of accuracy is fine but not impressive. The learned directed class projection is somewhat interesting. Therefore I will keep the rate of \"5 Borderline accept\".", "limitations": "Yes. The authors have discussed the limitations, e.g., co-training is limited on video datasets only. They have also covered potential negative societal impacts such as dataset biases.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "### Strengths\n\n+ The performance is good considering that CrossRoad is purely trained on top of video dataset without any image pretraining.\n\n+ The method is simple and the informative loss borrowed from BarlowTwins seems effective, especially when we want to include projection across different classification heads.\n\n### Weaknesses\n\n- Cross-dataset Training is not only a standalone problem in the video domain. There has been a few works on other tasks such as detection [1,2], segmentation [3]. The authors are suggested to consider discussing these methods in the related work for broader scope.\n\n[1] Zhou, Xingyi, et al. \"Simple multi-dataset detection.\" CVPR 2022.\n[2] Wang, Xudong, et al. \"Towards universal object detection by domain attention.\"CVPR 2019.\n[3] Lambert, John, et al. \"MSeg: A composite dataset for multi-domain semantic segmentation.\" CVPR 2020.\n\n- A few arguments need further justification, including\n  - The computation cost comparison between training video model from scratch v.s. pre-training on image dataset and then finetuning. (See Q1.)\n- Some settings in the ablation need further clarification, including:\n  - The reason of studying MViTv2 without relative position embedding (Q2.).\n  - \"Vanilla\" cross-dataset training v.s. \"- informative loss\". I don't fully get the difference. Do you mean \"Vanilla\" = CrossEntropy (CE) only, \"- informative loss\" = CE + projection loss?\n\n- What is $\\sigma_k$ in Eq (7) and L198 and how do you determine the value?\n\n- How does the learned directed class projection weights look like? Some visualization and discussion might be preferred.\n\n- How do you construct the video clips on ActivityNet? Do you uniformly cut the entire video into 10-second clips or only keep those temporal segments annotated as activities. This might be useful for later efforts trying to reproduce the results. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept