Beyond Traditional Transfer Learning: Co-finetuning for Action Localisation
Authors: Anurag Arnab
Xuehan Xiong
Alexey A. Gritsenko
Rob Romijnders
Josip Djolonga
Mostafa Dehghani
Chen Sun
Mario Lucic
Cordelia Schmid
Venue: Submitted to ICLR 2023 (2023)
Keywords: transformer
video
action recognition
action detection
multi-task learning
co-training
transfer learning

Abstract: Transfer learning is the predominant paradigm for training deep networks on small target datasets. Models are typically pretrained on large “upstream” datasets for classification, as such labels are easy to collect, and then finetuned on downstream” tasks such as action localisation, which are smaller due to their finer-grained annotations.

In this paper, we question this approach, and propose co-finetuning -- simultaneously training a single model on multiple “upstream” and “downstream” tasks. We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data, and also show how we can easily extend our approach to multiple “upstream” datasets to further improve performance. In particular, co-finetuning significantly improves the performance on rare classes in our downstream task, as it has a regularising effect, and enables the network to learn feature representations that transfer between different datasets. Finally, we observe how co-finetuning with public, video classification datasets, we are able to achieve state-of-the-art results for spatio-temporal action localisation on the challenging AVA and AVA-Kinetics datasets, outperforming recent works which develop intricate models.

Under review as a conference paper at ICLR 2023
BEYOND TRADITIONAL TRANSFER LEARNING: CO-FINETUNING FOR ACTION LOCALISATION
Anonymous authors Paper under double-bling review
ABSTRACT
Transfer learning is the predominant paradigm for training deep networks on small target datasets. Models typically pretrained on large "upstream" datasets for classification, as such labels are easy to collect, and then finetuned on "down- stream" tasks such action localisation, which are smaller due to their finer- grained annotations.
In this paper, we question this approach, and propose co-finetuning simultane- ously training a single model on multiple "upstream" and "downstream" tasks. We demonstrate that co-finetuning outperforms traditional transfer learning when using the same total amount of data, and also show how we can casily extend approach to multiple "upstream" datasets to further improve performance. In particular, co-fine no-finetuning significantly improves the performance on rare classes downstream task, it has a regularising effect, and enables the network learn feature representations that transfer between different datasets. Finally, we observe how co-finetuning with public, video classification datasets, we able to achieve significant improvements for spatio-temporal action localisation on the challenging AVA and AVA-Kinetics datasets, outperforming recent works which develop intricate models.
INTRODUCTION
The computer vision community has made impressive progress in video classification with deep learning, first with Convolutional Neural Networks (CNNs) (Krizhevsky et al. 2012] Carreira & Zisserman 2017 Feichtenhofer et al. 2019). and more recently with transformers (Vaswani al.j
2017 Arnab et al. [021a]] Fan et al. 2021). However, progress in other more challenging video understanding tasks, such as spatio-temporal action localisation (Pan et al. 2021 Tang et al. 2020
Zhao et al. 2022), has lagged behind significantly in comparison.
One major reason for this situation is the lack of data with fine-grained annotations which are not available for such complex tasks. To cope with this challenge. the de facto approach adopted the state-of-the-arg transfer learning (popularised by Girshick et al.) (2114). In the conventional set- large "upstream" dataset, which is typically labelled with classi-
ting, model is first pre-trained fication annotations as they are less expensive to collect. The model is then "finetuned" on smaller dataset, often for different task, where fewer labelled examples are available (Mensink al. (022). The intuition is that a model pre-trained on an auxiliary. "upstream" dataset learns generalisable fea- tures, and therefore its parameters do not need be significantly updated during finetuning. For video understanding, the most common "upstream" dataset is Kinetics (Kay et al. 2017). demon- strated by the fact that the majority of recent work addressing the task of sspatiottepoot action localisation pretrain on it (Zhao et al. 2022 Pan et al. 2021 Wu et al. 2019). Similarly for image- level tasks, ImageNet (Deng et al. 2009) is the most common "upstream" dataset.
Our objective in this paper is to train more accurate models for spatio-temporal action detection, and we do so by proposing an alternate training strategy of co-finetuning. Instead of using the additional classification data in a separate pre-training phase, we simultaneously train for both classification and detection tasks. Intuitively, the additional co-finetuning datasets can act as a regulariser during training. benefiting in particular the rare classes in the target dataset which the network could other- wise overfit on. Moreover, discriminative features which are learned for classification datasets may also transfer the detection dataset, even though the target task and labels are different.

---

Under review as a conference paper at ICLR 2023
MAP Improv improvement (%
Actions: Human-Object
Tail
Mid
Head
Play inde.
Work comp.
Dress
Read
Cut
Eat
Ride
Sail boot
Throw
Drink
Smoke
Warch obj.
Carry hole
Ansoee phone
Enter
Text
Push abj.
Shrat
Pull
Touch
Take photo
Turn
Hitt
Lnsten
Point to
Put down
Close
Open
Ciimé
Write
Drive
Lift. pick
mAP Improvementt (%)
MAP Improvement (%)
Actions: Human Human
Actions: Pase
-30
10
Mid
20
Mid
5
Head
10
Head
Fight hit
Kiss
Hand wave
Hand clap
Lift
Take
Give serve
Grab
Hug
Watch par.
Talk to
Hand slake
Push per.
Swim
Fall down
Dance
Run Jog
Lie Seep
Jump leap
Walk
Crouch knnel
Stand
Get up
Bend boo
Martial art
Sing
Listen
Figure I: Improvements achieved by our co- finntuninnnngg strategy on the AVA dataset Gu et al. AVA is a long-tailed dataset, and we attain significant improvements, particularly on the rare classes (shown by the "Tail" and "Mid" subsets) in the dataset. Our improvements are also consistent across the three action types defined in the AVA dataset. We split the 60 class labels from AVA into "Head" classes (> 10, 000 ground truth instances), "Tail" classes (< 1000 instances), and the remaining ones into "Mid" classes (detailed in Sec. 4.2
2018
Our thorough experimental amalyses confirm these intuitions. As shown by Fig. [ our co co-finetuning strategy improves spatio-temporal action localisation results for the vast majority of the action classes the AVA dataset, improving substantialy the rarer classes with few labelled examples. In particular, co-finetuning performs better than traditional transfer learning when using the same total amount of data. Moreover, with co-finetuning we can easily make use of additional "upstream" classification datasets during co-finetuning to improve results even further.
Our approach is thus in stark contrast previous works on spatio-temporal action detection which develop complex architectures to model long-range relationships using graph neural net- works (Amnab et al., 20211] Wang & Gupta, 2018, Sun et al.] [0188] Baradel al., 2018 Zhang
et al. 2019), external memories (Wu et al. 22099 Pan et al. [0[1]021]]] [ et al. (0200) and additional Tang Tang al., 2020) /Zhan et al. 2019). Instead, we use a simple detection architecture and modify only the training strategy
object detection proposals (Wang & Gupta 2018 Wu & Krahenbuhl] [202]]
to achieve higher accuracies, outperforming prior work.
We conduct thorough ablation analyses to validate our method, and make other findings too, such as the fact that although Kinetics (Kay et al.) 2017) the most common upstream pretraining dataset for video tasks, and used by all previous work addressing spatio-tempora action detection AVA, Moments in Time (Monfort et al., 2219)) is actually better. Finally, a by-produe of our strategy of co-finetuning with classification tasks is that network simultaneously perform these tasks, and is competitive with the state-of- oftee-of-the-ar on these datasets too.
2 RELATED WORK
We first discuss transfer- and multi-task learning, which our proposed co-finetuning approach is related to. We then review action detection models, this is our final task of interest.
Transfer- and multi-task learning The predominant paradigm for training a deep neural network to pre-train on a large "upstream" dataset with many annotations, and then finetune on the final "downstream" dataset of interest. This approach was notably employed by R-CNN (Girshick) ct al. 2014) which leveraged pre-trained image classification models on ImageNet (Deng et al.) 2009 for object detection the markedly smaller Pascal VOC dataset (Everingham et al. 2015 Since then, this strategy has become ubiquitous in training deep neural networks (which require large amounts of data to fit) across wide range of vision tasks (Mensink al. 2022). The training strategy of Girshick et al. (2014) is a form of "inductive transfer learning" based on the taxonomy of Pan & Yang (2009), and simply refer to it "traditional transfer learning" in this paper.
2

---

Under review as a conference paper at ICLR 2023
We further note that it is common perform multiple stages of transfer, in which case the "down-
stream" task can become the "upstream" task for the next stage of training (Mensink et al. 2022).
For example, video classification models are often initialised from image models pretrained on
ImageNet- or ImageNet-21K (Deng et al. 2009), and then finetuned on Kinetics (Wang et al.
2016 Carreira & Zisserman 2017; Arnab al.] 2021a |ertaasus et al. 202T: Liu et al. 2022
These video classification models are then again finetuned on the final task of interest, such as spatio-temporal action detection on AVA (Girdhar al. 2019 Wang et al. 2018 Zhao al. 2022)
In this work, we focus on co-finetuning video classification and action detection datasets together.
Our co-finetuning strategy, of jointly training a model on multiple upstream and downstream datasets simultaneously. is closely related to multi-task learning. Multi-task learning aims to develop models that can address multiple tasks whilst sharing parameters and computation among them (Caruana 1997]. However, in computer vision, the focus of many prior works has been to predict multiple outputs (for example semantic segmentation and surface normals) (Eigen & Fergus 2015 Kokkinos 2017 Kendall et al. 2018 Kurin et al. 2022) given a single input image. These works have typically observed that although these multi-task models are more versatile, their accuracies are lower than single-task models. Moreover, this accuracy deficit worsens as the number of tasks grows, or if multiple unrelated tasks are simultaneously performed (Kokkinos) [017 Zamir et al. [0188] Sener & Koltun| 2018). Training model to perform multiple tasks the same input often results in one task dominating the other, and numerous strategies have been proposed to mitigate this problem by adapting the losses or gradients from each task (Sener & Koltun 2018 Chen et al. 2020b)
Our scenario differs, in that although our network is capable of performing multiple tasks, it only performs a single task at a time for a given input. This is also the standard setting of multi-task learning in natural language processing (Collobert & Weston [008] |affel et al. 2019), though not in vision, with Maninis et al. (2019) referring to it as "single tasking of multiple tasks."
Our decision of performing a single task at time ensures that our training strategy is simple, and does not require additional hyperparameters to stabilise, like previous multi-task learning methods in computer vision. Moreover, we are primarily interested in improving performance on target task, and using the additional datasets to learn more generalisable features and for regularisation.
Spatio-temporal action detection models Current sfate-of-theearr action detection models (Pan
et al. 202T Fan et al., 2021 Tang et al. 2020) are based on the Fast R-CNN architecture (Girshick) 2015). and use external person detections as region proposals which are pooled and classified. We note, however, that some recent works (Chen et al. 2021 Zhao et al. 2022) build off DETR (Carion)
ct al. )020) and do not require external proposals, although they are not as performant.
Localising actions in space and time often requires capturing long-range contextual information, and state-of-the-art approaches have explicitly modelled this context to improve results. A common theme is to use graph neural networks explicitly model relationships between actors, objects and the scene (Amab et al. 2021b; Baradel et al.) 2018 [Snn et al., 2018 Wang & Guptal 2018)). Many off approaches also ohh-the-shelf object detectors to locate relevant scene context (Baradel et al., 2018 Herzig et al.] 2021 Wang & Gupttl [018 Zhang et al., 2019). To model long-range temporal contexts spanning several minutes, (Wu et al. 2019: Tang et al. (2020) have proposed external mem- ory banks which are populated by first running a video feature extractor over the input video, and are thus not trained end-to-end. The most accurate methods have included combination of external memories and spatio-tempone graphs (Pan et al. 2021 Tang et al., 2020). and are thus intricate models which require complex training and inference procedures due to the external memory bank.
Our proposed approach, in stark contrast, uses simple Fast R-CNN architecture, without any ex- plicit graph modelling (Arnab et al. 2021b Zhang et al. 2019 Sun et al. 2018), auxillary object detections (Herzig et al., 2021] Wang et al. 2020 Zhang et al. 2019) or external memories (Wul et al. 2019; Pan et al. [222 Tang et al. 2020), and achieves improvements over the baseline model that are competitive or even better than the complex models of (Arnab et al. 2021b Pan et al. 202T
Sun et al. 2018: Wu et al. 2019), by simply changing the training strategy.
3 PROPOSED APPROACH
Our objective is to train a model for spatio-temporal detection, using additional video classification data to improve our model's performance. We first describe our model in Sec. 3.1 before detailing
3

---

Under review as a conference paper at ICLR 2023
Classification Head
Transformer Backbone
2 g goAooo
Classification Head #t
Average Pool
Linear
E 3atkxwxdd
Tokenize
X R
Detection Head 41
Person Detector
Detection Head #2
ROH-Align
Linear
Figure 2: Model overview. A transformer backbone is used to extract spatio-temporal features which are then processed by task-specific heads. Note that we ro-finetune on multiple tasks simultancously.
how we co-finetune it using additional classification data in Sec. 3.2 Finally, we discuss our training
strategy in Sec. 3.3
3.1
MODEL
Our main objective is to study co-finetuning strategies for training action detection models. As such, we choose a transformer encoder model with separate heads for each classification and detection dataset that we co- fo-finetune with (Fig. 2
Given an input E TxxxWxx we first extract N tokens from the input video, z {21, 22. 2, J. with E Bixhxaxxd Here, d denotes the hidden dimension of the model, and the spatio-temporal dimensions of the tokens, (1, h, w), depend on the tubelet size (Arnab et al.) 2021a) when tokenizing the input. These tokens are then processed by a transformer encoder con- sisting of layers to result z². Depending on the task (video classification, or action detection), the encoded tokens, zz, are forwarded through dataset-specific head, shown in Fig,
As shown in Fig. 2 our detection head follows the Fast R-CNN meta- architecture (Girshick) 2015). We adopt this method as current and previous state-of-the-art ap- proaches (Pan et al. 2021 Feichtenhofer et al. [2019] Wu et al. 2019 Fan et al.) (02T) have used it. Specifically, we use person detections, p E xx obtained from an off-the-shie thel- person detector, to spatio-temporally pool the encoded tokens, z using ROI-Align (He et al., 2017). Here, N, denotes the number of people detected for the particular input video. We then obtai our predicted logits for the jth
Detection head
y, using linear projection. This is described by
z ROI-Align(z',, -P)).
y, Wz
Note that our person detections, p. are for the central keyframe of the video clip following (Pan jet al., 202T Feichtenhofer al., 2019 Wu et al. 2019 Fan et al.) (00T). For ROI-pooling, we extend the bounding box temporally to form a "tube" following prior work.
Classification head The video classification head is simple linear classifier. We extract representation of the video by spatio-temporally averaging all the encoded tokens. Therefore, z" Average(z*) Rd, and the logits predicted for each class for the ith dataset are y Wiz".
3.2 CO-FINETUNING STRATEGY
The de facto approach for training deep architectures for computer vision tasks is to pre-train the model on large "upstream" dataset, such ImageNet (Deng al., 22099, and then to finetune on the smaller target dataset. Each training procedure (pre-training or finetuning), is denoted by
arg min (x.g)E
L(f(x;0).v)
4

---

Under review as a conference paper at ICLR 2023
where D denotes the dataset consisting of pairs of videos, and ground truth labels, y. L is the loss function, and f is neural network with model parameters, If we denote {6). 8r as the parameters of the backbone and head respectively, "pre-training" can be understood as optimising all model parameters from random initialisation, whereas "finetuning" initialises the backbone model parameters, 0, from pre-training, and trains head parameters, 0,, for the final task of interest. Initialisation of network parameters / is critical, as it is a non-convex optimisation problem.
Note that it is common, particularly in video understanding, include multiple stages of pre- training. For example, many video models are initialised from networks pretrained on ImageNet. then finetuned on Kinetics and then finally finetuned for spatio-temporal detection (Girdhar et al. 2019 Köpüklü et al. [219 Wu et al. [2019] Zhao et al. (022).
In our proposed approach, we simultaneously finetune our model on multiple datasets: This includes our target dataset of interest, as well other datasets from which our model can learn expressive features, and which may help to regularise the model. We can express this as
arg min (r.))EPuUPg..UUN
E
((f(x00),))
where Na denotes the total number of datasets. Our model parameters are now 0 {0s, Ohl .ha..... ,0x, J as we have separate head parameters, On,, for each co-finetuning dataset.
3.2.1 IMPLEMENTATION
To co-finetune on multiple datasets simultaneously in Eq. 3 we construct each training minibatch by sampling examples from one particular dataset. We then perform parameter-update using SGD (or other first-order optimisers) with gradients computed from this batch, before sampling another dataset and iterating. We consider the following two alternatives for sampling minibatches:
Alternating Given Na datasets, we sample batches from datasets sequentially. For example, we first sample a batch from dataset D1, then D2, ... .N.. before continuing from D1 again.
Weighted sampling We sample batches from dataset with a probability proportional to the num- ber of examples in the dataset. This corresponds to concatenating several datasets together, and drawing batches randomly. It is also possible to set the sampling weights for each dataset, but avoid this approach to avoid introducing additional hyperparameters to tune.
3.2.2 DISCUSSION
Note that it is also possible to construct minibatch using a mixture of examples from each dataset. However, we found this approach to be less computationally efficient in initial experiments and did not pursue it further. Finally, we constructed our batches such that we first sampled only examples from dataset D,, then D: and so on, this would be equivalent to finetuning each dataset sequentially, with the difference that the optimiser state (such as first- and second-order moments of the gradient (Qian 1999 Kingma & Ba 20155) would be carried over from one dataset the next, which is not typically done when finetuning.
3.3 INTUITION BEIJIND CO-FINETUNING
In this section, we explain our intuitions behind why co-finetuning could be effective, given that by sequentially finetuning model on N₂ different datasets, the model would have access to the same total amount of data.
By co-finetuning on multiple datasets simultaneously, it is possible for the model to learn general, discriminitive visual patterns on one dataset that transfer to the other. And this effect may be more pronounced when the target dataset (such AVA (Gu al. 2018) for spatio-temporal action detec- tion) small and contains many tail classes with few labelled examples. Sequentially finetuning a model on multiple datasets may not have the same property due to "catastrophic forgetting" (French) 1999] the propensity of neural networks to degrade at previous tasks when finetuned on new one.
Another reason is that deep neural networks, and especially transformers, prone to overfitting, as evidenced by the numerous regularisation techniques that are typically employed when training them Touvron al. 2020 Arnab et al. 2021a Bello al. 2021 Wightman al. 2021). Training
5

---

Under review as a conference paper at ICLR 2023
with additional data, even if the additional co-finetuning datasets not for the task of interest, can therefore remediate this problem by acting as a regulariser. In particular, we would expect the ben- efits to be the most apparent on small and/or imbalanced datasets (since the network may otherwise easily memorise the classes with few examples). Datasets for more complex video understanding tasks, such as action detection, typically small due their annotation cost. We now validate these hypotheses experimentally in the next section.
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets AVA (Gu et al. 2018 is the largest dataset for spatio-temporal action detection, sisting of 430, 15-minute video clips obtained from movies. The dataset consists of 80 atomic actions, of which 60 are used for evaluation. These actions are annotated for all actors in the video, where one person is typically simultancously performing multiple actions. The dataset annotates keyframes at every second in the video, and following standard practice, we report the Frame AP at an IoU threshold of 0.5 using the annotations from version 2.2 (Gu et al. 2018).
Kineticy (Kay et al. 2017) is a collection of clip-classification datasets consisting of 10 second video clips focusing on human actions. The Kinetics datasets have grown progressively, with Kinetics 400, 600 and 700, annotated with 400, 600 and 700 action classes respectively. The dataset sizes range from 240,000 (Kinetics 400) to 530,000 (Kinetics 700). To avoid confusion with AVA-Kinetics, described below, we denote these datasets as Kinetics
AVA-Kinetics (Li et al. 2020) adds detection annotations, following the AVA protocol, to subset of Kinetics 700 videos. Note that only a single keyframe, in a 10-second Kineties clip, is annotated. To avoid confusion with Kinetics we use Kinetics to denote the Kinetics videos with detec- tion annotations. Therefore, the training and validation splits of AVA-Kinetics are the union of the respective AVA and Kinetics dataset splits.
Moments in Time (Monfort et al. 2019) consists of 800,000 clips, collected from YouTube, each 3-seconds long and annotated with one of the 339 classes.
Comerhing-Somethiing v2 Goyal et al. 2017 consists of about 220,000 short video clips annotated with 174 classes showing humans interacting with everyday objects.
Implementation details Our transformer backbone architecture is the ViViT Factorised En- coder (Arnab et al.] 2021a). with 16 16 x tubelets, following the authors' public implementation. We chose this backbone as it is a state-of oftth-of- video classification model. Note that ViViT, like
other popular transformers for video (Bertasius et al. 2021: Liu al. 2022; Zhang et al. 202T is
initialised from a ViT image model pretrained on ImageNer-211
2 21K (Degg et al. 2009).
We use the same person detections as prior work (Wu et al. 2019 Feichtenhofer al. 2019 Fan lt al. 2021: Arnab et al. (021b). For AVA, these were obtained using a Faster R-CNN (Ren et al.] 2015) detector, and released publicly by Wu et al. (2019), achieving an AP of 93.9 for person detection on the AVA validation set. For AVA-Kinetics, these were obtained using CenterNet (Zhou al. 2019 by Li et al.] (2020), obtaining an AP of 77.5 on Furthermore, note that we report results from processing a single view for all tasks.
for person detection.
All models are trained on 32 frames with a temporal stride of 2. We train for 15 epochs using synchronous SGD with momentum of 0.9, where an epoch is defined as the sum of the size of each dataset. Exhaustive details are included in Appendix C and we will release code upon acceptance.
4.2 ABLATION STUDY
We first conduct thorough ablation studies of our so-finetuning approach in Tables I through 4 For rapid experimentation, unless otherwise stated, we use a ViViT-Base Factorised Encoder as our backbone model. Videos are resized such that the longest and shortest sides have resolutions of 224 and 140 respectively (denoted as 140p). We use the "Weighted" minibatch sampling strategy (Sec. 3.2 unless otherwise stated.
Comparison of co-finetuning and traditional finetuning with the same data Tables I and 2 compare the effect of traditional transfer learning, and co-finetuning, with the same total amount

---

Under review as a conference paper at ICLR 2023
Table I: Comparison of traditional finetuning, and our proposed co- ro-finetuning, with the same total amount of data. Observe how co- co-finetuning consistently outperforms traditional finetuning. We report single-view accuracies for all datasets, using ViViT-Base backbone.
(a) Kinetics 400 as upstream classification dataset
(b) Moments in Time as upstream dataset
Pretrain
Finetune
AVA
K400
Pretrain
Finetune
AVA
MiT
K400
74.9
MIT
36.8
K400
AVA
23.4
MIT
AVA
24.8
K400 AVA
25.2
76.2
MiT AVA
26.1
38.1
of data, showing that co-finetuning achieves superior action detection results on AVA, as well as improved video classification accuracy, in all cases.
Note that in all experiments, our VIVIT backbone model is initialised from an ImageNet-21K pretrained checkpoint, following common prac- tice in training transformer models
Table Comparison of traditional finetuning, and our proposed co-finetuning, using two upstream datasets. Co- finetuning consistently outperforms traditional finetuning with the same overall data, even considering that there are multiple orders which two upstream datasets can be pre- trained in with traditional finetuning (K400~>MiT denotes first pretraining on K400 and then MiT).
for video (Arnab et al. 2021a Berta- Sius et al. 2021]] | Liu et al. [0222 Yan
al., 2022] Zhang et al. 2021 and
Pretrain
Finetune
AVA
K400
MiT
we omit this from the "Pretrain"
col-
K400 MIT
MIT
37.2
umn for clarity.
K400
76.3
Table aa) shows that co-finetuning on Kinetics 400 and AVA jointly improves the mAP on AVA by 1.8 points, or 7.7% relative, whilst using the same total amount of data. An added advantage of co-finetuning is that the model can per- form multiple tasks unlike traditional transfer learning, the model does not "forget" the upstream dataset (French) 1999) it is simultaneously optimised to perform both tasks. We can that single-crop classification performance on Kinetics 400 increases by 1.3 points. Moreover, there is a similar increase for Moments in Time (MiT) in Tab. (b).
K400 ~MIT
AVA
25.3
MiT-KK00
AVA
25.1
MIT K400 + AVA
26.3
76.7
38.9
Table Tb) shows that co-finetuning on MiT also improves AVA action localisation accuracy, achiev- ing an improvement of 1.3 points. Note that existing work reporting results AVA (Amab al.] 2021b Girdhar et al. 2019 Feichtenhofer et al. 2019 Pan et al., 2021) have used Kinetics as the upstream classification dataset for transfer learning. However, Tab. Ib)) shows that pretraining on MiT outperforms Kinetics 400 by 1.4 points. Note that whilst MiT contains more video clips than Kinetics 400 (800,000 versus 240.000). its clips are also much shorter 3 seconds compared the 10 seconds of Kinetics, and hence the total duration of video in both datasets is similar.
Finally, Tab, 2 considers the scenario where we have two upstream classification datasets, Kinetics and MTT. With traditional transfer learning, there are two orders in which we can pretrain on the upstream clip classification datasets: first on Kinetics and then MiT, or vice versa. We observe that co-finetuning outperforms both of these alternatives on AVA action detection, and moreover, does not require the additional training hyperparameter of choosing the order to use the upstream datasets.
Adding more co-finetuning datasets Table 3 shows how adding more clisccaasiiiiattion datasets for co-finetuning improves our spatio-temporal action detection performance on AVA. We ob- serve progressive improvements from adding Kinetics, Moments in Time (MiT) and Something- Something (SSv2) respectively, although the gains diminish with additional datasets. We chose these clip classification datasets for co-finetuning as they are the largest available public datasets that we are aware of. Note that Kinetics and Moments in Time consist of videos that are similar in domain to AVA as they are collected from YouTube, and feature people. SSv2 (Goyyl et al. 2017). on the other hand, consists of videos from a different domain, consisting mostly of objects being ma- nipulated, and requires a model to capture fine-grained motion patterns. Nevertheless, we observe noticeable improvements from using SSv2, suggesting that the additional data has regularising effect on the model, and that motion cues learned from SSv2 are useful for AVA detection.
What classes does co-finetuning benefit? A hypothesis for co- so-finetuming was that it enables model to learn discriminative visual patterns on one dataset that transfer to the other, and that this would particularly benefit rare classes in the target dataset that the model with otherwise overfit on.

---

Under review as a conference paper at ICLR 2023
Table 3: By using more clip-classification datasets for co-training, spatio-temporal action detection
performance, measured by the mAP on AVA, gradually increases. Note how the improvement is primarily in the rare classes, denoted by the "Mid" and "Tail" categories. Refer to Sec. 4.2 and
Fig. for additional details on how these splits were constructed.
Training / Co-training datasets
Head
Mid
Tail
Overall
K400 traditional finetuning baseline
65.5
29.7
6.9
23.4
K400 AVA
65.6
31.8
8.9
25.2
K400 + MiT + AVA
66.1
33.7
9.5
26.3
K400 MiT SSv2 + AVA
66.7 +1.2
34.1 4444
10.1
+3.2
26.8 26.8+3.44
To validate this hypothesis, we split the 60 class labels from AVA into "Head", "Mid" and "Tail" classes. "Head" classes are defined as those with more than 10,000 labelled ground truth examples in the training set, whereas "Tail" classes have less than 1000 instances. Classes which don't don't fall into either of these categories are defined as "Mid" classes. There are 8 "Head", 23 "Mid" and 29 "Tail" classes respectively, as can be seen in Fig. and Tab. B of the appendix.
As shown in Tab. 3 adding more co-finetuning datasets improves our AVA detection mAP for "Mid" and "Tail" classes. In particular, or 14.8% relative. The relative improvement for "Tail" classes is even more, at 46.4%. Figure details this further, showing the improverment from co-finetuning each action class.
best oo-finetuned model improves "Mid" classes by 4.4 points,
Improvements from training strategy compared to modelling T shows that by leveraging public clip classification datasets, we are able to increase performance on AVA, measured by the mAP, by 3.4 points. To put this in context, we notice that the spatio-temporal actor-object graph proposed by Arnab et al. (202Ib) improved the author's 's baseline by 2.2 points. The external memory Wu et al. (2019) which requires precomputing video features for the entire video offline before training and inference obtains an improvement over the baseline of 2.9 points. The method of Pan et al. (2021). which combines external memories and spatio-temporal graphs, improves over its respective baseline by 3.4 points, but is significantly more complex. Therefore, we conclude that co-finetuning on public clip-classification datasets provides a simple method of increasing action detection models by amounts commensurate with, even higher than, the latest state-of-the-ar methods. We caution, however. that definitive comparisons are difficult to make as each of the aforementioned methods uses slightly different baselines, backbones and code bases.
Minibatch sampling strategy for co- finetuning Table 4 evaluates the effect of the two different minibatch sampling strategies described in Sec. [.22 The "alternating"
Table 4: Effect of different minibatch sampling strategies. We fo-finetune on K400, MiT and AVA simultaneously.
AVA
K400
MiT
minibatch strategy samples an equal number of minibatches from each of the datasets during co-finetuning, regardless of the size of the dataset. Consequently, the accuracy on the largest dataset (MiT) decreases not enough iterations of backpropagation are applied to it. On the other hand, the accuracy on the smallest dataset (AVA) decreases as too many optimisation iterations performed it, and the network begins to overfit on it. The "Weighted sampling" strategy does not suffer these problems, and performs better overall, as batches are drawn from the co-finetuning datasets in proportion to the size of each dataset. As result, we the "Weighted sampling" strategy in all of our experiments.
Alternttnn Weighted sampling
25.0
76.4
34.8
26.3
76.7
38.9
Leveraging larger-scale pretraining and resolutions Using higher resolutions as well as back- bone models pretrained on larger datasets is common method of improving performance. Ta- bles and 7 of the appendix shows that we continue to achieve substantial improvements from co-finetuning in both of these scenarios. For large-scale pretraining,
pretrained our ViVIT back- bone on the larger Weak Textual Supervision (WTS) dataset of SStouu et al. (2020) which consists of about 60 million weakly-labelled, web-scraped videos.
4.3 COMPARISON THE STATE-OF-THE-ART
AVA We compare to the existing state-of-the-are approaches AVA in Tab. 5 All methods, excluding TubeR (Zhao et al. 2022) and WOO (Chen et al.] 202T) use external person detections as inputs the model. We the same person detections as Fan al. (2021): Feichtenhofer et al.
8

---

Under review as a conference paper at ICLR 2023
Table 5: Comparison to the state-of-the-ar (reported with mean Average Precision) on AVA Gu
a 2018) and AVA-Kinetics et al. (2020). For AVA, we use the latest v2.2 annotations. Previous et al.
work followed the traditional finetuning approach of pretraining on Kinetics and then finetuning on
AVA. Note that separate models are trained for AVA and AVA-Kinetics
Pretrain
Views
AVA
AVA-Kinetics
Action Transformer (Li et al. 2020)
K400
23.0
MVīT-B (aan et al.
2021
K400
27.3
Unified Arnab et al.
(021bb
K400
27.7
WOO C
Chen et al.
2021)
K600
28.3
MViT-B (Fan et al.
2021
K600
28.7
SlowFast RTOT (reichtenhofer et al.
2019)
K600
29.8
ACAR (Pan et al.
2021)
K600
31.4
AIA (Tang al. 2020)
K700
18
32.3
ACAR (Pan et al.
2021)
K700
33.3
35.8
TubeR Zhao al.
2022
IG-+K400
33.6
ViViT-L (K770++MT+5SV2)
32.8
33.1
ViViT-L (K700+MiT+SSV>
WTS
36.1
36.2
2019; Arnab et al. (b:21b); Wu et al. (2019) as aforementioned. For state-of-the-art comparisons, 320 on the shorter side. Similar to object detection 2018), find that increasing the spatial resolution We perform inference on single view, noting that prior work often averages results multiple resolutions, and left/right flips (Feichtenhofer al.,
our model uses frames with resolution of
in images (Lin al. 2017 Singh & Davis
improves results, as detailed in Appendix
2019 Arnab et al. 2021b Tang et al. 2020 Zhao et al. 2022).
achieve an AP of 32.8. The previous best supervised-leariigg result on AVA, was achieved by TubeR Zhao al. (2022) which used pretraining on the Instagram (IG) dataset (Ghadiyaram al. 2019) contain- ing 65 million videos. When we pre- treitrain our initial model with WTS (Stroud et al. 2020) (which consists of 60 million videos, and thus of a similar scale IG), we achieve mean AP of 36.1.
Table 5 shows that by co-finetuning just publicly available classification datasets,
AVA-Kineties The final column of Tab. compares previous methods AVA-Kinetics. We use the same person detections as Li et al. (222)), which obtain an mAP for person detection of 77.5 on the Kinetics split of this dataset. ACAR (Pan al. 2021 Chen et al. 2020a). in contrast, uses a better person detector which achieves an mAP of 84.4, but has not been released.
We can separate the effect of person detections by evaluating using ground truth bounding boxes stead. In this case, our model without "TIS-preraiinig achieves an mAP of 47.0 on both Kinetics Box and the overall dataset. ACAR 3.4 points lower using ground truth boxes Kinetics attaining 43.6 (the authors do not report the overall dataset mAP in this setting).
With WTS pretraining, our foneetuueed model achieves, to our knowledge, the best known result on this dataset with mAP of 36.2.
5 CONCLUSION AND FUTURE WORK
Recent advances in spatio-temporal action localisation have been driven by developing more com- plex models which use external memories to capture long-term temporal context (Wu et al., 2019 Pan et al. 2021; Tang al. 2020) construct spatio-temporal graphs of actors and objects (Arnab) ct al., 2021b Pan et al. 2021; Sun et al.) 2018 Zhang et al. 2019). contrast, we show how we can achieve similar accuracy improvements by using simple model, and altering the training strat- egy. By leveraging our proposed co-finetuning strategy with additional, public clip-classification datasets and large-scale pretraining achieved substantial improvements on the challenging AVA and AVA-Kinetics datasets. In particular, our co-finetuning method improved substantially on the rare classes in these long-tailed datasets, as it had regularising effect, enabling the network to learn feature representations that transfer between different datasets.
Future work is explore co-finetuning for other tasks and domains, and also use more complex models.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lučić, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021a.
Anurag Amnb, Chen Sun, and Cordelia Schmid. Unified graph structured models for video under- standing. In ICCV, 2021b.
Fabien Baradel, Natalia Neverova, Christian Wolf, Julien Mille, and Greg Mori. Object level visual reasoning in videos. In ECCV, 2018.
Irwan Bello, William Fedus, Xianzhi Du, Ekin Dogus Cubuk, Aravind Srinivas, Tsung- Yi Lin, Jonathon Shlens, and Barret Zoph. Revisiting resnets: Improved training and scaling strategies. In NeurlPS, 2021.
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is space-time attention all you need for video understanding? In arXiv preprint arXiv:2102.05095, 2021.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve. Nicolas Usunier, Alexander Kirillov, and
Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? new model and the kinetics dataset. In CVPR, 2017.
Rich Caruana. Multitask learning. Machine learning, 1997.
Shoufa Chen, Peize Sun, Enze Xie, Chongjian Ge, Jiannan Wu, Lan Ma, Jiajun Shen, and Ping Luo. Watch only once: An end-to-end video action detection framework. In ICCV, 2021.
Siyu Chen, Junting Pan, Guanglu Song. Manyuan Zhang. Hao Shao, Ziyi Lin, Jing Shao. Hongsheng Li, and Yu Liu. 1st place solution for ava-kinetic crossover in acitivitynet challenge 2020. arXiv preprint arXiv:2006. 09/ 16, 2020a.
Zhao Chen, Jiquan Ngiam, Yanping Huang, Thang Luong, Henrik Kretzschmar, Yuning Chai, and Dragomir Anguelov. Just pick a sign: Optimizing deep multitask models with gradient sign dropout. In NeurIPS, 2020b.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. In ICLR, 2021.
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi-scale convolutional architecture. ICCV, 2015.
Mark Everingham, SM Eslami, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. International journal of computer vision, 111(1), 2015.
Haoqi Fan, Bo Xiong, Karttikeya Mangalam, Yanghao Li, Zhicheng Yan, Jitendra Malik, and Christoph Feichtenhofer. Multiscale vision transformers. ICCV. 2021.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In
Robert M French. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 1999.
Deepti Ghadiyaram, Du Tran, and Dhruv Mahajan. Large-scale weakly supervied pre-training for video action recognition. In CVPR, 2019.
10

---

Under review as a conference paper at ICLR 2023
Rohit Girdhar, Joao Carreira, Carl Doersch, and Andrew Zisserman. Video action transformer net- work. CVPR, 2019.
Ross Girshick. Fast r-cnn. In ICCV, 2015.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for aceu- rate object detection and semantic segmentation. In CVPR, 2014.
Raghay Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In ICCV, 2017.
Chunhui Gu, Chen Sun, David Ross, Carl Vondrick, Caroline Pantofaru, Yeqing Li, Sudheendra Vijayanarasimhan, George Toderici, Susanna Ricco, Rahul Sukthankar, et al. Ava: A video dataset of ppatio-temporalyy localized atomic visual actions. CVPR, 2018.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Roei Herzig, Elad Ben-Avraham, Karttikeya Mangalam, Amir Bar, Gal Chechik, Anna Rohrbach, Trevor Darrell, and Amir Globerson. Object-region video transformers. arXiv preprint 10.0615,, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Weinberger. Deep networks with stochas- tic depth. In ECCV, 2016.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. In arXiv preprint arXiv: (tt5:::770.06660 06950, 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. CVPR, 2018.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. In ICLR, 2015.
lasonas Kokkinos. Ubernet: Training universal convolutional neural network for low-, mid- and high-level vision using diverse datasets and limited memory. In CVPR, 2017.
Okan Köpüklüä Xiangyu Wei, and Gerhard Rigoll. You only watch once: A unified enn architecture for real-time spatiotemporal action localization. arXiv preprint arXiv:19/1.0666 1.06644, 2019.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In NeurlPS, 2012.
Vitaly Kurin, Alessandro De Palma, Ilya Kostrikov, Shimon Whiteson, and M Pawan Ku- mar. In defense of the unitary scalarization for deep multi-takk learning. In arXiv preprint arXiv:2201.041 22122200..4 2022.
Ang Li, Meghana Thotakuri, David Ross, João Carreira, Alexander Vostrikov, and An- drew Zisserman. The ava-kinetics localized human actions video dataset. In arXiv preprint arXiv:2005.00214 2020.
Tsung-Yl Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In ICCV, 2017.
Ze Liu, Jia Ning, Yue Cao, Yixuan Wei. Zheng Zhang, Stephen Lin, and Han Hu. Video swin transformer. In CVPR, 2022.
Kevis-Kokitsi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multi- ple tasks. In CVPR, 2019.
Thomas Mensink, Jasper Uijlings, Alina Kuznetsova, Michael Gygli, and Vittorio Ferrari. Factors of influence for transfer learning across diverse appearance domains and task types. PAMI, 2022.

---

Under review as a conference paper at ICLR 2023
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million vidcos for event understanding. PAMI, 42(2):002-508, 2019.
Junting Pan, Siyu Chen, Mike Zheng Shou, Yu Liu, Jing Shao, and Hongsheng Li. Actor-context- actor relation network for spatio-temporal action localization. In CVPR, 2021.
Sinno Jialin Pan and Qiang Yang. A survey on transfer learning. IEEE Transactions on knowledge and data engineering, 22(10), 2009.
Ning Qian. On the momentum term in gradient descent learning algorithms. Neural networks, 12 (1), 1999.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. In arXiv preprint arXiv:1910. 10683, 2019.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS, 2015.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In NeurlPS, 2018.
Bharat Singh and Larry Davis. An analysis of scale invariance in object detection snip. In CVPR, 2018.
Jonathan C Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, and David Ross. Learning video representations from textual web supervision. arXiv preprint arXiv:2007.14937. 2020.
Chen Sun, Abhinav Shrivastava, Carl Vondrick, Kevin Murphy, Rahul Sukthankar, and Cordelia Schmid. Actor-centric relation network. In ECCV, 2018.
Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In CVPR, 2016.
Jiajun Tang, Jin Xia, Xinzhi Mu, Bo Pang, and Cewu Lu. Asynchronous interaction aggregation for action detection. In ECCV, 2020.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In arXiv preprint 2rXiv:20/2. 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. NeurIPS, 2017.
Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool. Temporal segment networks: Towards good practices for deep action recognition. In ECCV, 2016.
Sinong Wang, Belinda Li, Madian Khabsa, Han Fang, and Hao Ma. Linformer: Self-attention with linear complexity. In arXiv preprint arXiv:2006. 04768, 2020.
Xiaolong Wang and Abhinav Gupta. Videos space-time region graphs. In ECCV, 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He. Non-local neural networks. In CVPR, 2018
Ross Wightman, Hugo Touvron, and Hervé Jégou. Resnet strikes back: An improved training procedure timm. In arXiv preprint arX0v:2110.00476, 2021.
Chao-Yuaa Wu and Philipp Krahenbuhl. Towards long-form video understanding. In CVPR, 2021.
Chao- Yua- Wu, Christoph Feichtenhofer, Haoqi Fan, Kaiming He, Philipp Krahenbuhl, and Ross Girshick. Long-term feature banks for detailed video understanding. In CVPR, 2019.
12

---

Under review as a conference paper at ICLR 2023
Shen Yan, Xuehan Xiong, Anurag Arnab, Zhichao Lu, Mi Zhang, Chen Sun, and Cordelia Schmid. Multiview transformers for video recognition. In arXiv preprint arXIv:2201.04288, 2022.
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
Yanyi Zhang, Xinyu Li, Chunhui Liu, Bing Shuai, Yi Zhu, Biagio Brattoli, Hao Chen, Ivan Marsic, and Joseph Tighe. Vidtr: Video transformer without convolutions. In ICCV, 2021.
Yubo Zhang, Pavel Tokmakov, Martial Hebert, and Cordelia Schmid. A structured model for action detection. In CVPR, 2019.
Jiaojiao Zhao, Xinyu Li, Chunhui Liu, Shuai Bing, Hao Chen, Cees GM Snoek, and Joseph Tighe. Tuber: Tube-transformer for action detection. In CVPR, 2022.
Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl. Objects as points. arXiv:1904.07850, 2019.
arXiv preprint
13

---

Under review as a conference paper at ICLR 2023
Table 6: Effect of spatial resolution on performance on AVA, measured by the mAP. We denote the resolution height x width, where the height is the shorter side typically reported in the literature. Observe how increasing the spatial resolution consistently improves results for both the baseline and our co-finetuning method. Moreover, note how the improvements achieved by co-finetuning are consistent across these resolutions, using a ViViT-Base backbone.
Resolution
Baseline
Co-finetuning
140 224
23.4
26.3
220
352
26.9
29.1
260 416
27.5
30.1
320 512
28.5
31.0
Table 7: Effect of using the web-crawled WTS dataset (Stroud et al. 2020) for initial pre-training,
using a ViViT-B 320p model. Even with stronger initialisation, co-finetuning provides benefits.
AVA
Traditional finetuning
33.3
Co- Co-finetuning
35.2
APPENDIX
In this appendix, we include our broader impact statement (Sec. A). additional ablation experiments
(Sec. B and reproducibility statement and additional experimental details (Sec. C
A BROADER IMPACT
Our work presents an improved method for performing detection and classification tasks in video. These video recognition tasks represent general technology with a wide range potential appli- cations. While we are unaware of all potential applications, it is important to be aware that each application has own merits and societal implications depending on how the system is built and deployed. We also note that training datasets may contain biases that may render models trained on them unsuitable for certain applications.
B ADDITIONAL EXPERIMENTS
Co-finetuning compared to training for more iterations In this experiment, confirm whether our improvements from co-finetuning are just because we are actually training for longer:
We finetuned model for AVA for 53 epochs, which is the finetuning on AVA for 20 epochs and Kinetics 400 for 30 epochs. Traditional finetuning gets only 23.8 mAP, whilst co-finetuning achieves 25.2 mAP (Tab. [a], showing that our improvements are from our proposed method, not training longer.
number of iterations
Effect of spatial resolution on performance on AVA Table 6 studies the effect of spatial res- olution on action localisation performance on AVA, using ViViT-Base backbone. We observe consistent improvements from increasing the spatial resolution, and this is known to improve ac- curacy for object detection of images too (Lin et al. 2017; Singh Davis 2018). Moreover, the improvements achieved by co-finetuning are consistent across all the different resolutions that we evaluated.
Effeet of large-scale pre-training As mentioned the main paper, the VIVIT model (Arnab et al. 2021a) that
our backbone is initialised from ViT image model (Dosovitskiy et al.) 2021 that i8 trained on ImageNet-21KK (Deng et al. 2009), like other popular video architectures (Bertasius
al. 2021
Liu et al. 2022 Yan et al. 2022 Zhang al. 2021).
14

---

Under review as a conference paper at ICLR 2023
Frequency
100
Miaa AP (19
Watch per
Stand
Talk tolk-t
Usten to
Sit
Carry Walk
Bend bow
Lie silesp Iale
Run jeg
Anower phone Danca
Smoke
Crouch knnes)
Read
Martial 3rt
Grab
Sing to
open
Drive
Hend clsp
Get up
Hug
serve
Closs
Writs
Kiss
Licten
Hand shake
Take
Pus down
piox
Tokt
per
Hand wove
Pull
Dress
sown
climb
Throo
jump rest
Shoot
Cutl
Take photo
Hit
Work comp.
Turn
swin
Point
104
Weath
Give
Fal
Figure 3: Per-class results on the AVA dataset (Gu al. 2018) for our best model, achieving a mean AP of 36.1. The label frequency, shown with a log-scale, used to divide the classes into "Head", "Mid" and "Tail" categories.
In Tab. 7 we also consider initial pretraining on the larger Weak Textual Supervision (WTS) (Stroud al. 2020 dataset which consists of about 60 million weakly labelled videos scraped from the web. Table shows that co- ro-finetuning (with K400, MiT and SSv2) still improves our model's 's accuracy in this case by 1.9 points. Note that these models trained with resolution of 320 the shorter side.
Per-class results
Figure shows detailed per-class results of our best model on the AVA dataset.
C REPRODUCIBILITY STATEMENT AND ADDITIONAL EXPERIMENTAL DETAILS
Reproduciblity Statement We will release all code and models upon acceptance. We have also included exhaustive descriptions of our implementation details below and in Sec. 4.1 of the paper. We have also followed standard experimental protocols the community for all of the datasets that have used for our experiments.
Additional implementation details We trained all of our models with 32 frames sampled with a temporal stride of 2. We trained our models using synchronous SGD with momentum of 0.9, and a global batch size 128.
We set the base learning with a linear warm-up for the first 2.5 epochs. For our baseline strategy, we finetuned all models for 20 epochs on both AVA and AVA-Kinetics. For our co-finetuning strategy, the number of training epochs was set to 15.
to 0.2 for all models, and followed cosine learning rate schedule
For data augmentation, we applied scale jittering to the input frames with ratio uniformly sampled from the range [0.65, 1.1]. We also applied random jittering on the person detections used during training with a maximum perturbation ratio of 0.15, For additional regularisation, we also applied label smoothing (Szegedy et al. (016) and stochastic depth (Huang et 2016) and their coefficients were set to 0.1 and 0.2, respectively.
For AVA, our models are trained on detected person boxes and they are first thresholded with a con- fidence score of 0.8 following Wu et al.] (2019); Feichtenhofer et al. (2019); Arnab et al. (2021b); Fan et al. (2021). For AVA-Kinetics, ground-truth person detections are used as the samples for training and we keep detections with confidence score greater than 0.2 for evaluation. The differ- ences between the confidence score thresholds is because for AVA, we used the Faster-RCNN ctt al., 2015) detector originally trained by Wu et al.] (2019), and for AVn-Kinn, we used the CentreNet (Zhou et al. 2019) detector originally trained by al. (2020).
List of head, mid and tail classes Table lists the "Head", "Mid" and "Tail" classes based on their label frequencies in the AVA v2.2 (Gu et 2018) training set.
15

---

Under review as a conference paper at ICLR 2023
We defined "Head" classes having more than 10,000 labelled examples, and "Tail" classes have less than 1,000 examples. Classes not falling into either of these categories are labelled "Mid".
16

---

Under review as a conference paper at ICLR 2023
Table 8: Label frequencies of each class in the AVA v2.2 (Gu et al. 2018) training set. We define "Head" classes as having more than 10,000 labelled examples, and "Tail" classes having less than 1,000 examples. Classes which do not fall into either category are "Mid" classes.
Head classes)
watch (a person)
168
stand
166357
talk to (e.g., self, a person, group) carry/hold (an ob- ject)
110267
listen to (a person)
106816
sit
100323
80451
walk
40771
touch (an object)
17133
Mid (23 classes)
bend/bow
(at the
8349
lic/sleep
5356
ride (c.g., bike, car, a horse) dance
4808
waist)
run/jog
3337
answer phone
3279
3267
eat
3025
smoke
2991
fight/hit (a person)
2695
drink
2335
crouch/kneel
2321
read
2146
martial art sing to (e-g., self,
2117
grab (a person)
2003
watch (e.g.. TV)
1993
1643
play musical in- strument
1297
open (e.g., win-
1251
person, group)
dow, car door)
drive (e.g., car, truck)
1188
hand clap
1187
get up
1124
hug (a person)
1103
give/serve (an ob- ject) to (a person)
1073
Tail (29 classes)
close (e.g.. door,
786
write
777
sail boat
719
box)
kiss person)
714
listen (e.g., mu- sic)
668
hand shake
619
take (an object)
608
put down
533
lift/pick up
519
from (a person)
text on/look at a cellphone
415 lift (a person)
400
push (an object)
387
push (another per-
355
hand wave
351
pull (an object)
344
son)
dress/put on cloth-
342
fall down
291
climb (e.g.. mountain)
a
268
ing
throw
249
jump/leap
245
enter
225
shoot
222
cut
184
take photo turn (c.g., screw-
181
hit (an object)
177
work on
com-
176
137
puter
driver)
swim
111
point to (an object)
97
17


Reviews:
[{"review_id": "tAHnUwgPUHC", "reviewer": "Reviewer_kFF7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes learning a network for action classification by gathering the target data with external datasets in a multi-task scenario. The method consists of a video transformer architecture (ViViT) that produces spatio-temporal features which are then “ROI-aligned” and fed into a single classifier. For the tasks with no bounding boxes an average pooling is applied to the full spatio-temporal features. The main study for the paper resides on learning with multiple datasets at a time. \n\nThe method is tested primarily on AVA 2.2 as well as on Kinetics and Moments in Time. A set of ablation studies and comparisons against state of the art works is also provided, showing promising results. \n", "summary_of_the_review": "The paper is an interesting report, but does not hold a strong contribution in the domain of action recognition or multi-task learning. The insight seems to be that training with more datasets in a multi-task scenario helps learning a stronger backbone. However, it is shown that without a proper pre-training (in the large-scale dataset of WTS) the method does not achive competitive results. For these reasons I do not feel in the position to recommend this manuscript for acceptance.", "strength_and_weaknesses": "The paper is rather weak and does not shed any light neither on multi-task learning nor on the domain of action recognition. In my humble opinion, the paper basically delivers the message that “training simultaneously on several datasets at the same time improves performance”, which does not stand strong to be accepted at ICLR. \n\n\nThe paper lacks a proper complexity analysis which is needed to understand where does it stand in terms of the tradeoff pretraining/model size-GFLOPS/mAP. The results on Table 5 are a bit misleading: how would a ViViT-L pre-trained on Kinetics-700 and then fine-tuned with the other datasets perform? Different pre-trainings affect the performance of the models drastically on AVA. Authors are encouraged to follow recent works such as MemVIT [CVPR’22], MViTv2 [CVPR’22] or WOO, and split the results according to pre-training, architecture details and complexity (FLOPs and Params). Without such analysis the paper sets a nice technical report on how to train with a combination of supporting datasets. \n\nIt is also important to note that one of the main motivations of the proposed approach is that “many video models are initialised from networks pretrained on ImageNet, then finetuned on Kinetics and then finally finetuned for spatio-temporal detection. In our proposed approach, we simultaneously finetune our model on multiple datasets”. However, pretraining the model on WTS gives +3.3% mAP from 32.8 to 36.1! Isn’t this against the main purpose of the paper? \n\nFinally, I believe the paper should include a clear breakdown of resources and time consumption to train the models between using the three step approach of pretraining on ImageNet, then on Kinetics, then on AVA, w.r.t. using the co-finetuning. What is the converge rate for each of the approaches? This needs to be included.\n\nIn summary, my impression is that the paper, while showing some interesting results, it does not hold a contribution strong enough to be considered for ICLR. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "sGR4QLqgoX", "reviewer": "Reviewer_L9xn", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose \"co-finetuning\" for action localization. Namely, instead of commonly used pre-training and finetuning 2-step procedure, with \"co-finetuning\" the pre-training tasks and downstream tasks are trained simultaneously. The authors show that this performs better than the standard 2-step procedure on datasets including AVA.", "summary_of_the_review": "Overall, my main concerns are regarding the experiments and novelty. I overall find this paper below the acceptance threshold. ", "strength_and_weaknesses": "Strengths:\n+ The authors conducted the experiments on multiple different datasets (including MiT, K400, AVA, SSv2). This helps understand the generalization of the proposed method on different settings.\n\nWeakness:\n- Regarding the MiT-K400-AVA experiments in Table 2, \"MiT + K400 + AVA\" (26.3) only outperforms \"Pretrain on K400 -> MiT + finetune on AVA\" (25.3) by 1%. I don't find the difference of 1% very significant. Furthermore, while the authors do control that both experiments use the same number of iterations, there're many ways to pick the number of epochs for each dataset in \"K400 -> MiT -> AVA\". It's possible that an \"optimal\" pick will further shrink the performance gain. Thus, from the experiment results alone, I'm still not convinced the effectiveness of co-finetuning. Similar concerns also apply to results in Table 1 and 3.\n- Regarding the results in Table 1, I wonder why the K400 experiment in this paper obtains 74.9 using ViViT-B while the original ViViT paper obtains 79.9. I wonder if this suggest the K400 models are not sufficiently (pre-)trained in this paper. If that's the case, that might contribute to why co-finetuning helps --- it allows the model to train more on K400.\n- Regarding novelty, I still don't find the proposed method very different from multi-task learning. Furmore, there are also prior works, such as \"Simple Multi-dataset Detection\" by Zhou, Koltun, and Krahenbuhl, that also train on multiple datasets. I find the discussions comparing with those papers missing.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Jo3Yg9r0FH", "reviewer": "Reviewer_mBx1", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper challenges the widely-used \"pretrain-then-finetune\" pipeline, ie, the model is first pre-trained on a large \"upstream\" dataset, and then is finetuned on a smaller \"downstream\" dataset. In contrast, this paper proposes a co-finetuning training strategy. As the name suggests, they try to train a single model on multiple \"upstream\" and \"downstream\" tasks simultaneously. Results on the spatio-temporal action localization tasks show that the model with co-finetuning strategy achieves better performance than the conventional pretrain-then-finetune strategy.", "summary_of_the_review": "The paper focuses an interesting problem, and tries to challenge the prevalent pretrain-then-finetune pipeline. However, the proposed method is a plain multi-task training, which has no new technique contributions. Meanwhile, the proposed method is general, and it would be more convincing by reporting more results on other different tasks.", "strength_and_weaknesses": "## Strengths\n+ The direction for exploring more effective knowledge transfer is interesting and promising. If a new effective and efficient paradigm can be discovered, it will make a huge contribution to our community.\n\n## Weaknesses\n+ The proposed idea is quite simple and naive. It follows conventional multi-task training, where multiple tasks share the same backbone. Meanwhile, they propose two types of sampling strategies for obtaining sample mini-batches. Thus, there is no technique contributions in this paper.\n+ Since the method looks general enough, it would be more convincing to share more results on different tasks. In another word, it is not clear why this proposed framework is more suitable for spatio-temporal grounding than other tasks?\n+ From Table 4, the results are sensitive to different minibatch sampling strategies. For example, the method with \"Alternating\" samples achieves much worse results than the baseline models (cf. Table 2 and Table 4)", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Bz7bAIdAfwj", "reviewer": "Reviewer_wkyw", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes to leverage multiple upstream datasets to co-finetuning for spatio-temporal action localization. The main contribution of this work is a transfer learning model that involves multiple classification heads for downstream detection tasks.", "summary_of_the_review": "This paper advocates co-finetuning as transfer learning technique for spatio-temporal action localization, however, if the focus is on \"transfer learning\", the authors should compare with more recent transfer learning approaches instead of simply traditional fine-tuning, if the focus is a simple approach for the downstream task, as I mentioned, the performance is worse than current SoTA.", "strength_and_weaknesses": "Strength:\nThe observation that involving multiple upstream tasks in improving spatio-temporal action localization.\n\nWeakness:\n1. The novelty is trivial, the idea of transfer knowledge classifier for object detection is already proposed in previous work[1]. This work extend similar idea to a new task with some improvements like using multiple upstream tasks. The way to use multiple upstream tasks is simply co-finetuning, which is less novel.\n\n2. The experiment is not convincing, Table 1 and 2 only compare with traditional fine-tuning, however, there are plenty of transfering learning approaches that available and you could compare with some of them. A simple package can be found here:https://github.com/thuml/Transfer-Learning-Library\n\n3. Table 5 misses some SoTA methods, which can be found here:https://paperswithcode.com/sota/action-recognition-on-ava-v2-2\nit seems that the reported performance is worse than current SoTA.\n\n\n\n[1]Guo, Shuxuan, Jose M. Alvarez, and Mathieu Salzmann. \"Distilling Image Classifiers in Object Detectors.\" Advances in Neural Information Processing Systems 34 (2021): 1036-1047.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject