PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Authors: Valerii Likhosherstov
Anurag Arnab
Krzysztof Marcin Choromanski
Mario Lucic
Yi Tay
Mostafa Dehghani
Venue: Accepted by TMLR ()

Abstract: Can we train a single transformer model capable of processing multiple modalities and datasets, whilst sharing almost all of its learnable parameters? We present PolyViT, a model trained on images, audio and video to answer this question. PolyViT consists of a single transformer backbone, modality-specific tokenizers and task-specific output heads. By co-training on different tasks of a single modality, we are able to achieve significant accuracy improvements on 5 standard video- and audio-classification datasets. Furthermore, co-training PolyViT on multiple modalities and tasks leads to a parameter-efficient model which generalizes across multiple domains. In particular, our multi-modal PolyViT trained on 9 datasets across 3 modalities uses 8.3 times fewer parameters and outperforms a state-of-the-art single-task baseline on 2 of these datasets, whilst achieving competitive performance on the others. Finally, this simple and practical approach necessitates less hyperparameter tuning as the per-task hyperparameters can be readily reused.

Published in Transactions on Machine Learning Research (12/2022)
PolyViT: Co-training Vision Transformers on Images, Videos and Audio
Valerii Likhosherstov University of Cambridge
vl30, @com. uk
Anurag Arnab* Google Research
aarnabdggoogle.co com
Krzysztof Choromanski Google Research
kchoro eatboro@ggogle.coo com
Mario Lucic
lucic@google.com
Google Research
Yi Tay Google Research
yitay@google..om com
Adrian Weller
au665 uk
University of Cambridge & The Alan Turing Institute
Mostafa Dehghani"
dehghani@ggoolee.com
Google Research
Reviewed OpenReview: /ttps::/openrevieee net/ formm/ id= zknqZeUCLO
Abstract
Can we train a single transformer model capable processing multiple modalities and datasots, whilst sharing almost all of its Icarnable parameters? We present PolyVIT, model trained on images, audio and vidco answer this question. PolyViT consists of single transformer backbone, modality-specifie tokenizers and task-specific output heads. able to achieve significant
By co-training different tasks of single modality, accuracy improvements on 5 standard vidco- and audio-lassiiication datascts. Furthermore, co-training Poly on multiple modalities and lasks leads lo a parameter-effiiien model which generalizes across multiple domains. In particular, our multi-modal Poly trained on datasets across modalities uses 8.3 times fewer parameters and outperforms a state-of- the-art single-task baseline 2 of these datasets, whilst achieving competitive performance the others. Finally, this simple and practical approach necessitates less hyperparameter tuning the per-task hyperparameters can be readily reused. To facilitate further rescarch, we have released code at httpp::/ggithui
1 Introduction
Transformers (Vaswani al. 2017) are a flexible family of neural pequence-to-sequena models. While they were originally designed for natural language processing, they have recently been adapted a range of perception tasks, such as classification of images (Dosovitskiy et al. 2021), video (Arnab et al. 2021 and audio (Gong et al. 2021). h. Despite recent advances across different domains and tasks, current stalle-f-thheart methods train separate model with different model parameters for each task at hand.
"Equal contribution.
TTe work done during author's internship Google.

---

Published in Transactions on Machine Learning Research (12/2022)
PolyViT Tokenizer
PolyViT Encoder
Task Head
Input
Task #2 Input
Task #2 Head
Image Toxerizer
BaK #3
Trensformet Leyer #1
Transform mer Lyyer #2
Taskc #3 Head
Imput
Pemise Peeee Layer #L
Task 84
Tassk #4 Head
Imput
laak #5
Video Takerizzr
Task #5 Head
Input
Teak 96 Input
Task #6 Head
Task #7 Input
Autio
ask #7 Head
Figure 1: Overview of ViI. Our model is capable of performing multiple tasks spanning different modal- ities, and processes single task at a time. The architecture consists of transformer encoder shared among all tasks, modality-specifie input tokenizers and task-specific output heads.
In this work, we present simple yet effective method of training a single, unified model (Fig. 1 that achieves competitive, or state-of- thr--rrttt-- results for image, video-, and audio-classificttinn We go beyond using a common architecture for different modalitics (Jacgle al. 202T), as we also share model paramoters across tasks and modalities, thus enabling potential synergies. Our approach motivated both technically, by the fact that transformers generic architectures that can operate on any modality that can be tokenized, and intuitively, since human perception is inherently multimodal and performed by single brain.
Our main technique is co-training: training a single model on multiple classification tasks (across potentially multiple modalities) simultaneously. We consider various settings, and simultaneously solve as many as 9 different image, video- and audio-clasiffecttion tasks. Our model capable of performing multiple tasks, but performs a single task at time for given input (see Fig.
Although similar techniques have been explored in computer vision Maninis et al. 2019 and natural language processing (Raffel et al. 2019). to knowledge, this is the first approach using multiple modalitics and achieving state-of-the-ari performance.
The proposed co-tra co-training approach has several benefits: It is parameter-efficient as we share the transformer parameters for each the n tasks of interest, approximately reducing the number parameters by a lactor of n. This has practical advantages when deploying models on computationally constrained devices (such smartphones) with limited memory which may not otherwise be able to fit the weights of different models (landola et al. 2017 Dehghani et al. 2021a). Morcover, maintaining single model for multiple tasks simplifies model deployment and online updates (landola et al., 2017). Furthermore, co-training tasks of the same modality leads to accuracy improvements on each individual task whilst also linearly decreasing total parameters. In particular, we achieve significant improvements on video and audio classification across different datasets. This is facilitated by our observation that co-training has a regularizing effect, that improves performance on smaller datasets that large transformer models would otherwise overfit on. Tn addition, when we extend co-training to multiple tasks and modalitics, we observe that our accuracy still competitive with the state-of-the-art whilst being even more parameter-efficient our model trained on 9 datasets uses 8.3 times fewer parameters whilst having at most 1.2% accuracy drop compared to state- of-the-art single-task basclines. Finally, linear probing experiments show that this multi-task, multi-modal model is able to learn representations that generalize across multiple tasks and domains. Once again, this has practical advantages when deploying models, as it shows that we can add new capabilities to the model by simply training an additional linear classifier.
In addition all the benefits outlined above, our co- co-training setup is simple and practical to implement. It does not require hyperparameter tuning for each combination of co-training datasets, as can readily
2

---

Published in Transactions on Machine Learning Research (12/2022)
adapt the settings of standard, single-task training. In addition, co-training does not increase the overall training cost either, as the total number of training steps does not exceed that of the sum of each single-task baseline.
2
Related Work
Our model is related to multi-task learning and transformer models, which we discuss below.
Multi-task learning Multi-task learning aims to develop models that can address multiple tasks whilst sharing parameters and computation between them |Caruana 1997). In computer vision, multiple papers have developed models which predict multiple outputs (for example semantic segmentation and surface normals), given a single input image (Figen & Fergus 2015 Kokkinos 2017 Zhang et al. 2014b. Numerous works have also observed that although multi-task models are more versatile, their accuracies are lower than singlo-task models, and this accuracy deficit increases with the mumber of tasks, or by simultancously performing unrelated tasks [Kokkinos] 2017 Zamir et al. 2018 McCann et al.) 2018). Moreover, jointly training a network to simultancously perform multiple tasks has typically required careful calibration of the individual tasks, to ensure that none of the task-specific losses dominates another. Methods to mitigate this include gradient- normalization (Chen et al., (018)) and -surgery (Yu et al.) 2020) and adaptive loss weights [ennrr & Koltun 2018 Kendall et al. 2018) and curriculums (Guo et al.
2018 among others.
Our work differs in that although our network is capable of performing multiple tasks, it performs one task at a time for a given input. Note that this setting is also more suited to the case of handling multiple input modalities. Such an approach was also taken by (Maninis et al.] 2019) who named it "single-tasking of multiple tasks" in the context of computer vision. However, in natural language processing (NLP), this setting is still referred to as "multi-task learning" (Collobert & Weston 2008). Furthermore, our co- training strategy is simple, and alternates between performing SGD for batches of separate tasks. For high-capacity transformer models, we find that co-training on multiple datascts simultancously helps to regularize the model on a dataset that it would otherwise overfit on, thus achieving accuracy improvements from co-training. Previous works have improved performance on additional tasks only by introducing extra task-specific parameters (Misra et al. 2016 Houlsby et al. 2019 which are typically conditioned on the input RRebffi et al. 2017: Maninis et al. 2019).
We also note that similar co-training setups to our work have been explored in NLP. A recent paradigm in NLP has been to reduce different tasks to common, unified framework (Raffel et al. 2019
Brown et al,
2020 McCann et al. 2018). This common interface allows co-training single model to perform multiple
tasks, as it effectively involves concatenating multiple datasets together (Raffel et al.
2019 Khashabi et al.
2020 Tay et al. 2020)
Although the majority of previous multi-lask learning works have considered only a single modality, (Kisser ltt al. 2017) presented an early effort on multi-modal models. Their propose a heterogeneous model consisted of convolutional layers to process images, and attention and mixture-of-cxperts layers to model text. This work, however, did not analyse how to co-train these different tasks and modalities as our paper. Most recently, proposed using multi-task learning as the mean for transfer learning and showed that co-training upstream and downstream tasks, we can improve the transfer results compared to traditional pretraining- finetuning as sequential processes.
Most recently, Arnab et al. (022) proposed using multi-task learning as means for transfer learning and demonstrated that by co-training upstream and downstream tasks, the transfer results can be improved compared to traditional pretraining pretraining-inettuingg sequential processes.
Transformer models Our model, motivated by (Dosovitskiy et al. 2021), ) can readily handle diverse modalities, as transformers operate on any sequence of tokens. Relevant to us, Perceiver (Jaegle el al. 2021) and Perceiver-1O (Jaegle et al., 2022) is a transformer architecture that can process different modalities. Instead of tokenizing images or audio spectrograms with non-overlapping patches like (Dosovitskiy et al., 2021) and (Gong et al. 2021) respectively, (Jacgle et al.) (021) operate directly on the raw input by projecting it into a smaller, latent set of tokens using cross-altention. Although this architecture is capable of processing
3

---

Published in Transactions on Machine Learning Research (12/2022)
different modalities, the authors train separate networks with separate parameters for each task. Therefore, they do not consider co-training scenarios like our work. MRT (Nagrani et al. 2021 on the other hand, proposes a transformer model to fuse different modalities (for example audio and rgb frames of videos) to solve a single task. Once again, separate model paramcters are used for cach task.
UniT (Hu & Singh 2021 co-train a transformer-basel model, but specifically for vision-and-languagge tasks. The authors use an encoder-decoder architecture (Vaswani et al. 2017), where only the decoder is shared among different tasks, and the encoder is specialized for each modality. In particular, the visual encoder is DeTR (Carion et al. 2020) and the text encoder is BERT (Devlin et al.) 2019), and each component is pretrained separately. In contrast to our work, they do not consider scenarios where the entire transformer backbone is shared among different tasks, nor do they thoroughly analyze how to co-train multiple tasks and modalities like our work. Furthermore, their approach does not outperform single-task baselines as our work does. Other papers concentrating on multi-task learning of vision-and-language tasks include (T.u) ct al. 2019 Li et al. 2020; Lu et al., 2020). On a separate track, (Bain et al.] 2021) use a single transformer encoder to process both images and videos for video-text retrieval. However, their model is still trained single dataset and task, and the authors process images with the transformer as part of complex, murriuulumleeaning strategy. This is in contrast with our work which simultancously trains a model for multiple tasks across different modalities.
Additionally, we note that transformers have been used to process multiple modalities (Akbari et al. 2021 Lee et al. 2021) for cross-modal self-supervised learning (Alayrac et al., 2020 Miech et al. 2020). (Lee et al. 2021) train a transformer on top of visual and audio features obtained from convolutional networks. And to make model training computationally feasible, they perform low-rank approximations of the parameter matrices (Sainath et al. 2013 Yang et al., 2015 to reduce the total number of model parameters. These approaches are thus complementary to our work which shares almost all parameters in the model among different tasks.
Finally, we note that a concurrent work, Unified-10 (L et al. 2022) proposes a sequence-to-sequence model which can can perform various vision and language lasks with the same model. Although this work can handle a wide range of tasks, it is however, not competitive with the state-of-thee-rt as our approach.
3 Preliminaries
We define a modality as the type of input processed by the network. In this work, we consider images, audio, and video (specifically, the sequence of image frames in video) as three separate modalities. We perform classification it is fundamental problem whose solutions are often extended to more complex ones (Girshick et al. 2014 He et al. 2017). By task, we refer to pair of input modality and set of classes from which one or multiple classes are to be selected for a given input. Each task corresponds directly to a dataset, for example, ImageNet-1IK (Deng et al. 2009) for image classification or Kinetics 400 (Kay et al. 2017 for video classification.
3.1 Vision Transformers and extensions
The Vision Transformer (VIT, Dosovitskiy et al. 2021 is a transformer- based architecture for image classification that closely follows Vaswani et al. (2017). In contrast to language which intuitively tokenized into words, ViT extracts tokens from the input image, IMG Nxxx3 by splitting it into N [H//h] x [W/w] non-overlapping patches, X1.-..,XN XN Rhxwx3 Each patch, Iú, is then projected into token z; Rd by linear operator E, z; Ex; (input embedding operator). All tokens are then concatenated into sequence, which is also prepended with a learnable class loken 2ets E Rd. Learnable posilional embeddings p E [[N+ are also added to this sequence as the transformer is otherwise permutation invariant. We denote this tokenization process as
=
Zels
Exi
Exp] + +pp
(1)
Note that the linear operator can also be seen as a 2D convolution with kernel of size h x w and strides (h, w). The sequence of tokens, 2, is then processed by transformer encoder, consisting of T. layers. Fach
4

---

Published in Transactions on Machine Learning Research (12/2022)
layer, L, is applied sequentially, and performs the transformations,
yf MSA (LN (z (zl-)) z z' MLP (LN (y")) ye,
(2)
where MSA denotes multi-head self-attention (Vawwani et al. 2017). MLP is a neural network with a single
hidden layer and a GeLU nonlinearity (Heudrycks Gimpel 2016). and LN denotes layer normalization (Ba et al. 2016).
For C-class classification problem, the class logits produced by the model obtained by applying output linear head on the encoded classification token, z'is:
W Wouthiss + Dout E ERC,
(4)
where W
Wout out E RCxd and oott E RC are the linear head's learnable parameters.
Extensions of ViT to audio and video The Audio Spectrogram Transformer (AST) GGog et al. 2021 follows the same architecture as ViT, with the only difference that ils inputs are log-mel spectrogramss. Spectrograms are image-like, time- fime-ueeyeency representations of audio, and can be tokenized like images. Morcover, the best AST model was initialized from ViT models pretrained on large image datasots.
Video Vision Transformers (ViViT) (Arnab et al., 2021 are an extension of ViT to video. The authors proposed four model variants, and we consider the unfactorized one (Model in (Arnab et al.) (021 This model differs from VIT in the input tokenization process, which it extends from 2D image patches to 3D spatio-temporal "tubelets". Namely, video input x R=xxxxxx is split into N [F/f] [//h] x [W/w] non-overlapping tubelets Xi,..., XN E RRxxxxxx3 Following VIT, linear operator E which can be interpreted as a 3D convolution, projects {xi} into a sequence of tokens { = E x; E IRd), and repeats computations
Initialization Finally, note that ViT, VIVIT and AST all achieve their highost performance when pre- trained large-scale dataset such as ImageNet-211K (Deng et al., 2009) or JFT (Sun et. al. 2017). More specifically, ViT was initially pretrained on ImageNet-2I1 211 or JFT, and then finetuned at higher resolution on target datascts such as ImageNet-1K. ViVIT and AST also initialize their models from large-scale, image- pretrained models (Abnar et al.) 2021). In all of these cases, the positional embeddings, P, which depend on the sequence length N (and thus the input resolution), are interpolated from the pretrained model to the finetuned model. Furthermore, the 3D embedding projection of ViVIT, EVD, initialized from the 2D projection of ViT, IIMG (Arnab et al. 2021).
The similarities between VIT, VIVIT and AST allow to construct multi-modal model with shared transformer encoder, and separate input tokenizers as described next.
inference
4
Co-training ViT on images, audio and video
4.1 PolyViT architecture
ViT is single architecture that is capable of processing inputs from multiple modalities. As shown in Fig. we share transformer encoder among different tasks and modalities, cnabling up to a lincar reduction in parameters with the number of tasks. Note that ViT with L layers acts like an L-layer VIT when processing images, L-layer AST when processing audio, and an L-layer unfactorized ViVIT when processing video. While capable of handling multiple modalities, it performs one task from one modality in given forward pass.
As shown in Fig. VilyVIT employs modality-specific class tokens, BMG Vin AED Wels zlla input embedding operators, DIMG E E and positional embeddings Ii p p A This allows the network to encode modality-speciff information that can be leveraged by the subsequent, shared transformer backbone. It also accounts for the fact that the number of tokens per modality may vary.
5

---

Published in Transactions on Machine Learning Research (12/2022)
A separate output linear head (Eq. 4 is then used for each task with the following learnable weights:
W out
out E xRCx bout b mode E RC
(5)
where mod E (IMG, VID, AUD] a modality, j E ".....,,mody is a task index in the set of lasks for that modality, 1mno is the number of tasks for that modality and Cj the number of classes for that task. Note that the output heads are the only task-specific parameters. The input embodding operators, positional embeddings and class lokens are shared by all lasks within modality.
To increase model capacity when co-training on a large number of tasks and modalities simultaneously, we can optionally include Ladapt 2 0 modality-specifie transformer layers (which we denote as modality- adaptor layers). These transformer layers are applied directly after tokenization. In this case, there are Lishared I. - Ladupp layers which are shared among all modalities and tasks. We can think of this case as using& a shallower transformer encoder, but a deeper subnetwork to extract tokens from each modality.
As almost all computation and parameters within our architecture within the L layers of the transformer encoder, if there are n tasks, we reduce the total number of parameters by factor of approxinately n when Liharred = L. This is in comparison to standard, single-task training. Note that the overall inference time does not change, as PolyVit ViT still performs one task per forward pass.
4.2 Co-training procedure
We optimize all Poly model parameters, 0, simultaneously across all the tasks that we are co-training with first-order gradient-based optimizers such as stochastic gradient descent (SGD). For brevity, we will refer to each iteration of one of these optimizers as an "SGD step" below. There are a myriad of design choices on how to construct training batches, compute gradients, and tune the training hyperparameters.
In all cases, we construct our training minibatches using examples from a single task. This design choice allows us to evaluate gradients and perform a parameter update using the same training hyperparameters (e.g., learning rate, batch size, and momentum) as conventional single-task baseline. As result, we can perform co-training on multiple tasks without any additional hyperparameter tuning compared to the single- task baseline (Dosovitskiy et al. 2021 Gong et al., 2021 Arnab et al. 2021). making co-training simple to perform in practice, and alleviating the need to perform large hyperparameter sweeps in order to achieve competitive accuracy. Note that without this property, we would need to tune training hyperparameters on the product set of all co-training datasets, which would be computationally infeasible. Constructing mini- batches from a single task (where each example has the same mumber of tokens) has further computational advantages on GPU- or TPU-acoolcrators, as tokens do not need to be padded a maximum sequence length.
During co-training, for each SGD step, we sample task (dataset), then sample minibatch from that task, evaluate a gradient and then perform parameter update. An important considcration is the order in which we sample tasks and whether we accunulate gradients over different minibatches and tasks. We describe several task sampling schedules below and in Fig. 2 We first denote Uj as the number of SGD steps for the single-task baseline that the original authors reported for their best model, where j E {I...., T) indexes the task and T = TIMG TTAID + Twvn is the total number of tasks. Furthermore, we define U as the total number of SGD steps during co-training.
Task- Task-by-tast by-task In this schedule, the first Un SGD steps are performed with task in, the next Uμ steps using task j2 and so on, where ii.,..,jjr) jrl is a random task order.
This deterministic schedule alternates between tasks in a fixed, repeating order. Concretely, we perform a single SGD step for each task in sequence before repeating the same order. We set U D M which implies U/T training steps per task.
Alternating
Uj
Uniform task sampling This is a stochastic version of the schedule above, where the task for each SGD step is sampled from a uniform distribution, with probability 1/T. We implement it such that the number
6

---

Published in Transactions on Machine Learning Research (12/2022)
Task #1
Task-by-task
Task #2
Alternating
Uniform task sampling
Task #3
Weighted task
sampling
Accumulating
gradients
Figure 2: Task sampling schedules. Each element within task corresponds to the number of training steps performed for that task by the baseline model.
of training steps for task j is exactly U/T, by randomly permuting array with U elements, where U/T elements correspond to each task.
Weighted task sampling In this stochastic schedule, we sample each task with weight proportional to the mumber of training steps in the single-task baseline. Therefore, U = Ej= Uy and the sampling weight for task j is U,/U. We implement this schedule as above, to ensure that we perform exactly U, steps for task j.
Accumulating gradients For T tasks, perform forward and backward pass minibatch for cach task, summing the gradients over each task. We then perform a single parameter update with the average of the accumulated gradients, thus effectively using a larger batch size encompassing all the tasks being co-trained. Here, we set U= (
4.3 Initialization of PolyViT
As described in Sec. 3.1 ViT, VIVIT and AST models are initialized from models pretrained on ImageNet- 21K or JFT before being finctuned for the task of interest. In our experiments, we also finetune from ViT model pretrained on ImageNet-21K, and follow the initialization methods for the positional embeddings, p. and input embeddings, E, for each modality as described in Dosovitskiy et al. (2021) and Arnab et al. h2021).
When we use modality-adaptor layers, that is Ladapt > 0, the first. Ladapt layers for each modality are initialized with the same first Ladapt layers from the pretrained VIT model. These parameters are however allowed to change from each other during training. Similarly, shared PolyViT layers are initialized from the last Lshared transformer encoder layers from the pretrained ViT model. Note that the output linear heads are all initialized randomly.

---

Published in Transactions on Machine Learning Research (12/2022)
The effect of the task sampling schedule on co-training performance on multiple modalities and tasks. The highest accuracy is shown in bold, the second-highest is underlined. The "Weighted" task sampling method consistently achieves the highest accuracy for 8 out of 9 tasks, and second-highest on the (highest).
Table 1:
remainder. Results are on the validation set. Color bar: (lowest)
Image
Video
Audio
Schedule
Im1K
C100
C10
Pets
R45
K400
MiT
MiniAS
VGG
Task-by-lask
0.3
0.8
11.7
1.9
2.0
0.3
0.3
1.6
37.2
Accumulated
88.1
90.0
98.8
94.0
96.1
58.0
22.5
22.9
27.3
Alternating
86.0
89.4
99.2
94.0
95.8
69.7
30.0
31.4
44.6
Uniform
85.8
89.3
98.6
94.6
96.1
68.8
29.3
30.6
44.1
Weighted
86.9
90.4
99.3
96.5
97.0
71.6
32.5
33.5
49.2
5 Experiments
5.1 Experimental Setup
We train PilyViT simultancously 9 diverse classification tasks spanning the image, video, and audio modalities. Note that datasets and tasks have one-to-one correspondence. We chose this setup of 9 tasks, the datasets include a large variation in domain and training set sizes. Furthermore, the single- task baseline training hyperparameters vary substantially between the tasks. Consequently, we believe this presents a challenging co-training setup.
When co-training for image classification, we use ImageNet-1K, CIFAR-10 and -100, Oxford-IIIT Pets, and RESISC45. For video, we use Kinetics 400 and Moments in Time, and for audio, AudioSet and VGGSound. Exhaustive details of these datasets are in Appendix A As in Nagrani et al. (2021), evaluate the whole AudioSet validation set, but use smaller balanced subset referred to Mini-AudioSet (MiniAS) for initial experiments. We then use a larger, balanced subset of 500 000 examples (referred to AS-500k) for our state-of-the-arl comparisons following Nagrani et al. 2021)) We follow standard evaluation protocols for each task, reporting classification accuracy (%) for all tasks except AudioSet, where we report mean average precision (mAP) as it multilabel datasct.
We set the training hyperparameters for these tasks (and those of the single-task baselines) using the values reported by Dosovitskiy et al. (2021 for image tasks, Arnab et al. (2021) for video tasks and Nagrani et al. (2021) for audio tasks (detailed in Appendix A). Note that the "audio-only" model of Nagrani et al. (2021). which we use as our baseline, is identical to AST Gong et al. (2021), and we choose it since the authors have evaluated on more datasets.
We perform experiments with two standard transformer encoder configurations: Base (number of layers, L 12, hidden dimension d 768, attention heads h 12) and Large (L 24, 1024, h 16) following Devlin et al.] (2019) Dosovitskiy al. (2021). As in Dosovitskiy VolyVIT model and basclines with ViT pretrained ImageNet-21K. We refer to this initialized model VrT-Im21K. ViT is implemented in Scenic (Dehghani et al. 2021b) and the code for training and evaluation of the model is available at
al. (2021), we initialize
5.2 Selecting the best task sampling schedule for co-training
We begin by analyzing the effect of the different task sampling schedules listed in Sec. 4.2 full, aforementioned 9-task set-up with Poly ViT-Base and all encoder layers shared (Lshared L Leadapt = 0).
We use the
12,
As shown Tab. 1 the "Task- schedule performs poorly, and only achieves decent performance on one task, it suffers from catastrophic forgetting (French) 1999). The "Accumulatod" sampling strategy requires using single learning rate for all tasks (since the accumulated gradient over all tasks is used for
8

---

Published in Transactions on Machine Learning Research (12/2022)
Table 2: Co-training with PolyViT-Base. As indicated by the "#Models" column, some rows correspond multiple trained models. In this case, report the total number of parameters across all models. Poly VIT co-trained a single-modality outperforms single-task baselines in most cases, whereas PiTyViT co-trained
on multiple modalities achieves competitive performance with a large reduction in parameters (test set results). Color bar: (lowest)
(highest). Further details in Appendix A
Image
Video
Andio
Model
#Modlls
#Params
Im1K
C100
C10
Pets
R45
K400
MIT
MiniAS
VGG
ViT-Im2IK Linear probe
93M
80.7
76.2
91.7
91.8
81.7
64.0
25.5
11.3
15.7
Single-task baseline
773M
83.1
92.0
99.0
94.5
96.7
78.7
33.8
29.3
51.7
VITy 1 modality
3
263M
84.3
93.3
99.1
95.1
96.4
80.2
36.5
36.7
51.6
PolyVIT, Endupt = 0
93M
83.1
91.2
99.0
95.0
96.7
77.5
33.2
32.3
50.6
PolyViI, Ludupt L/2
178M
82.8
91.5
99.0
95.0
96.6
79.4
35.3
33.1
51.5
performing a parameter update). As used learning rate of 0.03, which is the learning rate used by the image tasks, and significantly lower than the learning rates for the video and audio tasks of the basclines (details in Appendix A), this method only performs well on image datasets. The " "Alternating", "Uniform" and "Weighted" strategies perform the best, showing that task-specific learning rates, and switching between gradient-updates for different tasks is crucial for accuracy.
In particular, the "Weighted" sampling method performs the best, achieving the highest accuracies on of the 9 tasks (and second-highest. the remainder), motivating us to use it for all subsequent experiments. We postulate that this is the case, because unlike the "Uniform" and " "Alternating" strategies, the "Weighted" strategy performs a different number of update steps for each task. For example, small datasets with a lower number of training steps in their baseline training configurations are sampled less frequently, which can help to mitigate overfitting. An example is the Pets task, that is only sampled for 500 iterations, out of the 417 000 total steps, or just 0.11% of the SGD updates. Another advantage of the "Weighted" strategy is that it performs the same number of steps per task as single-task baseline. Therefore, uses the same computational resources during training as 9 separate, single-task baselines. Our experiment also shows that if we do not have training hyperparameters for new task, we can simply tune them separately in the single- task set ling, and then reuse them for co-training. This approach requires significantly less computation than tuning training hyperparameters directly in the co-training setup.
5.3 Co-training with PolyViT
Table 2 presents approaches for training models to solve 9 different tasks across the image, vidco and audio modalities. We consider two variants of ViTy The first PiT for a single modality, where we co-train three separate Vil models on all the tasks from either the image, video or audio modalities. The second is the multi-modal PoT scenario where we co-train on all nine tasks across three modalities. Here, set Ladapt to and L/2 respectively to understand the effect of the number of modality-adaptor and shared layers.
Baselines. We compare ViT to the following two baselines, which illustrate two alternatives to co- training.
Single-task baseline The first bascline is to train separate singlo-task models for each datasct, either VIT, ViVIT or AST depending on the modality. This results in accuracies comparable to the stale-of-the-art on the respective datasets, but also the largest number of total parameters.
ViT-Im21K Linear probe baseline The second bascline is to use VIT model initialized on ImageNet-21K (ViT-Im21K) and to "freeze" the encoder of the network and train only the linear output heads (Eq. 5 for each task. Positional embeddings, p, and input embeddings, E are initialized following the methods used by VIT, ViVIT or AST as described in Soc. 3.I] This bascline has the same number of parameters as Poly ViT with Ludapt O. We choose a model pretrained on ImageNet-22 (Im21K) as this the largest, publicly available image dataset that aware of.
9

---

Published in Transactions on Machine Learning Research (12/2022)
Table 3: Linear probing of PolyViT and single-task baselines. Similar to the protocol for evaluating self- supervised representation learning, train only linear classifier on top of "frozen" transformer encoder. Note how ViT co-trained on all tasks transfers well to all other datasets and modalities. Models trained on audio do not transfer well to images and video, and vice versa. All models are pretrained on ImageNet- 21K, and then optionally finetuned on downstream datasets. Color bar: (lowest)
(highest).
Image
Video
Audio
C-ch101
SUN397
Dmlab
PCAM
Epic K.
5.5 v2
DTD
16000
MiT
154000
Model
Finetuning
WmT-K22KK pretrained
88.9
75.7
41.0
72.1
46.9
80.2
10.0
17.8
66.6
4.9
10.8
VIT
ImageNet-1K IK
91.0
79.3
45.6
71.9
52.5
80.7
12.2
18.5
67.9
5.3
12.0
PolyVIT
Image tasks
90.7
80.0
45.2
72.5
53.8
81.2
12.1
17.9
67.9
5.3
11.9
VIVIT
MiT
85.2
73.8
43.0
69.9
5.4.9
81.7
14.9
26.3
74.2
5.1
11.9
PIlyVIT
Video tasks
89.2
77.5
53.5
83.8
27.9
79.7
5.3
12.2
AST PolyVIT
VGGSound
29.0
7.6
29.8
34.7
45.1
79.5
2.9
4.6
10.6
9.7
21.7
Andio tasks
38.8
14.7
31.4
40.1
43.2
78.4
3.0
5.8
14.5
10.3
22.0
PolyVIT Ludopt =0
All All
91.0
78.2
45.8
71.8
52.3
81.9
16.8
27.9
77.8
9.6
20.6
PolyVIT Ludopt L,2
90.7
77.8
52.5
82.3
79.4
9.9
21.1
Discussion. Table 2 shows that PolyViT trained on single modality achieves the highest performance on 7 of the 9 datasets. On the remaining two, the accuracy difference is negligible, as it is at most 0.3%. Moreover, the total number of parameters is 3 times less than the single-task baselines. Single-modality co-training improves accuracy the most on the smaller datasets within the modality (Kinetics 400 in video, Mini-Audiosi AudioSet for audio, and CIFAR-100 for images; full dataset details in Appendix A). This suggests that co-training acts as a regularizer, as noted by Caruana (1997), that facilitates learning on smaller datasets where high-capacity models would otherwise overfit.
Multi-modal Vil (final two rows) achieves competitive performance whilst using substantially fewer parameters. In particular, PolyViT with all transformer encoder layers shared between modalities (Ludapt = 0) is within 1.2% of the single-task baselines across all datasets whilst using 8.3 times fewer parameters. This model also comprelensively outperforms the ViT-Im21K Linear probe baseline which has the same number of parameters. Sharing half the transformer layers between modalities (Iandapt = L/22) increases model capacity, and the model improves upon the corresponding single-task baseline on 4 datasets, whilst being at most 0.5% worse on the others. The total number of paramcters is still reduced by factor of 4.3 compared to the single-lask baselines.
Our results are consistent when using the Large model backbone as shown in Appendix
5.4 Evaluating learned representations with linear probes
We now evaluate the feature representations learned by PolyViT by simply appending and training only a new lincar head (Eq. 4 for new task. This evaluation therefore follows the experimental sctting commonly used in self-supervised learning to evaluate the quality of learned representations (Chen et. al. 2020 Grill et al. 2020). Note that the new task come from any one of the three modalities that Pily is trained on, since the modality-adaptor layers (if present) are modality-specific rather than task-spocific.
In particular, we evaluate on a number of new image, audio and video datasets as detailed in Appendix D For image classification, we include Caltech101, SUN397, DmLab, DTD, Kitti Distance and PatchCamelyon, which are datascts from the Visual Task Adaptation Benchmark (Zhai al. 2019 not in our co-training set., SerehiiinnSmmmtingg and Kinetics 600. Finally,
For video classification, we also include Epic Kitchens, for audio classification, we the audio versions of Moments in Time and Kinetics 400.
10

---

Published in Transactions on Machine Learning Research (12/2022)
Table 4: Comparison to MBT Nagrani et al. 22021) the current published state-of-the-art using the same protocols. The second and third rows show that MBT, when first trained on AudioSct and then finctuned VGGSound, and vice-versa, does not perform as well PolyViT, showing that the regularizing benefits of co-training are not simply because the co-trained model has access to more data.
VGGSound
AudioSet mAP
Model
# Moddls
# #Params
Top 1
Top 5
MBT (audio-only)
2
172M
52.3
78.1
44.3
MBT: AS500k + VGGSound
1
87M
54.4
81.4
34.2
MBT: VGGSound + AS500k
1
87M
22.1
43.5
44.4
ViT
87M
55.1
80.4
44.5
Models and basclines. We use PolyViT-Base and take linear probes of all the PolyViT models from Sec. 5.3 i.c. three single-modality models and two multi-modal models trained on all tasks with Ludapt = 0 and Lndapt = L/2 respectively. Our baseline models are those not performing co-training. Namely, we use VIT trained only on ImageNet-21K (ViT-Im21K) as a baseline, followed by ViT, ViVIT and AST initialized from ViT-Im21K and finctuned on ImageNet, Moments in Time and VGGSomn respectively (since these are the largest datasets for each respective modality).
Discussion. Table 3 shows how Poly trained multiple modalities learns cross-modal feature repre- sentations that perform well all 11 linear evaluation tasks across three different modalities (last two rows). This holds even when all the layers of the Poly ViT transformer layer are shared, and thus the total number of parameters is roughly equal to single-task model. PilyViT where the first half of the transformer encoder layers are modality-specific (final row), has more parameters and in general performs better. Furthermore, for the Epic Kitchens (video), Something SovethinnSSomehhng v2 (video) and Caltech 101 (image) datasets, multi- modal Poly VIT transfers better than single-modality baselines. Table 3 thus demonstrates how co-training multiple modalities facilitates learning powerful, transferable feature representations that can be used multiple downstream lasks.
Models trained on only a single modality, as expected, do not in general learn feature representations that transfer well to other modalitics. In particular, models trained on audio tasks do not transfer at all to images and videos, and vice versa. Models trained on video, however, still perform well on images, with rideo-trained models performing the best on the DmLab, PCAM and KITTI-Distance datasets. We believe this is due to the commonalities between the image and video modalities. Observe that in the majority of cases, single-modality Vil models perform better on linear probing than the corresponding single-task baselines, especially for video and audio.
5.5 Improving accuracy with single-modality co-training
Motivated by the performance of single-modality co-training in Tab. 2 we perform larger-scale co-training experiments with this method on audio and video classification. Tables and 5 show that we achieve significant improvements in both these domains whilst also using substantially fewer parameteters.
Audio classification. We compare to the current statc-of-tho-art using audio information only, MBT (Na- grani et. al. 2021), using the same Base backbone, training on the balanced AS-500k subset, and other experimental settings as the authors (Nagrani et al. 2021). As shown in Tab. 4 surpass the state-of-the. art on both datasets (AudioSet and VGGSound), whilst using about half the total parameters. We observe larger improvements (2.8) on VGGSound, the smaller dataset. This is line with our findings from Sec. 5.3 and shows that co-training has a regularizing effect. that reduces overfitting and improves performance the most on smaller datasets.
The second and third rows of Tab. 4 also show that training MBT on AudioSet and then finetuning on VGGSound, vice versa, produces worse results than our co-training method. This shows that the regu-
11

---

Published in Transactions on Machine Learning Research (12/2022)
Table 5: Comparison to ViVIT (Arnab et al. 2021) using the same experimental settings as Arnab et al.
(2021p. K400 and K600 denote Kinctics-400 and -600 respectively. MiT denotes the Moments in Time dataset.
K400
K600
MiT
Model
#Models
#Params
Top
Top 5
Top
Top
Top
Top 5
VIVIT
3
913M
80.6
94.7
82.5
95.6
38.0
64.9
VIT
308M
82.4
95.0
82.9
95.5
38.6
65.5
larization benefits of co-training not solely from having access to more data than single-task baselines. As expected, finetuning MBT on the target dataset causes accuracy to degrade on the original dataset.
Video classification. We co-train Poly ViTT-agee with a smaller tubelet size (and hence greater number of tokens) of x 16 16 on Kinetics-400, -600 and Moments Time. We compare to ViVIT (Arnab et al. 2021 which a leading approach on this task, and directly comparable our work as it uses the same architecture, initialization and number of tokens. As shown in Tab. we surpass VIVIT on all three datasets. Once again, the largest improvement of 1.8% is Kinetics 400, which is also the smallest datasct, co-training has regularizing effect. Moreover, by co-training on three datasets, reduce the total number of parameters required by almost three compared to separately trained ViViT models. Appendix compares our models to other previous works on these audio and video datasets.
6 Conclusion and Future Work
By co-training Poly VIT on a single modality, we have achieved signficant accuracy improvements on three video and two audio datasets, while reducing the total number of parameters linearly compared to single- task models. PolyVill co-trained on multiple modalities is even more parameter-efiiee -fficteeeff still competitive with the srato-of-the-art, and learns feature representations that generalize across multiple modalitics. This enables us learn new (asks by simply learning an additional output head. Co-training is simple and do not need to tune hyperparameters practical, hyperparamcters from singlo-task models. Moreover, we can achieve accuracy improvements re-use training from training for the same number of total steps. Currently we do not co-train on large-scale upstream datasets such as ImageNet-21K (Deng et al. 2009) and C4 (Raffel et al. 2019), and we do not handle the text modality either. We aim to explore this, and co-training with the text modality, in future work. An initial exploration of this idea in Appendix F] Besides, we have only considered classification tasks, and not more complex tasks like object detection captioning, and aim to explore this direction, e.g. bringing OWL-VIT [Minderer et al. 2022 to the ensemble of PoyViT, in future work.
the joint space of all datasets, but can simply
Broader Impact
Our work presents method for performing image- audio- and video-classification with a single parameter- efficient model. Classification of perceptual data (images, audio and video) is general technology with unaware of all potential applications, it is important to
wide range of potential applications. While be aware that cach application has its own merits and socictal implications depending on the intentions of the individuals building and using the system. We also note that training datasets contain biases that may render models trained on them unsuitable for certain applications. It possible that people use classification models (intentionally or not) to make decisions that impact different groups society differently.
12

---

Max Bain, Arsha Nagrani, Güi Varol, and Andrew Zisserman. Frozen in time: A joint video and image Published in Transactions on Machine Learning Research (12/2022)
References
Samira Abnar, Mostafa Dehghani, Behnam Neyshabur, and Hanie Sodghi. Exploring the limits of large scale pre-training arXiv preprint arXi::2100. 02095, 2021.
Hassan Akbari, Linagzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong, Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. In ICCV, 2021.
Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelovié, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman. Self-supervised multimodal versatile networks. In NewrIPS, 2020.
Anurag Arnab, Mostafa Dehghani, Georg Heigold, Chen Sun, Mario Lucic, and Cordelia Schmid. Vivit: A video vision transformer. In ICCV, 2021.
Anurag Arnab, Xuchan Xiong, Alexcy Gritsenko, Rob Romijnders, Josip Djolonga, Mostafa Dehghani, Chen Sun, Mario Lučié, and Cordelia Schmid. Beyond transfer learning: Co-finetuning for action localisation. arXiv preprint arXic:2207.03877 2022.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. arXiv:1607.000 06450, 2016.
Layer normalization. arXiv preprint
encoder for end-to-end retrieval. In ICCV, 2021.
Tom B Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In arXiv preprint Xrx:::0005. 14165, 2020.
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
Rich Caruana. Multitask learning. Machine learning, 1997.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for rastivvv learning of visual representations. In ICML, 2020.
Zhao Chen, Vijay Badrinarayanan, Chen- Yu Lee, and Andrew Rabinovich. Gradnorm: Gradient normaliza- tion for adaptive loss balancing in deep multitask networks. In ICML, 2018.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In ICML, 2008
Mostafa Dehghani, Anurag Arnab, Lucas Beyer, Ashish Vaswani, and Yi Tay. The efficiency misnomer. arXiv preprint xrXi::2110.1288) 2021a.
Mostafa Dehghani, Alexey Gritsenko, Anurag Arnab, Matthias Minderer, and Yi Tay. Scenic: A JAX library for computer vision rescarch and boyond. arXiv preprint 0rXiv:2100111400 2021b.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirec-
tional transformers for language understanding. In NAACL, 2019.
Alexcy Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x 16 words: Transformers for image recognition at scale. In ICLR, 2021.
13

---

Published in Transactions on Machine Learning Research (12/2022)
David Eigen and Rob Fergus. Predicting depth, surface normals and semantic labels with common multi- scale convolutional architecture. In ICCV, 2015.
Robert M French. Catastrophic forgetting in connectionist. networks. Trends in cognilive sciences, 1999.
Ross Girshick, Jeff Donahue, Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR, 2014.
Yuan Gong, Yu-An Chung, and James Glass. 2021.
AST: Audio Spectrogram Transformer. In Proc. Interspeech,
Sean-Bastion Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent: A new approach to self-supervised learning. In NeurIPS, 2020.
Michelle Guo, Albert Haque, De-An Huang, Serena Yeung, and Li Fei-Fei. Dynamic task prioritization for multitask learning. In ECCV, 2018.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In ICCV, 2017.
Dan Hendrycks and Kevin Gimpel. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. arXiv preprint arXic:1606.cm 08415, 2016.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficien transfer learning for nlp. In JCML, 2019.
Ronghang Hu and Amanpreet Singh. Unit: Multimodal multitask learning with a unified transformer. ICCV, 2021.
Gao Huang, Yu Sun, Zhuang Liu, Daniel Sedra, and Kilian Q. Weinberger. Deep networks with stochastic depth. In ECCV, 2016.
Forrest N Tandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer. Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size. In ICLR, 2017.
Andrew Jaegle, Felix Gimeno, Andrew Brock, Andrew Zisserman, Oriol Vinyals, and Joao Carreira. Per- ceiver: General perception with iterative allention. In Xive preprint arXiv:2103.083 03.03206. 2021.
Andrew Jaegle, Sebastian Borgeaud,
Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, et al. Perceiver io: A general architecture for structured inputs & outputs. ICLR, 2022.
Lukasz Kaisor, Aidan N Gomez, Noam Shazoer, Ashish Vaswani, Niki Parmar, Llion Jones, and Jakob Uszkoreit. One model to learn them all. In arXiv preprinl ar Xive: (tl:1706.555 706.05137. 2017.
Will Kay, João Carreira, Karen Simonyyn, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasinhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset. arXio preprint aix: 105::10050000000 06950, 2017.
Alex Kendall, Yarin Gal, and Roberto Cipolla. Multi-task learning using uncertainty to weigh losses for scene geometry and semantics. In CVPR, 2018.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- nanch Hajishirzi. Unifiedqa: Crossing format boundaries with a single qa system. arXiv preprint urXiv:2005.00700 2020.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska Barwinska, al. Overcoming catastrophic forgetting in neural networks. Proceedings of the national academy of sciences. 114(13):3521 3526, 2017.
14

---

Published in Transactions on Machine Learning Research (12/2022)
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. In NewrIPS, 2017.
lasonas Kokkinos. Ubernet:
Training a universal convolutional neural network for low-, mid-, and high-level vision using diverse datasets and limited memory. In CVPR, 2017.
Sangho Lee, Youngjae Yu, Gunhee Kim, Thomas Breuel, Jan Kautz, and Yale Song. Parameter efficient multimodal transformers for video representation learning. In ICLR, 2021.
Gen Li, Nan Duan, Yuejian Fang, Ming Gong, and Daxin Jiang, Unicoder-vl: A universal encoder for vision and language by cross-modal pre-training. In AAAI, 2020.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-language tasks. In NeurIPS, 2019.
Jiasen Lu, Vedanuj Goswami, Marcus Rohrbach, Devi Parikh, and Stefan Lee. 12-in-1: Multi-task vision and language representation learning. In CVPR, 2020.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In arXiv preprint arXiv:2206.0890 2022.
Kevis- Kokiisi Maninis, Ilija Radosavovic, and lasonas Kokkinos. Attentive single-tasking of multiple tasks.
In CVPR, 2019.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering. arXiv preprint (1ff::88060870000 2018.
Antoine Miech, Jean-Baptiste Alayrac, Lucas Smaira, Ivan Laptev, Josef Sivic, and Andrew Zisserman. End-to-end learning of visual representations from uncurated instructional videos. In CVPR, 2020.
Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Doso- vitskiy, Aravindh Mahendran, Anurag Arnab, Mostafa Dehghani, Zhuoran Shen, al. Simple open- vocabulary object detection with vision transformers. arXiv preprint arXiv:2205.000 062330 2022.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In CVPR, 2016.
Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. Attention bottlenecks for multimodal fusion. In NeurIPS, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. arXiv preprint 1910::1110.0000 2019.
Alviss Rebuffi, Hakan Bilen, and Andrea Vedaldi. Learning multiple visual domains with residual adapters. In NewrIPS, 2017.
Tara N Sainath, Brian Kingsbury, Vikas Sindhwani, Ebru Arisoy, and Bhuvana Ramabhadran. Low-rank matrix factorization for deep neural network training with high-dimensional output targets. In ICASSP, 2013.
Ozan Sener and Vladlen Koltun. Multi-task learning as multi-objective optimization. In NewrIPS, 2018.
Chen Sun, Abhinav Shrivastava, Saurabh Singh, and Abhinav Gupta. Revisiting unrcasonable effectiveness of data in deep learning era. ICCV, 2017.
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. Hypergrid transformers: Towards single model for multiple tasks. In ICLR, 2020.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Mllia Polosukhin. Attention is all you need. In NewrIPS, 2017.
15

---

Published in Transactions on Machine Learning Research (12/2022)
Zichao Yang, Marcin Moczulski, Misha Denil, Nando De Freitas, Alex Smola, Le Song, and Ziyu Wang. Deep fried convnets. In TCCV, 2015.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning. In NeurIPS, 2020.
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In CVPR, 2018.
Xiaohua Zhai, Joan Puigcerver, Alexander Kolesnikov, Pierre Ruyssen, Carlos Riquelme, Mario Lucic, Josip Djolonga, Andre Susano Pinto, Maxim Neumann, Alexey Dosovitskiy, et al. A large-scale study of repre- sentation learning with the visual lask adaptation benchmark. ar preprint 1rXii:1910.000 2019.
Hongyi Zhang, Moustapha Cissé, Yann N. Dauphin, and David Lopez-Paz. Mixup: Beyond empirical risk minimization. orXiv preprinl ar Xiv: 1710.091000 12, 2017.
Zhanpeng Zhang, Ping Luo, Chen Change Loy, and Xiaoou Tang. Facial landmark detection by deep multi- task learning. In ECCV, 2014.
16

---

Published in Transactions on Machine Learning Research (12/2022)
Appendix
Appendix A contains additional details about our experimental settings, providing more information to Section 5.1 of the main paper. Appendix B provides more details for the experiments in Section 5.2 of the main paper. Appendix C shows further experimental details and results corresponding to Section 5.3 of the main paper. Appendix D
provides additional details about the experiments in Section 5.4 of the paper. Finally. Appendix E provides additional experimental details and results corresponding to Section 5.5 of the main paper.
A Experimental set-up: additional details
Task details and input dimensions. See Tables and For each task, the number of linear warmup steps is set as reported in (Dosovisskiy et al.) 2021 Arnab et al.) 2021 Nagrani et al., 2021). When co- training, we simply use the sum of all warmup steps for each co-trained lask. We use a single momentum state when co-training, i.c. we don't maintain separate momentum states for cach task or modality. Similar to Dosovitskiy et al. (2021), we select the best learning rate on set {0.03, 0.1, using the validation score. For video and audio datasets,
reuse learning rates reported in Arnab et al. (2021) and Nagrani ct al. 2021 respectively. As in (Arnab et al. 2021 Nagrani et al. 2021), we use zero initialization for output head kernels Wout. For image datascts, on a single-task evaluation, we find that the LeCun normal W initializer Klambauee et al. (2077) works best. For all tasks, we perfform gradient clipping with a maximum gradient norm of 1, as this was also used by (Dosovitskiy et al. 2021 Arnab et al. 2021 Nagrani et al. 2021
For the "ViT-Im21K linear probe" baseline, we the same training procedure for single-task baselines, with the difference that 1) only the head parameters updated and 2) image tasks, run separate learning rate grid searches on the set {0.03, 0.1, 0.3}.
Train, validation and test splits. Similarly to Dosovitskiy et al. (2021). take 2% of CIFAR 10/100 train sets for validation, 10% of Pets train set for validation and 1% of ImageNet-Il train set for validation. We use standard test sets for these datasets. For RESTSC45, we use 20% of the train set for validation and 20% for testing. We use standard train, validation and test sets for video and audio tasks.
Augmentation and regularization. We don't 't use augmentation for image tasks. We do video and andio preprocessing and augmentation as done in Arnab et al. [2021]; Nagrani et al. 2021 respectively. For audio tasks, in (Nagrani et al. 2021), we use Mixup (Zhang et al. 2017) with a 0.3 and stochastic depth regularization (Huang et al. 2016) with p = 0.3. Stochastic depth is applied along both audio adaptor and shared layers.
B Selecting the best task sampling schedule: additional experimental details
For the accumulating schedule, we set learning rale to the smallest value across tasks (0.03). We draw a random task order for the Task-by-task schedule, which is as follows: C100 - MiT K400 - MiniAS VGG + Pets + C10 + ImlK R45.
C Co-training with PolyVIT: additional experimental details and results
Evaluation on video and audio tasks. To get test performance on video and audio tasks, we perform multiple-crop evaluation as described in Arnab et al. (202I): Nagrani et al. (2021) for videos and audio respectively.
Results for the Large configuration. See Table Since Nagrani et al. (2021 don't report results on Large configuration, for audio tasks we do an additional hyperparameter tuning for singlo-task basclines validation sets. As a result, Mixup a 0.7 for MiniAS and VGGSound respectively. Also, we
17

---

Published in Transactions on Machine Learning Research (12/2022)
Table 6: Experimental set-up: tasks and their properties. For image tasks, the indicated learning rates are obtained by grid search over {0.03, 0.1, 0.3) single-task baselines using the validation set accuracy. These values are used for single-takk baselines and for PilyViT variants.
Abbre-
Moda-
Clas-
Train
Train steps
Learning rate
Warmup steps
W out
Dataset
viation
lity
ses
size
init
LeCun
CIFAR 100
C100
Image
100
50.0K
10K
0.03
500
normal
LeCun
CIFAR 10
C10
Image
10
50.0K
10K
0.08
500
normal
Oxford-IIIT
LeCun
Pets
Image
37
3.68K
500
0.08
100
Pets
normal
LeCun
RESISC45
R45
Image
45
31.5K
2.5K
200
normal
LeCun
ImageNet-1k
lm1K
Image
1000
1.28M
20K
0.08
500
normal
100.7K
Kinetics 400
K400
Video
400
215K
0.1
2.5 epochs
Zeros
(30 epochs)
Moments in Time
123.6K 6K (10 epochs)
MrT
Video
339
791K
0.25
2.5 epochs
Zeros
15.9K (50 epochs) 135K
Mini-Audioset
MiniAS
Audio
327
20.1K
0.5
2.5 epochs
Zeros
VGGSound
VGG
Audio
309
172K
0.5
2.5 epochs
Zeros
(50 epochs)
Table Input dimensions for different modalities. Sequence length is computed as [+[[T/)))[(/)))>(V//// (one class token and patch tokens). Note that for shared transformer layers, reuse the same parameters for sequences of different lengths.
Input size, [ W
Patch size, [tx]h
Sequence length
Modality
Batch size
Image (pretraining)
224 x 224
16 x 16
197
4096
Image
384 x 384
16 x 16
577
512
Video
32 224 x 224
16 x 16
1569
Audio (spectrogram)
800 128
16 16
401
64
use 30 epochs for MiniAS instead of 50 for the Basc model. In addition, we run scparate lcarning rate grid searches for all image lasks, separately for single-task baselines and ViT-Im21K linear probes. We apply all mentioned hyperparameter changes, obtained for the single-task baselines, to all PolyViT runs. In all other aspects, Large set-up the same as Basc.
D Linear probes: additional experimental details
Task details. See Table g For linear probes, we use the same input dimensions as reported in Table 7 For image tasks, we reuse the number train and warmup steps from the RESISC45 task (Table 6 For video and audio tasks, we used hyperparameters reported in (Arnab et al.] 2021) and (Nagrani al.) 2021 respectively, with the difference that only optimize outpul head parameters during training. As for the co-training setup,
multiple-crop evaluation video and audio tasks.
Train, validation and test splits. For image tasks, we use 2% of the train set as validation set and standard test sets. We use standard train, validation and test sets for video and audio tasks.
18

---

Published in Transactions on Machine Learning Research (12/2022)
Co-training with PilyViT, Large model configuration. Test accuracy (%) and mAP (for MiniAS, %) are reported. As indicated by the "# models" column, some rows correspond multiple models, then the total number of parameters is computed across all models. Color bar: (lowest)
Table 8:
(highest)
Image
Video
Audio
Model
#Models # Params
C100
C10
Pets
R45
ImlK
K400
MIT
MiniAS
VGG
ViT-Im21l Linear probe
312M
84.4
95.6
91.8
89.2
82.6
67.7
26.8
12.8
19.1
Single-task baseline
9
3033M
93.3
99.2
94.8
97.3
85.1
79.6
37.1
30.0
51.8
ViTyViT, modality
3
917M
93.9
99.4
95.5
96.9
85.1
80.6
38.8
37.9
50.7
VII, Lndape
312M
91.4
99.0
94.7
96.8
82.6
78.9
35.8
33.3
49.9
VilyViT, Ludupt L/2
615M
91.1
99.1
95.0
97.0
82.8
81.0
37.7
34.1
50.4
Converting patch and positional embeddings for cross-modal probes. In order to take lincar probes of image-only models (ViT and Poly VIT trained on images) on audio tasks (and vice versa). we leave patch embeddings as they are and 2D-interpolate positional embeddings to the correct resolution. When taking linear probes of video-only models on image or audio tasks, order to obtain 16 x 16 patch embeddings, we take a sum along the first (frame) axis of 3D video patch emboddings of shape x x 16. In order to adapt positional embeddings, we take a mean value of positional embeddings for each frame, and then -D-interpellate the result to the correct resolution. When taking linear probes of image- audio-only models on vidco tasks, we repeat 2D patch embeddings along the frame axis in order to obtain 3D patch embeddings. We also 2D-interpolate positional embeddings to the frame resolution and repeat them for each frame.
Augmentation and regularization. We don't 't use augmentation for image tasks. We do video and audio preprocessing and augmentation done in (Arab et al. 2021 Nagrani et al. 2021 respectively. As in (Arnab et al. 2021), we use Mixup (Zhang et al. 2017 with 0.3 for the S-S v2 task.
Table 9: Tasks used for linear probes. validation set performance.
Indicated learning rate grid search done for all models using
Dataset (task)
Abbre- viation
Moda- lity
Train steps
Learning rate
Warmup steps
out init
Grid search, (0.03, 0.1, 0.3)
LeCun normal
Caltech101
C-ch101
Image
2.5K
200
Grid search, {0.03, 0.1,0.3) Grid search, {0.03, 0.1, 0.3) Grid search, (0.03, 0.1, Grid search, (0.03, 0.1.0.1.00 0.3) Grid search, (0.03, 0.3)
LeCun normal
SUN397
SUN397
Image
2.5K
200
LeCun
Dmlab
Dmlab
Image
2.5K
200
normal
LeCun
DTD
DTD
Image
2.5K
200
normal
LeCun
KITTI Distance
KITTI
Image
2.5K
200
normal
LeCun
PatchCanelyon
PCAM
Image
2.5K
200
normal
Epic Kitchens
Epic K.
Video
30 epochs
0.5
2.5 epochs
Zeros
Something Something v2
Something-Somethingg
S-S v2
Video
35 epochs
0.4
2.5 epochs
Zeros
Kinetics 600
K600
Video
30 epochs
0.1
2.5 epochs
Zeros
Moments in Time (audio)
MrT-A
Audio
10 epochs
0.5
2.5 epochs
Zeros
Kineties 400 (audio)
K400-A
Audio
epocls
2.5 epochs
Zeros
19

---

Published in Transactions on Machine Learning Research (12/2022)
Table 10: Extended comparison to ViVIT (Arnab et al. 2021 The second row shows a ViVIT model, initialized from ImageNet-21K, and then finetuned on Moments Time, Kinctics 600 and then Kinctics 400. This model has seen the same amount of training data as PolyViTI yet does not perform well as PolyViT, showing that the improvements from co-training not solely because PolyViT has access to more training data.
K400
K600
MIT
Model
# Models
# Params
Top 1
Top
Top
1
Top 5
Top
Top 5
ViViT
3
913M
80.6
94.7
82.5
95.6
38.0
64.9
VIVTT: MFT -> K600 - K400
308M
81.3
94.5
78.8
94.0
27.3
52.0
VilyVIT
1
308M
82.4
95.0
82.9
95.5
38.6
65.5
E
performance on one modality: additional experimental details
and results
Additional results on video datasets Table 10 contains an extended comparison to VIVIT (Arnab et al., 2021). compared Table 5 of the main paper. The second row ("VIVIT: MiT - K600 - K400") shows results when we train VIVIT model, by first finetuning
ImageNet-21K initialized model on Moments in Time, then Kinetics 600, and then finally Kinetics 400. This model has seen the same amount of training data as Poly but performs worse on Kinetics 400 than ViT (Poly achieves 82.4, and ViVIT achieves 81.3). This ViVIT model, does however, still outperform a ViVIT model finetuned solely on Kinetics 400 from ImageNet-21K initialization (first row). This result, like Table 4 the main paper for audio, shows that the benefits of co-training are not only because the co-trained Poly ViT model has access to more data.
For our additional baseline, ('VIVIT: MIT K600 K400"), retain the output linear head for cach class. Consequently, the accuracy for MIT and Kineties 600 degrades as the model is trained on Kinetics 400. Note that Kinetics 600 is superset of Kinetics 400, which is why the overall accuracy drop on Kinetics 600 is low. Furthermore, note that the goal of this paper is not to consider the "continual learning" (Kirkpatrick eet al., 2017) problem, which aims to train a model on a new dataset, without losing performance on previous datasets the model was trained on.
Detailed experimental settings For the PolyViT experiment on the video modality, we reuse hyper- parameters reported in Arnab et al. (2021) for Kinetics 400/600 and Moments in Time. See Table 12 for the dataset details and exact hyperparameters used during the experiment. These hyperparameters coincide with those reported in Table 0 for Kinetics 400 and Moments in Time and in Table g for Kinetics 600. The only difference is that we use more granular 3D patch size (2 x 16 x 16) and Large model configuration.
For the audio experiment, similarly, we reuse all hyperparameters reported in Nagrani et al. 20211) for AS-500k and VGGSound experiments (audio-only). See Table 13 for the dataset details and exact hyper- parameters used for the experiment. These hyperparameters almost coincide with those reported in Table with change MiniAS - AS-500k. The only exception is that we use 30 epochs and Mixup 0.5 for AS-500k.
F Upstream co-training of image and text
This section presents initial experiment of "upstream" pre-training of image classification JFT (Sun et al. 2017) and BERR-pretraining on Wikibooks (Devlin et al. 2019). Here, we train PolyViT-Base model from random initialisation. In contrast to the experiments of the main paper which finetuned ImageNet-211 pretrained checkpoints, we cannot use pretrained models here due to the vast domain differences bet ween the image and text modalities.
For JFT classification and BERT pretraining, we follow the experimental hyperparameters and protocols of the original authors (Dosovitskiy al. 2021 Devlin et al. 2019
20

---

Published in Transactions on Machine Learning Research (12/2022)
"Upstream finetuning" on JFT image classfication and BERT text-pretraining. Note that models are trained from random initialisation.
Table 11:
Image (JFT)
Text (Wikibooks)
Accuracy
MLM Accuracy
Next Sentence Prediction
ViT-Base
48.0
BERT- PolyViT-Base 50.0
72.8
99.0
70.9
97.9
As shown in Table 11 compared to single-task baseline, VilyVIT improves slightly on JFT classification, but degrades marginally on text classification metrics.
Note that in this experiment, we have not evaluated the representations learned by the network, for example, by finetuning on dowstrean lasks. We leave this, and further effors on "upstream co-training" for future work.
Table 12:
Set-up for the co-training on videos. Train steps and warmup steps are summed get the number of train and warmup steps during co-training as we use the "Weighted" task sampling method.
Moda-
Clas-
Train
Train
Batch
Learning
Warmup
Wout
Dataset
lity
size
steps
size
rate
steps
init
101K
Kinetics 400
Video
400
215K
64
0.1
2.5 epochs
Zeros
(30 epochs) 170K
Kinetics 600
Video
600
363K
64
0.1
2.5 epochs
Zeros
(30 epochs)
Moments in Time
123.6K
Video
339
791K
64
0.25
2.5 epochs
Zeros
(10 epochs)
Table 13: Set-up for the co-training on audio. Train steps and warmup steps are summed to get the number of train and warmup steps during co-training as we use the "Weighted" task sampling method.
Moda-
Clas-
Train
Train steps
Batch
Learning rate
Warmup
Went
Dataset
Mixup
lity
size
size
steps
init
239K
AS-500k
Audio
527
509K
0.5
64
0.5
2.5 epochs
Zeros
(30 epochs)
135K
VGGSound
Audio
309
172K
0.3
64
0.5
2.5 epochs
Zeros
(50 epochs)
21


Reviews:
[{"review_id": "BWSJb9jD8k", "reviewer": "Reviewer_nHdK", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**:\n1. The approach is very simple, i.e. specific tokenizers for each modality and a specific head for each task. It is easy to implement and can constitute a solid baseline for future work.\n2. The results show that the model is very effective: Table 2 shows how PolyViT achieves comparable performance to single-task baselines while using much fewer parameters. These results are confirmed in the linear probing case (Table 3), showing the generalization capabilities of the features learned by the model.\n3. Table 1 provides a thorough overview of the different training choices and how they impact the final results. These clearly show that taking into account the size of each dataset and training the model for all tasks at once is fundamental for achieving good results.\n4. The writing is clear and easy to follow.\n\n\n**Weaknesses**:\n1. The model is not compared with any baseline on multi-domain/task learning. Arguably, the main competitors is MultiModel (Kaiser et al. 2017). The introduction states that unlike (Kaiser et al. 2017), PolyViT achieves results comparable to the state of the art. However, the two models are compared on different tasks and modalities (e.g. the former image-text-audio-categorical, the latter image-video-audio), with (Kaiser et al. 2017) focus their analyses on tasks beyond classification (e.g. captioning, translation). To support the claim that PolyViT is better than MultiModel, the two should be compared on the same ground. Note also that the paper should describe the shortcomings of the model (tested only on classification) w.r.t. to MultiModel and other previous works (e.g. Hu and Singh 2021) tested on more complex tasks.\n\n1. Since the model can already be applied to text, showing how PolyViT works on this would have made the article more complete. In case PolyViT struggles with text, the manuscript should report and discuss such findings.\n\n1. From the experiments it is unclear how the performance of PolyViT changes w.r.t. the number of tasks per modality. For instance, if the model is trained on one task per modality, the performance would still benefit from co-training? Analyzing this dependence would add more insights into the effectiveness of the PolyViT framework and possible directions for future works. Note also that Table 3 shows unclear trends in this regard (i.e. audio does not benefit from datasets of other modalities while images and video ones achieve comparable performance).\n\n1. While the article is well-written, the abstract does not describe how PolyViT works (i.e. modality-specific tokenizers, task-specific heads). \n\n1. As a final, minor note: it would have been interesting to compare the approach with other common multi-domain learning strategies (e.g. the trend initiated by Rebuffi et al. 2017) to show the extent to which co-training is sufficient in this multimodal setup. This would not only add baselines to the comparisons but also more insights on whether it is important to refine part of the backbone (beyond the *L_adapt* variant) in this multimodal setup.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The article presents PolyViT, a single transformer model that can address multiple tasks in multiple modalities. The model uses modality-specific tokenizers and task-specific heads. During training, the model sees data of the multiple modalities and tasks, with the manuscript analyzing different training choices (e.g. task order, sampling, etc.). Results show that the approach achieves results comparable to models trained for the specific modalities and tasks. ", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Following on the weaknesses described above, I would recommend the following revisions (ordered per importance):\n1. The manuscript should provide a more thorough comparison with previous works (MultiModel in particular). This holds both for quantitative comparisons (i.e. either using the same setup or adapting the model to the considered benchmarks) as well as discussions on the limitations of PolyViT (i.e. focus on classifications). \n1. Are there reasons why PolyViT cannot be readily applied to text? Even including negative results would be helpful to add more insights to the work. \n1. Analyze how PolyViT performs if trained by varying the number of tasks per modality and on extreme setups (e.g. 1 task per modality).\n1. (minor) The abstract should describe PolyViT.\n1. (minor) Table 4, for VGGSound top-5, the bold is misplaced.", "broader_impact_concerns": "None but the broader impact section of the paper perfectly details potential misuses of the users and biases on the datasets.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "g4TLSWqjYk", "reviewer": "Reviewer_7MGN", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is well written in general. Initial analyses show that the proposed method outperforms a state-of-the-art single-task baseline on 2 of these datasets, whilst achieving competitive performance on the others.\n\nThere are several unclear statements in the paper, some of which are discussed below.\n\nA major weakness of the paper is the lack of detailed analyses. The current paper proposes a black box method to train a multi-modal transformer with parameter sharing, which achieves good accuracy. To improve the paper, detailed analyses of its underlying mechanism and several ablations should be provided. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper studies training a single transformer model to process multiple modalities and datasets, whilst sharing almost all of its learnable parameters. For this purpose, a new transformer model called PolyViT was proposed. \n\nBy co-training on different tasks and modality, PolyViT achieves significant accuracy improvements on several video- and audio-classification datasets. The obtained parameter-efficient model generalizes across multiple domains outperforming a state-of-the-art single-task baseline on 2 of these datasets, whilst achieving competitive performance on the others.\n", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "\nCould you please explain what you mean by \"interpolated from the pretrained model\" in the positional embeddings, p, which depend on the sequence length N (and thus the input resolution), are interpolated from the pretrained model to the finetuned model?\n\nHow do you accumulate gradients in different sampling schedules? Do you just take the average?\n\nTraining phases should be further analyzed. For instance, have you observed any vanishing/exploding gradients, why adaptor layers do not help, why does accuracy drop too much in some cases in Table 3 etc.?\n\nWhat is the array referred in \"by randomly permuting an array with U elements\"?\n\nCould you please further explore the claim \"Models trained on only a single modality, as expected, do not in general learn feature representations that transfer well to other modalities. In particular, models trained on audio tasks do not transfer at all to images and videos, and vice versa\"?\n\nHow do you train modality-specific transformer layers (denoted as modality adaptor layers)? This part was not explained clearly.\n\nHave you used other optimization methods for training? Is there any particular reason for choosing SGD? An ablation of optimizers would be useful.\n\n", "broader_impact_concerns": "Broader impact concerns were briefly discussed.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "9gf33nhmIs", "reviewer": "Reviewer_iqC8", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "<Strengths>\n1. This work shows that a single transformer model trained on multi-modal tasks can outperform single-task SOTA models.\n\n2. It proposes multiple co-training schemes with different task sampling schedules – Task-by-task, accumulated, Alternating, Uniform, and Weighted. \n\n<Weaknesses>\n1. I think the main issue of this work is that it is somewhat outdated by recent progress of unified multimodal transformer models.\n- For example, recent works like Unified-IO (https://unified-io.allenai.org/) by AI2 and Perceiver IO (https://www.deepmind.com/open-source/perceiver-io)  by DeepMind address the training of transformers in much more challenging multimodal settings – more data, more parameters, more modalities and more tasks. \n\n2. The technical novelty is limited. \n- Training a single transformer model with multimodal datasets is not new. \n- Using different configurations of multimodal data in a minibatch is not new. \n- The proposed PolyViT may seem to be a marginal straightforward upgrade of ViT  toward multimodal data with some heuristics. \n\n3. Experiment evaluations are not convincing. \n- This work mainly compares with ViT models as shown in Table 2-3. \n- There is no comparison with SOTA unified multi-modal transformer models.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work trains a single transformer model named PolyViT to perform multi-modal tasks. It proposes and compares various co-training procedures with different task sampling schedules. This work trains multi-modal PolyViT on 9 datasets across three modalities (of image, audio, and video) and outperforms state-of-the-art single-task baselines (MBT and ViVIT) on VGGSOund and Kinetecs-400/600.", "claims_and_evidence_yn": "", "claims_explanation": "", "audience_yn": "", "audience_explanation": "", "requested_changes": "Please see the comments in Weaknesses. \n\nMy general feeling on this work is that it is decent but outdated by recent unified models (Unified IO, Perceived IO, and many more) in various aspects – technical novelty, model and data scales, and experiments. \n", "broader_impact_concerns": "No specific ethical concern in this work. ", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper is reviewed by three experts on this topic. All the reviewers give very good comments and suggestions. The authours provide extensive and insightful discussion, as well as some additional experiments. Most of concerns on the paper are addressed. Although two reviewers have still concern about the novelty of this model and are thus slightly negative to the paper, the AE suggested the acceptance, due to several reasons.
(1) Overall, the paper is well written, and clean. The experiments are well supporting the claims and contributions. 
(2) A single simple and effective model to unify several datasets from different tasks/domains is truly valuable to the community in general.
(3) Some engineering techniques are useful and of good value to the audience.

I would suggest the accept with minor, in order to give the authous the final chance of proof-reading and update the manuscript,  inserting the code links and so on. BTW, is the word 'Poly' short for something, or just using the meaning of the word here?


Decision: Accept (poster)