Title: Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception
Venue: NeurIPS 2023
Accepted: yes
Keywords: Alternating Gradient Descent

Multimodal

Mixture of Experts

AGD

MoE

Deep Learning

Optimization
Abstract: We present Integrated Multimodal Perception (IMP), a simple and scalable multimodal multi-task training and modeling approach. IMP integrates multimodal inputs including image, video, text, and audio into a single Transformer encoder with minimal modality-specific components. IMP makes use of a novel design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts (MoE) for efficient model & task scaling. We conduct extensive empirical studies and reveal the following key insights:
    1) performing gradient descent updates by alternating on diverse modalities, loss functions, and tasks, with varying input resolutions, efficiently improves the model.
    2) sparsification with MoE on a single modality-agnostic encoder substantially improves the performance, outperforming dense models that use modality-specific encoders or additional fusion layers and greatly mitigating the conflicts between modalities. 
IMP achieves competitive performance on a wide range of downstream tasks including video classification, image classification, image-text, and video-text retrieval. Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves new state-of-the-art in zero-shot video classification: 77.0% on Kinetics-400, 76.8% on Kinetics-600, and 68.3% on Kinetics-700, improving the previous state-of-the-art by +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total training computational cost.

Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

Abstract

We present Integrated Multimodal Perception (IMP), a simple and scalable multimodal multi-task training and modeling approach. IMP integrates multimodal inputs including image, video, text, and audio into a single Transformer encoder with minimal modality-specific components. IMP makes use of a novel design that combines Alternating Gradient Descent (AGD) and Mixture-of-Experts (MoE) for efficient model & task scaling. We conduct extensive empirical studies and reveal the following key insights: 1) performing gradient descent updates by alternating on diverse modalities, loss functions, and tasks, with varying input resolutions, efficiently improves the model. 2) sparsification with MoE on a single modalityagnostic encoder substantially improves the performance, outperforming dense models that use modality-specific encoders or additional fusion layers and greatly mitigates the conflicts between modalities. IMP achieves competitive performance on a wide range of downstream tasks including video classification, image classification, image-text, and video-text retrieval. Most notably, we train a sparse IMP-MoE-L focusing on video tasks that achieves new state-of-the-art in zero-shot video classification: 77.0% on Kinetics-400, 76.8% on Kinetics-600, and 68.3% on Kinetics-700, improving the previous state-of-the-art by +5%, +6.7%, and +5.8%, respectively, while using only 15% of their total training computational cost.

1 Introduction

The human perception system is profoundly multimodal. We perceive the world through the integration of a vast array of sensory systems across domains - visual, auditory, olfactory, somatic, etc . Neurons for multimodal integration have been found in both multisensory convergence zones Calvert [2001] and unimodal regions Driver and Noesselt [2008] in the human brain. Studies in developmental psychology also suggest that interrelating simultaneous multimodal sensations is key for perceptutal learning Smith and Gasser [2005]. Inspired by these findings, we see an opportunity for combined multisensory learning in machine learning systems as well.

The rapid rise of large-scale multitask frameworks and models [Raffel et al., 2020, Roberts et al., 2022, Radford et al., 2021, Yu et al., 2022, Wang et al., 2022] provides foundations for integrating capabilities that unify many disparate tasks under one model. However, given the vast quantity of independent variables involved in designing such a system, achieving an integrated multimodal machine learning model still remains an open research direction. More specifically, designing a multi-task model that integrates many multimodal signals is challenging due to various reasons: i. Different modalities require structurally different I/O signatures to properly train. ii. When training across multiple datasets, some modalities or objectives may not exist or cannot be applied, depending

on the input data and the task to perform. iii. The presence of multiple input modalities calls for careful considerations on the architectural design and allocation of parameters to certain modalities, often requiring extensive hyperparameter tuning to find the best use of computational resources.

Intuitively, as we scale a model, it becomes increasingly expensive to redesign the architecture or search for a better training objective. The issue is exacerbated in multimodal multi-task modeling, where we need to consider the combination of input modalities or datasets, loss functions, and tasks at large scales. Therefore, we would like to find a training approach that can be scaled incrementally: for any new task or objective, regardless of its input shape or output loss, we should be able to add it to the existing pretraining without compromising the previous tasks.

We navigate this problem by exploring ways in which we could train one multimodal model such that it (1) leverages as many existing datasets as possible, (2) can train on any combination of tasks or loss functions, and (3) does not slow down with the addition of any new dataset, task, or loss function. By solving all of these points simultaneously, a multimodal model could scale with an increasingly diverse and rich set of training data without needing to redesign the training framework when new tasks get integrated.

We observe in our empirical results that the combination of diverse, heterogeneous tasks that have been previously established as strong objectives individually ( e.g. , supervised classification and self-supervised contrastive learning) across multiple modalities are not only complementary, but can offer better convergence than training on individual tasks. By implementing Alternating Gradient Descent (AGD) and Mixture-of-Experts (MoE) via recently developed JAX primitives, we enable our model to use a fraction of the computational cost and memory required by similar large-scale perception models [Radford et al., 2021, Jia et al., 2021, Yu et al., 2022], despite the addition of multiple modalities which would normally require 2-8 × compute at similar batch sizes.

Given this context, our contributions and findings are as follows:

We define an integrated modality-agnostic encoder model, and leverage a strong combination of image-text contrastive, video-text contrastive, video-audio contrastive, and image/video/audio classification losses during pretraining to create an I ntegrated M ultimodal P erception (IMP) model, as shown in Figure 1.

Contrasting the conventional approach of summing the losses of multiple objectives, we show that alternating between objectives results in a design that allows seamless integration of virtually any number of tasks and datasets without significant memory overhead and results in better downstream evaluations.

We show that optimization between multiple heterogeneous multimodal tasks is complementary and results in a higher quality model than trained on any individual task.

To train on large batches of video and audio modalities without reducing training efficiency or loss of accuracy, we design a dynamic mixture of various resolutions, sequence lengths, and batch sizes throughout pretraining, and alternate training on all input variations.

We enable our model with MoE, showing strong performance gains compared to a more conventional multi-tower contrastive model, even when appplying MoE to both towers.

We scale our resulting MoE-IMP model to 2B sparse parameters with similar compute to ViT-L (300M parameters), resulting in state-of-the-art performance on several large-scale multimodal video understanding datasets.

2 Related Work

The optimality of AGD optimization vs. averaging the losses (or gradients) has been explored in prior work [Jain et al., 2017, Pascal et al., 2021]. Alternating multimodal multi-task training with AGD has been explored in PolyViT [Likhosherstov et al., 2021], which analyzes different methods to combining heterogeneous task in a single model. The work reports similar findings to our own, that combining objectives can be mutually beneficial and alternating between datasets weighted by the number of examples provides one of the best methods for optimization. Our work extends this to a much more generic setup supporting virtually any combination of modalities, tasks, resolutions, etc.

The use of sparse MoE for multimodal modeling can be seen in recent works like LIMoE [Mustafa et al., 2022], which uses a single MoE encoder for image-text tasks, and VL-MoE [Shen et al., 2023],

(a) The IMP model architecture.

Objective: 1

1D Text

Projection

3D Vision Patch

Projection

1D Waveform

Patch Projection

Data

Embed + Flatten

Heads

Router

FFN 1

FFN N

Self-Attention

…

x L/2 Sparse Layers

Linear Classifiers

V→T

T→V

T→A

V→A

A→V

A→T

JFT

WTS

AudioSet

…

2D Spectrogram

Patch Projection

1D Pos Encode

1D Pos Encode

2D Pos Encode

3D Pos Encode

Objectives

Noise-Contrastive

Estimation (NCE)

Softmax

Cross-Entropy

Binary

Cross-Entropy

V→T

T→V

T→A

V→A

A→V

A→T

IMP Model

Loss Value: N

Optimizer

Objective: N

FFN

Self-Attention

Data: 1

x L/2 Dense Layers

Data: N

…

Loss Value: 1

Gradient

Step: N

Gradient

Step: 1

…

…

Sample Data-Objective Pairs

(Any Subset)

Perform Data-Objective-Specific

Gradient Step

Perform Inference and Loss

Computation

Figure 1: An overview of the IMP Training and Architecture. A mixture of datasets with varying modalities, resolutions, and objectives are randomly sampled at each optimization step and fed into the model. The model is updated alternatingly given the data-objective pairs. We use jax.jit to compile and cache computation graphs to keep each step efficient while also allowing I/O shapes to change for every optimization step without requiring any costly padding or masking strategies.

which uses modality-specific experts for image-text modeling. Our work extends this concept further, introducing video and audio modalities with alternating training on multiple tasks and resolutions without requiring modality-specific experts.

Due to the inherent complexities of integrating modalities in one model, work has been done by simplifying the problem and focusing on a small set of universal objectives applicable to all modalities [Yu et al., 2022, Wang et al., 2022]. Alternatively, some works focused on applying padding or masking strategies to handle the incompatible I/O signatures from different tasks. We found in either case, this severely limits the ability for a model to leverage pre-existing large-scale datasets or to scale to entirely new modalities. The historical reasons are that: (i) most existing models are designed for specific input modalities e.g. , language [Brown et al., 2020, Chowdhery et al., 2022], vision [Dosovitskiy et al., 2020], or audio [Baevski et al., 2020]; (ii) different modalities are typically delegated to separate network weights for the best performance [Radford et al., 2021, Jia et al., 2021]; (iii) optimization difficulty with multiple modalities [Wu et al., 2022, Chen et al., 2022].

3 Method

3.1 Alternating Gradient Descent (AGD)

One of the core pillars of our approach to multimodal understanding is task scalability . I.e., different combinations of data and loss objectives should be interchangeable throughout training, while the addition of any new data or objective should not cause memory or computation overhead. We found that a common issue in training large-scale foundation models in a distributed setting is that input signatures and loss objectives need to be static to avoid major inefficiencies. Accelerated graph compilation APIs allow for low-level graph optimizations that maximize hardware FLOPs utilization on distributed devices such as GPUs and TPUs, but come at a cost of requiring static I/O signatures.

One approach to handle the issue with the static input signature would be to use mixed batching , where all possible inputs are constructed, and inapplicable inputs for a given dataset are padded and outputs are masked accordingly at each training step. However, this comes at a great efficiency cost, since the more tasks that are added the more time is spent computing on padded inputs. The issue with having multiple objective functions is usually resolved by mixed mini-batching , where a batch is divided to multiple mini-batches with their corresponding objective functions. The gradients for each mini-batch and objective function pair are accumulated across multiple mini-batches and the model weights are updated once using an aggregated gradient. However, this approach is also difficult to scale since the gradients across multiple mini-batches are accumulated in memory and per-task batch size naturally reduces as we add more tasks.

We propose a more generic solution based on AGD [Jain et al., 2017], allowing any changes to the three elements of the optimization system: inputs, model, and objective. AGD can be seen as a superset of the conventional SGD, where at each gradient step, a different objective may be optimized given different sets of model weights and/or input modalities. More specifically, any input modality with arbitrary shape could consume any subset of the model while focusing on minimizing any specific combination of the objective functions. According to Jain et al. [2017] it is proven that if each of such optimization steps are convex individually, an alternation between them leads to a guaranteed convergence; as opposed to mini-batching where gradient accumulation could result in sub-optimal results. From a technical point of view, this approach requires compiling multiple computation graphs, one for each unique optimization step. To enable efficient execution of multiple computation graphs, JAX offers native Just-in-Time (JIT) compilation with the jax.jit API 2 , which compiles the graph-of-interest at runtime and compiles a new graph if a change in structure is seen in any of the next optimization steps. Graphs themselves are cached by JAX in an in-memory lookup table so that tasks only need to be compiled once. We empirically observe that graph memory consumption constitute a negligible portion of the entire training. In our experiments, we tested up to 20 unique task structures with no significant reduction in training speed, with compilation taking only 0 . 34% of the total training time, up to the largest scales.

The main AGD algorithm for our multimodal framework is provided in Algorithm 1. We note that we carefully design the loop to be as agnostic as possible with respect to the data-model-objective triplet. This allows for greater flexibility in defining logic inside the model to handle the processing of individual modalities and input shapes. The sampling function can also incorporate state from the optimization process itself. E.g., the loss value of a given step can be used as a reward signal to affect the sampling behavior in the next samplings [Piergiovanni et al., 2023, Mindermann et al., 2022]. In our default setup, we sample each unique task on a given step from a (single trial) Multinomial distribution with probabilities directly proportional to the number of examples in each task. We defer more complicated or reward-based settings to future studies.

3.1.1 AGD-Specific Efficiency Considerations

We notice that in each forward call, certain model states such as activations are stored by default to be used later in the backward call for gradient calculation. This is an established compute optimization at the low-level graph compilation in XLA and similar APIs. Although this trick helps a lot with the training time, it significantly consumes memory and creates memory overhead if more than one graph is compiled and used during training. To reduce memory overhead, we use JAX's

native rematerialization API, jax.checkpoint 3 to save memory by not checkpointing any of the intermediate model states. In our experiments, we observe an average reduction of 70-80% TPU HBMusage while resulting in only 18-20% longer step times.

We also notice that large models with many different objectives may still incur long compilation times. Therefore, we apply scan-over-layers with jax.lax.scan , a method which rolls all of the Transformer layers into a single layer that is called multiple times using different weights (instead of compiling the same function multiple times). This alone results in 15-30x faster compilation time depending on the model length. We observe increasing relative time savings with larger model sizes.

Furthermore, we accomplish these across distributed accelerators through the jax.pjit API 4 , which distributes JIT compilation across multiple accelerators.

3.2 Objectives

Our goal in designing the IMP model is to reuse objectives that have been shown to be robust for learning each modality. Hence, we choose the two most established supervised and unsupervised learning objectives: i. Supervised Classification using Softmax/Binary Cross-Entropy(SCE/BCE), ii. Cross-Modal Noise-Contrastive Estimiation (NCE). Unless otherwise specified, we do not sum any of the above losses or accumulate gradients as would be done traditionally. Instead we apply backprop on each objective individually with AGD.

3.3 Architecture

Figure 1 shows a high-level overview of the architecture of IMP, which consists of three main modules: i. The Embedder , which accepts specific modalities and embeds them in a shared modality-agnostic space; ii. The MoE Encoder , which computes semantic contextual embeddings from the embedded tokens; iii. The Heads , which produce all the final predictions from the Encoder by re-projecting its embeddings back into a modality-specific space. We briefly explain each module here and provide more details in Appendix.

One design decision important to multimodal modeling is how to allocate parameters to each modality. As seen in works like BASIC [Pham et al., 2021], an asymmetric modality-specific design can be more optimal than using a similar-sized model for each modality. However, this comes at the cost of requiring additional hyperparameter tuning to find the optimal parameterization. As we show later in the next section, we observe that through the use of model sparsification with MoE, a unified encoder design coupled with certain modality-specific pre- and post-encoder layers is more optimal than a traditional multi-encoder setup as seen in CLIP model variants.

We follow VATT [Akbari et al., 2021], AudioMAE [Huang et al., 2022], and T5 [Raffel et al., 2020] to extract the vision, audio, and text embeddings, respectively. We add learnable 3D/2D/1D positional encodings to the embeddings and project them to a space with the same dimensionality as the model's. We pass these embedded inputs regardless of modality as-is through the shared encoder, which is a standard Transformer architecture equipped with Mixture-of-Experts FFN layers. We follow V-MoE [Riquelme et al., 2021] and LIMoE [Mustafa et al., 2022] for expert allocation. This can be seen as an inductive bias, allowing each expert to be allocated to multiple modalities if the optimization benefits. One immediate benefit is that the addition of new modalities for fine-tuning does not need any specific changes to the encoder, unlike modality-specific experts which require additional modifications and input handling [Wang et al., 2022, Shen et al., 2023].

We apply modality-specific heads on the encoder representations to produce the final outputs for loss and prediction calculations. For classification objectives, we apply a dataset-specific linear classifier to the average-pooled outputs. For noise-contrastive estimation (NCE), we closely follow the CLIP architecture, applying separate feedforward heads for each modality-to-common-space projection.

3.4 Multi-Resolution Training

One major issue when training Transformers on video data is that computation and memory efficiency are usually bottlenecked due to Transformer's quadratic complexity as a function of the input length.

To counteract this, we propose to adjust batch size or resolution to compensate the additional temporal tokens, hence achieving a similar total number of input tokens compared to a single-frame still image.

We first fix a set tokens per batch T = B × T F × T H × T W , which is factorized by the batch size B , frame tokens T F , height tokens T H , and width tokens T W representing each patchified video. We observe that we can further factorize each batch by trading off different dimensions such that the total number of input tokens per step are roughly equal so that peak memory usage is preserved. For example, we can halve the spatial resolution while quadrupling the number of frames. This can increase convergence especially at the start of training, and provide a more memory efficient encoding of each objective. Furthermore, we leverage DropToken [Akbari et al., 2021] as an additional method to reduce tokens per batch by randomly dropping a fixed ratio of tokens per example. We find that for T F temporal frame tokens, we can randomly drop a ratio of 1 -1 T F tokens per example to match the same tokens per batch as images. For certain objectives we find that a different mix of trade-offs is more optimal. For example, contrastive objectives favor large batch sizes, so we reduce the resolution or apply DropToken to be more memory efficient. On the other hand, classification objectives do not need as large batch sizes for optimal convergence, hence we reduce the batch size while increasing the spatiotemporal tokens.

4 Experiments and Results

4.1 Training Setup

Datasets. Our datasets consist of a diverse set of learnable signals across multiple modalities. We use WebLI [Chen et al., 2022], LAION-400M [Schuhmann et al., 2021], WIT [Srinivasan et al., 2021], CC12M [Changpinyo et al., 2021], and VCC [Nagrani et al., 2022] for vision-text contrastive learning; JFT-3B [Zhai et al., 2022], I21K [Ridnik et al., 2021], and WTS-70M [Stroud et al., 2020] for both supervised classification and label-based vision-text contrastive estimation (similar to BASIC [Pham et al., 2021]); HT100M [Miech et al., 2019] and AudioSet [Gemmeke et al., 2017] for vision-audio-text triplet contrastive loss (similar to V ATT [Akbari et al., 2021]).

We use a proportionally weighted sampling algorithm, executing each task in succession. To ensure that datasets are evenly sampled, we weight each task by the number of examples, normalized to a probability distribution. For each dataset variant with different resolution sizes, we apply the same weight. For a fair evaluation on downstream tasks, we filter all near-domain examples from our pretraining datasets (about 5M examples total).

Multi-Resolution Strategy. In our experiments, we always configure the input parameters so that the number of frame tokens are always equal to 4. This will result in the base tokens per video batch being exactly 4x of image's. For video datasets, we construct three variants and uniformly sample from each variant during training: i. Reduce the resolution by half in each dimension, ii. Reduce the batch size by 4x, iii. Apply DropToken d = 1 -1 T F = 0 . 75 . For image datasets, we also apply a similar strategy but for the purpose of high-resolution learning. In addition to the base resolution, we have two extra variants: i. Reduce the batch size by 4x and double each spatial dimension, ii. Apply DropToken d = 1 -1 4 = 0 . 75 .

Training Parameters. For our final experiments, we train with a patch size of 4x16x16 on base input resolutions of 16x256x256 and 4x256x256 on video and image modalities respectively, resulting in a total of 1024 and 256 patches per sample. The text inputs in ImageNet21K and JFT are truncated to 16 tokens to improve step efficiency with no loss of information, while keeping the text length of the rest of the datasets to a maximum of 256 tokens. We use a base batch size of 65536 and train using the Adam optimizer, a peak learning rate of 1e-3 with a cosine schedule, and apply no weight decay. For MoE parameters, we apply experts-choose routing with a topc capacity factor of 1.0 and do not apply any jittering to the routing or other auxiliary losses. Training results in roughly 16B examples seen, or about 5T tokens. Taken together, these datasets represent about 11B unique image-text and video-audio-text examples.

During inference, we evaluate on the largest available resolution that the model was trained on, i.e., 16x512x512, and use a total of 8 clips per video at approximately 12.5 fps on all evaluated datasets.

4.2 Main Results

We scale up and tune IMP for best performance on video datasets and evaluate it on a variety of downstream tasks and datasets to understand how it generalizes to other modalities. Table 1 shows the zero-shot classification capabilities of the model on several image, video, and audio datasets.

We note that IMP significantly outperforms previous state-of-the-art regardless of the model size and sets new record on Kinetics [Kay et al., 2017, Carreira et al., 2018, 2019], UCF101 [Soomro et al., 2012], and HMDB-51 [Kuehne et al., 2011] top-1 accuracy. Compared to the previous state-of-the-art, VideoCoCa [Yan et al., 2022], we train IMP-MoE-L on 256 TPU v4 chips for 6 days, representing only 15% of the total training cost of VideoCoCa. Considering the total number of parameters per token (PPT), IMP also outperforms the previous comparable state-of-the-art model, CoCa, on ImageNet [Russakovsky et al., 2015] and CIFAR-100 with a relatively large margin. However, we observe that the model falls behind the state-of-the-art in zero-shot audio classification on ESC50 [Piczak, 2015]. This might be explained by the fact that the total training examples for audio modality are almost negligible compared to image and video. Hence, the model has a very strong performance on video and image. We argue that this could be resolved by simply introducing more samples and a more balanced train scheduling method, which we differ to future studies.

4.3 Ablation

In this section, we hightlight experimentation with some key results which motivate the chosen set of features for our final IMP model. We refer the reader to Appendix for more ablation on several other aspects of the model. The experiments in this section use IMP-S or IMP-B trained for 250k steps with a base batch size of 8192. We set a fixed video/image resolution of 16x224x224/4x224x224 using a patch size of 4x16x16. Unless otherwise specified, we do not apply multi-scale resolution.

Combined objectives are mutually beneficial. We train the model on ImageNet21k with two objectives: Noise-Contrastive Estimation (NCE) and Softmax Cross-Entropy (SCE) and explore the following: i. train on the objectives separately, ii. combine the objectives by summing them, and iii. alternating (AGD) between the objectives on each step. In the case of alternating, for a fair comparison so that training time is equivalent, we fix the same number of steps (250k) so that each objective only optimizes 50% of the total steps (125k). We evaluate on ImageNet1k and CIFAR-100 by image-to-text retrieval and linear probing on the frozen model's features and report the results in Table 2. It is not a surprise that classification objective benefits fine-tuning evals the most, while contrastive objective benefits open vocabulary classification. However, we observe that combining both objectives is better than optimizing on them individually. And alternating between the objectives is better than non-AGD objective mixing. These results are similar to the findings of PolyViT [Likhosherstov et al., 2021], which report optimal performance on alternating the objectives, weighted by the size of each dataset. This motivates us to fix one objective per training step and alternate optimization between them.

ImageNet ZS

CIFAR100 Linear

CIFAR100 ZS

Flickr T

I

Flickr I

T

COCO T

I

COCO I

T

10

20

30

40

50

60

70

80

90

17.6

61.5

14.9

28.4

31.6

15.9

18.6

39.1

80.8

48.1

34.4

40.0

19.3

24.2

27.1

83.4

44.2

34.2

39.3

20.8

24.7

49.4

84.6

60.0

38.6

44.6

22.5

28.5

49.7

85.2

62.8

51.5

66.6

25.7

33.4

CC

CC + I21K (NCE)

CC + I21K (Softmax)

CC + I21K (NCE+Softmax)

CC + I21K (NCE+Softmax) + LAION

ImageNet ZS

Flickr T

I

Flickr I

T

COCO T

I

COCO I

T

20

30

40

50

60

70

49.7

51.5

66.6

25.7

33.4

51.3

52.6

68.2

26.2

34.5

51.0

52.7

68.6

25.9

33.7

50.1

53.2

69.2

26.4

33.8

51.4

54.4

69.2

27.2

34.6

224×224

224×224

320×320 (drop 0.5)

224×224

320×320 (batch 0.5×)

224×224

320×320 (drop 0.5)

320×320 (batch 0.5×)

224×224

320×320 (drop 0.5)

320×320 (batch 0.5×)

160×160 (batch 2×)

Multi-task multi-dataset AGD is also mutually beneficial. In Figure 2, we compare the result of adding additional datasets to the pretraining mixture. We additionally compare results across Flickr30k [Young et al., 2014] and COCO [Lin et al., 2014] datasets. We start with CC12M dataset and gradually add new datasets and objectives. Most notably, we compare the addition of I21K dataset, showing complementary improvement when combining NCE and SCE objectives. Similar to I21K isolated experiments, adding SCE benefits the entire pretraining mixture. While SCE benefits zero-shot results, NCE benefits linear probing results too. Certain dataset combinations (CC+I21K, CC+LAION) cause instability at the beginning of training. Adding a classification objective has a stabilizing effect, significantly reducing the chance of slow convergence. Optimizing on LAION directly is difficult, but benefits training a lot more when mixed in with other datasets. This motivates us to further integrate a larger set of diverse datasets and objectives.

Multi-scale resolution provides universal improvement. Figure 3 shows a comparison of using different combinations of resolution, batch size, and DropToken as input. In all settings, we fix the total tokens per batch, and we ensure that all training runs use the same number of total steps. We see that certain types of datasets respond well to DropToken while others may not. CC with double the batch size and DropToken 0.5 improves zero-shot image classification. Droptoken + 320x320 image on I21K SCE pretrain is better for linear probing and image-text retrieval. Adding multiple versions of smaller batch size + higher res, DropToken + higher res, larger batch size + lower res, can significantly improve downstream results. We find that dynamic mixtures of resolution, batch size, and DropToken are always helpful.

ImageNet ZS

CIFAR100 Linear

CIFAR100 ZS

Flickr T

I

Flickr I

T

COCO T

I

COCO I

T

20

30

40

50

60

70

80

90

49.7

85.2

62.8

51.5

66.6

25.7

33.4

60.9

82.8

62.5

49.7

65.6

25.4

33.8

75.0

84.9

68.7

65.3

78.9

35.4

45.1

80.7

88.0

74.9

66.9

80.4

35.1

49.9

IMP-S

IMP-MoE-S, 4 experts

IMP-B

IMP-MoE-B, 16 experts

ImageNet ZS

CIFAR100 Linear

CIFAR100 ZS

Flickr T

I

Flickr I

T

COCO T

I

COCO I

T

30

40

50

60

70

80

90

63.9

84.7

71.5

64.7

78.9

37.0

47.9

34.9

59.6

84.7

65.9

60.3

71.9

37.9

47.9

52.5

81.3

87.3

72.6

66.3

81.4

41.8

54.3

36.2

83.2

88.6

74.1

67.0

80.9

38.1

48.4

67.1

82.2

88.6

73.9

65.8

78.6

38.2

48.3

65.8

IMP-B (image only)

IMP-B (image + video)

IMP-MoE-B, 16 experts (image only)

IMP-MoE-B, 16 experts (image + video)

IMP-MoE-B, 16 experts (image + video + audio)

ImageNet ZS

CIFAR100 Linear

CIFAR100 ZS

COCO T

I

COCO I

T

30

40

50

60

70

80

90

57.7

81.0

66.0

35.3

59.8

84.2

69.4

38.5

47.8

63.2

84.1

71.2

40.1

52.3

68.1

85.6

72.5

39.7

51.1

Two Tower (82M)

Single T ower (86M)

Two Tower (172M)

Single T ower MoE, 4 experts (103M)

MoE provides universal improvement across modalities, and resolves the single-tower encoder parameter bottleneck. The main challenge in designing a unified encoder tower as we have described is that parameters must be split between multiple modalities, thus harming accuracy. Compared to a two-tower contrastive model, the encoder of a unified image-text model contains half the parameters, while keeping training efficiency the same. One direction we explore is whether a large increase in parameters from MoE is sufficient to resolve parameter bottlenecks. In Figure 4, we observe that simply replacing a dense model with an equivalent MoE model with just 4 experts, we can provide a large gain in accuracy, especially for zero-shot metrics. This provides a promising indication that MoEs can be used to bridge the multimodal gap. We observe that with the addition of MoE, we can significantly close the gap between multiple modalities as seen in Figure 5. Since experts are free to choose which tokens are allocated to different experts, we observe strong alignment between experts and modalities.

Single-tower MoE outperforms multi-tower dense variants. We find that out of all the variants we tested, a unified MoE encoder provided the most parameter and compute efficient design, while significantly outperforming a multi-encoder modality-specific model in downstream results, as seen in Figure 6. When comparing two-tower models, we can either split the parameters to be roughly

equal in size to a single tower, or duplicate the towers to double the parameter count while providing equivalent computation. We observe that multi-tower dense models are more compute efficient than single-tower dense models, but less parameter efficient. However, a single-tower MoE model is both more compute and parameter efficient than all variants, showing improved generalization and using fewer parameters with the same compute budget as a multi-tower dense model. These results show universal superior parameter and compute efficiency and higher accuracy by using just 4 experts. This observation suggests that our method can be used for integrated multimodal multi-task modeling without worrying about the complications of modality-specific design choices or downstream performance degradation as observed in previous modality-agnostic designs [Akbari et al., 2021].

5 Conclusion

In this paper we presented an integrated training and modeling approach for multimodal perception using AGD and MoE. We observed that AGD enables task scalability and multi-resolution training, which improves the training convergence and the model's generalization capabilities. On the other hand, we found that MoE can play a very important role in integrating multiple modalities into one unified model. Given these findings, we scaled the model with hyperparamters tuned specifically for video understanding and achieved state-of-the-art performance in zero-shot video action recognition with a significant margin. Furthermore, we observed that the model also generalizes on other modalities and achieves competitive downstream results. In a nutshell, IMP opens a door to data ( e.g. , modality, resolution, etc .) and task scalability - two important directions that have been neglected in many multimodal understanding works due to the inherent technical limitations. Due to the vast range of elements involved in this system, we defer multiple directions to be explored in future work: 1. generative objectives and model architectures, 2. causal MoE for generation, 3. sophisticated methods for data-objective sampling, 4. more downstream evaluations.

Acknowledgments and Disclosure of Funding

We would like to thank Joan Puigcerver, Carlos Riquelme, and Basil Mustafa for their advice on MoE implementation and analysis; Anselm Levskaya for his help with advanced core JAX and Flax implementation; the T5X team for their support for scalable model partitioning, and Erica Moreira and Victor Gomez for their help with resource allocation.

References

Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS , 2021.

Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. NeurIPS , 2020.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. NeurIPS , 2020.

Gemma A Calvert. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral cortex , 2001.

Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340 , 2018.

Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics700 human action dataset. arXiv preprint arXiv:1907.06987 , 2019.

Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiv preprint arXiv:2209.06794 , 2022.

Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311 , 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Jon Driver and Toemme Noesselt. Multisensory interplay reveals crossmodal influences on 'sensory-specific'brain regions, neural responses, and judgments. Neuron , 2008.

Jort F Gemmeke, Daniel PW Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In ICASSP , 2017.

Shreyank N Gowda, Marcus Rohrbach, and Laura Sevilla-Lara. Smart frame selection for action recognition. In AAAI , 2021.

Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. NeurIPS , 2022.

Prateek Jain, Purushottam Kar, et al. Non-convex optimization for machine learning. Foundations and Trends® in Machine Learning , 2017.

Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML . PMLR, 2021.

Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 , 2017.

Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre. Hmdb: a large video database for human motion recognition. In ICCV , 2011.

Valerii Likhosherstov, Anurag Arnab, Krzysztof Choromanski, Mario Lucic, Yi Tay, Adrian Weller, and Mostafa Dehghani. Polyvit: Co-training vision transformers on images, videos and audio. arXiv preprint arXiv:2111.12993 , 2021.

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Antoine Miech, Dimitri Zhukov, Jean-Baptiste Alayrac, Makarand Tapaswi, Ivan Laptev, and Josef Sivic. Howto100m: Learning a text-video embedding by watching hundred million narrated video clips. In ICCV , 2019.

Sören Mindermann, Jan M Brauner, Muhammed T Razzak, Mrinank Sharma, Andreas Kirsch, Winnie Xu, Benedikt Höltgen, Aidan N Gomez, Adrien Morisot, Sebastian Farquhar, et al. Prioritized training on points that are learnable, worth learning, and not yet learnt. In ICML . PMLR, 2022.

Basil Mustafa, Carlos Riquelme, Joan Puigcerver, Rodolphe Jenatton, and Neil Houlsby. Multimodal contrastive learning with limoe: the language-image mixture of experts. In NeurIPS , 2022.

Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. Learning audio-video modalities from image captions. arXiv preprint arXiv:2204.00679 , 2022.

Bolin Ni, Houwen Peng, Minghao Chen, Songyang Zhang, Gaofeng Meng, Jianlong Fu, Shiming Xiang, and Haibin Ling. Expanding language-image pretrained models for general video recognition. In ECCV , 2022.

Lucas Pascal, Pietro Michiardi, Xavier Bost, Benoit Huet, and Maria A Zuluaga. Improved optimization strategies for deep multi-task networks. arXiv preprint arXiv:2109.11678 , 2021.

Hieu Pham, Zihang Dai, Golnaz Ghiasi, Kenji Kawaguchi, Hanxiao Liu, Adams Wei Yu, Jiahui Yu, Yi-Ting Chen, Minh-Thang Luong, Yonghui Wu, et al. Combined scaling for open-vocabulary image classification. arXiv preprint arXiv: 2111.10050 , 2021.

Karol J Piczak. Esc: Dataset for environmental sound classification. In ACM MM , 2015.

AJ Piergiovanni, Weicheng Kuo, Wei Li, and Anelia Angelova. Dynamic pretraining of vision-language models, 2023. URL https://openreview.net/forum?id=QcffIcjq8bl .

Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In ICML . PMLR, 2021.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. JMLR , 2020.

Tal Ridnik, Emanuel Ben-Baruch, Asaf Noy, and Lihi Zelnik-Manor. Imagenet-21k pretraining for the masses. arXiv preprint arXiv:2104.10972 , 2021.

Carlos Riquelme, Joan Puigcerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. NeurIPS , 2021.

Adam Roberts, Hyung Won Chung, Anselm Levskaya, Gaurav Mishra, James Bradbury, Daniel Andor, Sharan Narang, Brian Lester, Colin Gaffney, Afroz Mohiuddin, et al. Scaling up models and data with t5x and seqio. arXiv preprint arXiv:2203.17189 , 2022.

Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. IJCV , 2015.

Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarczyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filtered 400 million image-text pairs. arXiv preprint arXiv:2111.02114 , 2021.

Sheng Shen, Zhewei Yao, Chunyuan Li, Trevor Darrell, Kurt Keutzer, and Yuxiong He. Scaling vision-language models with sparse mixture of experts. arXiv preprint arXiv:2303.07226 , 2023.

Linda Smith and Michael Gasser. The development of embodied cognition: Six lessons from babies. Artificial life , 2005.

Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402 , 2012.

Krishna Srinivasan, Karthik Raman, Jiecao Chen, Michael Bendersky, and Marc Najork. Wit: Wikipedia-based image text dataset for multimodal multilingual machine learning. In SIGIR , 2021.

Jonathan C Stroud, Zhichao Lu, Chen Sun, Jia Deng, Rahul Sukthankar, Cordelia Schmid, and David A Ross. Learning video representations from textual web supervision. arXiv preprint arXiv:2007.14937 , 2020.

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a foreign language: Beit pretraining for all vision and vision-language tasks. arXiv preprint arXiv:2208.10442 , 2022.

Junru Wu, Yi Liang, Feng Han, Hassan Akbari, Zhangyang Wang, and Cong Yu. Scaling multimodal pre-training via cross-modality gradient harmonization. In NeurIPS , 2022.

Wenhao Wu, Zhun Sun, and Wanli Ouyang. Transferring textual knowledge for visual recognition. arXiv preprint arXiv:2207.01297 , 2022.

Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Bidirectional cross-modal knowledge exploration for video recognition with pre-trained visionlanguage models. arXiv preprint arXiv:2301.00182 , 2022.

Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979 , 2022.

Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics , 2014.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR , 2022.

Peer reviews:
comments: SUMMARY:
This work proposes a solution to train multimodal multi-task training and demonstrate the model trained via this strategy outperforms other approaches in zero-short learning problems. The proposed solution mainly relies on current solutions including AGD, JAX library, DropToken, etc. it seems that this work can be helpful in multimodal related works.

STRENGTHS:
1) The proposed training strategy and solution can help related work in multimodal research area. 
2) The training strategy demonstrated better performance on zero-short learning approach over other compared works.

WEAKNESSES:
1) If this work focuses on the training strategy, more studies on downstream tasks are needed. 
2) Although the work focuses on the training solution, more analysis of multimodal experimental results analysis are needed. For example, the analysis of the improvement over other approaches,  especially what kind of cases the new approach help improve the performance?

QUESTIONS:
1) Could the author demonstrate the major difference compared to AGD? In line 119, the author mentioned that it’s a more generic solution based on AGD, however, not clear about the innovation and difference.
2) How can the proposed approach be applied on these cases where one or two modalities are missing? 
3) Will the proposed approach handle the situation like training with several different datasets together? And how to deal with the difference  among them? Is there any other additional steps needed?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-05

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Integrated Multimodal Perception, which can use image, video, and audio datasets for multimodal and multitask training. The method is scalable, benefiting from the alternating gradient descent as it can alternate diverse modalities, loss functions, and datasets to perform gradient descent. Moreover, using a mixture of experts can help handle different modalities with only a modality-agnostic encoder. The multi-resolution training accelerates the training by using different spatial resolutions, sequence lengths, and batch sizes.\n\nSTRENGTHS:\n1. This paper provides a generic multimodal training solution involving input, model, and objective and successfully trained models with a combination of image, video, and audio datasets.\n\n2. It achieves SOTA zero-shot results on video action recognition.\n\n3. The paper is well-written and easy to follow.\n\nWEAKNESSES:\n1. Ablation studies show that alternating between the objectives on each step is better than combining them by summing them. Is this true for all cases or only for large-scale pretraining? Many papers on various topics sum up multiple losses for training. For example, in semi-supervised learning, the loss general consists of two parts: supervised loss and unsupervised loss. We may need to give a proper context for this conclusion.\n\n2. The method's two technical components, alternating gradient descent and mixture of experts, mainly follow previous works. This paper integrates them and applies them to a larger scale of pretraining.\n\nQUESTIONS:\n1. Algorithm 1 shows the sampling uses step t and model state S_t, but Lines 142-143 say that sampling is directly proportional to the number of examples in each dataset. That is, the more samples a task has, the more likely the dataset gets sampled. It would be better to make the algorithm consistent with the descriptions. Moreover, in Algorithm 1, should there be another mini-batch sampling given a chosen task? Should the sampling function f be only for task/dataset sampling?\n\n2. Line 227 mentions filtering examples close to the downstream tasks. How is the filtering conducted?\n\n3. Why the ablation study in Section 4.3 uses spatial resolution 224 while the main results in Table 1 use resolution 256 during training?\n\n4. The paper mentions using JAX in several places, like jax.jit compiling graph-of-interest at runtime, jax.checkpoint saving memory by not checkpointing any of the 152 intermediate model states, and jax.lax.scan accelerating the graph compilation. I'd like to know whether Pytorch can also do so. I know Pytorch also has checkpointing functionality, and Pytorch 2.0 provides torch.compile() API.\n\n5. In Figure 3, a 224x224 image can result in 196 tokens, while a 320x320 image corresponds to 400 tokens. After dropping 50% tokens, it has 200 tokens longers than 196 tokens. Do you use padding to make them have the same length in experiments? \n\n6. Line 237 mentions a patch size of 4x16x16. What does the 4 represent? An image generally has 3 RGB channels. Shouldn't it be 3x16x16? Line 238 says image resolution is 4x256x256, which also confuses me.\n\nLIMITATIONS:\nThe experiments seem not to show whether pretraining with one modality can help improve another modality's downstream performance. For example, whether the image pretraining datasets help improve the downstream video action recognition results and whether using the video pretraining datasets can boost the downstream performance on ImageNet-1K.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes a scalable multimodal multitasking approach. It combines alternating gradient descent and mixture-of-experts to train a unified model. The extensive experiments verify the effectiveness of the proposed method. By scaling up the model, this method sets up a new state-of-the-art in zero-shot video classification.

STRENGTHS:
- The proposed method achieves excellent performance in zero-shot video classification. It further confirms improving data and task scalability in multimodal learning is promising.

- This paper is well written and easy to understand.

WEAKNESSES:
- Using alternating gradient descent for efficient multimodal multitasking learning is not new, which has been explored in PolyViT. It seems that the proposed method is an extension with more engineering improvement. Also, the success of MoE architecture has been validated on image-text tasks. The contribution of this paper is to incrementally extend this architecture to video and audio modalities.

- Compared with other methods, the proposed methods use more training data. Does the improvement mainly come from the scale of the training data? If PolyViT is pretrained on a similar scale of the data, does the proposed method still have advantages?

- typo. Line 264, "differ" -> "defer"

QUESTIONS:
- In Figure 5, the audio modality contributes less to the performance improvement on the image (or video)-text tasks. How about the importance of image (or video) to audio-related tasks?

LIMITATIONS:
Yes, the authors have adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-14

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposed Integrated Multimodal Perception (IMP) for multi-modal multi-task learning. IMP consists of a shared Mixture-of-Experts (MoE) Encoder as well as modality-specific embedding layers and heads re-projecting representations to modality-specific space. Optimizing towards both supervised classification losses and self-supervised contrastive learning NCE losses, the authors proposed to adopt Alternative Gradient Descent (AGD) during training to accommodate different inputs, outputs and learning objectives without bringing too much memory / computation overhead when incorporating more tasks. Furthermore, a multi-resolution strategy was also proposed to train IMP on large batches of video and audio modalities without losing efficiency or accuracy, where various resolutions, sequence lengths, and batch sizes are used dynamically. Experiments on a good amount of public datasets show that the proposed method can achieve better or competitive performance on various downstream zero-shot classification tasks except audio classification on ESC-50. Ablation studies support some of the design choices.\n\nSTRENGTHS:\n1. The paper introduced a possible way to integrate arbitrary number of modalities into one model. The AGD + unified encoder + MoE design seems an interesting solution without losing efficiency and accuracy when incorporating more modalities **in some situations**.\n\n2. The proposed dynamic resolution strategy is somewhat novel and effective.\n\n3. Experiments on considerably various public datasets demonstrate the effectiveness of the proposed method on zero-shot classification when comparing with previous state-of-the-arts\n\n4. Ablation studies on important model designs were conducted to give a more comprehensive understanding of the methods\n\nWEAKNESSES:\n1. One critical problem of IMP is that it seems to suffer from performance drop when incorporating new modalities as per to Fig. 5, possibly due to the shared encoder. Although MoE can alleviate it to some extent in some cases, it is not always working. For example, in Fig. 5, for COCO Text->Image and Image->Text, and K400 zero-shot, IMP can achieve better or similar performance when new modalities of videos are introduced, while IMP-MoE-16Exp obtained worse accuracy. This defect somewhat goes against the major claim of this paper.\n\n2. Another problem is that the model performance on different tasks subjects to change depending on training sample distribution, according to Tab. 1 and Line 261-264. While tuning the distribution of samples of different modalities can improve audio classification, it is not sure whether or not performance on other modality classification will drop, based on Weakness 1. It is also not quite sure how robust the proposed method is against any changes of sample distribution, which may hinder the incorporation of more datasets in the future, going against the major claim again.\n\nQUESTIONS:\nWhile the paper's major claim/contribution is a new method for multi-modal multi-task learning which can integrate any number of modalities/tasks with arbitrary inputs/outputs/objectives, Weaknesses 1 and 2 are critical and should be addressed before an accept decision can be made, in my opinion. Please address them accordingly.\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-17"}