SlowGuess/NeurIPS_2023_OCR huggingface dataset:
MODEL	PPT	TPU-DAYS	IN1K	C100	K400	K600	K700	UCF101	HMDB51	ESC-50
CLIP [Radford et al., 2021]	400M	-	76.2	-	-	-	-	-	-	-
CoCa-B [Yu et al., 2022]	380M	1.8k	82.6	-	-	-	-	-	-	-
X-CLIP [Ni et al., 2022]	400M	-	-	-	65.2	-	-	72.0	-	-
BIKE [Wu et al., 2022]	230M	-	-	-	-	68.5	-	80.8	52.8	-
Text4Vis [Wu et al., 2022]	230M	-	-	-	68.9	-	-	85.8	-	-
AudioCLIP [Gowda et al., 2021]	430M	-	-	-	-	-	-	-	-	69.4
IMP-B	86M	120	80.5	82.4	63.6	62.1	49.9	64.2	39.7	32.8
IMP-MoE-B	90M	150	83.2	84.9	68.2	65.7	52.1	88.7	46.6	47.8
IMP-MoE-L	350M	1.5k	83.9	87.0	77.0	76.8	68.3	91.5	59.1	65.1
Large-scale models
LIMoE [Mustafa et al., 2022]	680M	-	84.1	-	-	-	-	-	-	-
LiT ViT-g [Chen et al., 2022]	2B	-	84.5	83.6	-	-	-	-	-	-
CoCa [Yu et al., 2022]	2B	10k	86.3	-	-	-	-	-	-	-
VideoCoCa [Yan et al., 2022]	2B	10.5k	-	-	72.0	70.1	62.5	86.6	58.6	-
Table 1: Zero-Shot Classification (top-1) results on image, video, and audio datasets. IMP achieves a new state-of-the-art on zero-shot video action recognition by a wide margin with significantly low training cost. Considering the total number of Parameters Per Token (PPT), IMP also significantly outperforms comparable models on zero-shot image classification.
We scale up and tune IMP for best performance on video datasets and evaluate it on a variety of downstream tasks and datasets to understand how it generalizes to other modalities. Table 1 shows the zero-shot classification capabilities of the model on several image, video, and audio datasets.
We note that IMP significantly outperforms previous state-of-the-art regardless of the model size and sets new record on Kinetics [Kay et al., 2017, Carreira et al., 2018, 2019], UCF101 [Soomro et al., 2012], and HMDB-51 [Kuehne et al., 2011] top-1 accuracy. Compared to the previous state-of-the-art, VideoCoCa [Yan et al., 2022], we train IMP-MoE-L on 256 TPU v4 chips for 6 days, representing only  $15\%$  of the total training cost of VideoCoCa. Considering the total number of parameters per token (PPT), IMP also outperforms the previous comparable state-of-the-art model, CoCa, on ImageNet [Russakovsky et al., 2015] and CIFAR-100 with a relatively large margin. However, we observe that the model falls behind the state-of-the-art in zero-shot audio classification on ESC-50 [Piczak, 2015]. This might be explained by the fact that the total training examples for audio modality are almost negligible compared to image and video. Hence, the model has a very strong performance on video and image. We argue that this could be resolved by simply introducing more samples and a more balanced train scheduling method, which we differ to future studies.
4.3 Ablation
In this section, we highlight experimentation with some key results which motivate the chosen set of features for our final IMP model. We refer the reader to Appendix for more ablation on several other aspects of the model. The experiments in this section use IMP-S or IMP-B trained for 250k steps with a base batch size of 8192. We set a fixed video/image resolution of  $16 \times 224 \times 224 / 4 \times 224 \times 224$  using a patch size of  $4 \times 16 \times 16$ . Unless otherwise specified, we do not apply multi-scale resolution.
Combined objectives are mutually beneficial. We train the model on ImageNet21k with two objectives: Noise-Contrastive Estimation (NCE) and Softmax Cross-Entropy (SCE) and explore the following: i. train on the objectives separately, ii. combine the objectives by summing them, and iii. alternating (AGD) between the objectives on each step. In the case of alternating, for a fair comparison so that training time is equivalent, we fix the same number of steps  $(250\mathrm{k})$  so that each objective only optimizes  $50\%$  of the total steps  $(125\mathrm{k})$ . We evaluate on ImageNet1k and CIFAR-100 by image-to-text retrieval and linear probing on the frozen model's features and report the results in Table 2. It is not a surprise that classification objective benefits fine-tuning evals the most, while contrastive objective benefits open vocabulary classification. However, we observe that combining both objectives is better than optimizing on them individually. And alternating between the objectives is better than non-AGD objective mixing. These results are similar to the findings of PolyViT [Likhosherstov et al., 2021], which report optimal performance on alternating the objectives, weighted by the size of each dataset. This motivates us to fix one objective per training step and alternate optimization between them.
OBJECTIVE	IMAGENET1K	CIFAR-100
LINEAR	I→T	LINEAR	I→T
NCE	39.8	46.7	84.0	49.8
Softmax	41.1	-	82.6	-
NCE + Softmax, Sum	47.6	46.7	82.4	51.6
NCE + Softmax, Alternating	49.9	48.0	84.1	52.4
Table 2: Combining multiple objectives during pre-training.. Alternating between both objectives offer the best performance, despite training on the same number of total steps.
Figure 2: Combining multiple datasets and objectives using AGD. We integrate CC, I21k, and LAION with NCE and SCE using AGD and observe consistent improvement in downstream results. We also observe that NCE and SCE are mutually beneficial. Further optimality is provided by adding larger and more diverse datasets like LAION.
Figure 3: Combining multiple input sizes using AGD. We sample each variant with equal probability. Downstream results improve as we add more variants to the training mixture.
Multi-task multi-dataset AGD is also mutually beneficial. In Figure 2, we compare the result of adding additional datasets to the pretraining mixture. We additionally compare results across Flickr30k [Young et al., 2014] and COCO [Lin et al., 2014] datasets. We start with CC12M dataset and gradually add new datasets and objectives. Most notably, we compare the addition of I21K dataset, showing complementary improvement when combining NCE and SCE objectives. Similar to I21K isolated experiments, adding SCE benefits the entire pretraining mixture. While SCE benefits zero-shot results, NCE benefits linear probing results too. Certain dataset combinations (CC+I21K, CC+LAION) cause instability at the beginning of training. Adding a classification objective has a stabilizing effect, significantly reducing the chance of slow convergence. Optimizing on LAION directly is difficult, but benefits training a lot more when mixed in with other datasets. This motivates us to further integrate a larger set of diverse datasets and objectives.
Multi-scale resolution provides universal improvement. Figure 3 shows a comparison of using different combinations of resolution, batch size, and DropToken as input. In all settings, we fix the total tokens per batch, and we ensure that all training runs use the same number of total steps. We see that certain types of datasets respond well to DropToken while others may not. CC with double the batch size and DropToken 0.5 improves zero-shot image classification. Droptoken + 320x320 image on I21K SCE pretrain is better for linear probing and image-text retrieval. Adding multiple versions of smaller batch size + higher res, DropToken + higher res, larger batch size + lower res, can significantly improve downstream results. We find that dynamic mixtures of resolution, batch size, and DropToken are always helpful.
Figure 4: IMP with Mixture-of-Experts. Results show that using a modest 4 experts increases the model's accuracy substantially on ImageNet zero-shot evaluation. When we scale up the experts to 16, we see a consistent and significant improvement across all downstream evaluations.
Figure 5: Improved multimodal perception using MoE. Results show significant improvement on diverse multimodal perception when we use the MoE variant of IMP. The addition of audio reduces accuracy on image and video metrics across the board, but is much less prominent when using MoE.
Figure 6: Comparison of single-tower vs. multi-tower designs on IMP-B. A single-tower MoE model is both parameter efficient and compute efficient compared to multi-tower dense variants.
MoE provides universal improvement across modalities, and resolves the single-tower encoder parameter bottleneck. The main challenge in designing a unified encoder tower as we have described is that parameters must be split between multiple modalities, thus harming accuracy. Compared to a two-tower contrastive model, the encoder of a unified image-text model contains half the parameters, while keeping training efficiency the same. One direction we explore is whether a large increase in parameters from MoE is sufficient to resolve parameter bottlenecks. In Figure 4, we observe that simply replacing a dense model with an equivalent MoE model with just 4 experts, we can provide a large gain in accuracy, especially for zero-shot metrics. This provides a promising indication that MoEs can be used to bridge the multimodal gap. We observe that with the addition of MoE, we can significantly close the gap between multiple modalities as seen in Figure 5. Since experts are free to choose which tokens are allocated to different experts, we observe strong alignment between experts and modalities.
Single-tower MoE outperforms multi-tower dense variants. We find that out of all the variants we tested, a unified MoE encoder provided the most parameter and compute efficient design, while significantly outperforming a multi-encoder modality-specific model in downstream results, as seen in Figure 6. When comparing two-tower models, we can either split the parameters to be roughly
equal in size to a single tower, or duplicate the towers to double the parameter count while providing equivalent computation. We observe that multi-tower dense models are more compute efficient than single-tower dense models, but less parameter efficient. However, a single-tower MoE model is both more compute and parameter efficient than all variants, showing improved generalization and using fewer parameters with the same compute budget as a multi-tower dense model. These results show universal superior parameter and compute efficiency and higher accuracy by using just 4 experts. This observation suggests that our method can be used for integrated multimodal multi-task modeling without worrying about the complications of modality-specific design choices or downstream performance degradation as observed in previous modality-agnostic designs [Akbari et al., 2021].
In this paper we presented an integrated training and modeling approach for multimodal perception using AGD and MoE. We observed that AGD enables task scalability and multi-resolution training, which improves the training convergence and the model's generalization capabilities. On the other hand, we found that MoE can play a very important role in integrating multiple modalities into one unified model. Given these findings, we scaled the model with hyperparameters tuned specifically for video understanding and achieved state-of-the-art performance in zero-shot video action recognition with a significant margin. Furthermore, we observed that the model also generalizes on other modalities and achieves competitive downstream results. In a nutshell, IMP opens a door to data (e.g., modality, resolution, etc.) and task scalability — two important directions that have been neglected in many multimodal understanding works due to the inherent technical limitations. Due to the vast range of elements involved in this system, we defer multiple directions to be explored in future work: 1. generative objectives and model architectures, 2. causal MoE for generation, 3. sophisticated methods for data-objective sampling, 4. more downstream evaluations.
We would like to thank Joan Puigcerver, Carlos Riquelme, and Basil Mustafa for their advice on MoE implementation and analysis; Anselm Levskaya for his help with advanced core JAX and Flax implementation; the T5X team for their support for scalable model partitioning, and Erica Moreira and Victor Gomez for their help with resource allocation.
[1] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text. NeurIPS, 2021.
[2] Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework for self-supervised learning of speech representations. NeurIPS, 2020.
[3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. *NeurIPS*, 2020.
[4] Gemma A Calvert. Crossmodal processing in the human brain: insights from functional neuroimaging studies. Cerebral cortex, 2001.
[5] Joao Carreira, Eric Noland, Andras Banki-Horvath, Chloe Hillier, and Andrew Zisserman. A short note about kinetics-600. arXiv preprint arXiv:1808.01340, 2018.
[6] Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint arXiv:1907.06987, 2019.
[7] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR, 2021.
[43] Junru Wu, Yi Liang, Feng Han, Hassan Akbari, Zhangyang Wang, and Cong Yu. Scaling multimodal pre-training via cross-modality gradient harmonization. In NeurIPS, 2022.
[44] Wenhao Wu, Zhun Sun, and Wanli Ouyang. Transferring textual knowledge for visual recognition. arXiv preprint arXiv:2207.01297, 2022.
[45] Wenhao Wu, Xiaohan Wang, Haipeng Luo, Jingdong Wang, Yi Yang, and Wanli Ouyang. Bidirectional cross-modal knowledge exploration for video recognition with pre-trained vision-language models. arXiv preprint arXiv:2301.00182, 2022.
[46] Shen Yan, Tao Zhu, Zirui Wang, Yuan Cao, Mi Zhang, Soham Ghosh, Yonghui Wu, and Jiahui Yu. Video-text modeling with zero-shot transfer from contrastive captioners. arXiv preprint arXiv:2212.04979, 2022.
[47] Peter Young, Alice Lai, Micah Hodosh, and Julia Hockenmaier. From image descriptions to visual denotations: New similarity metrics for semantic inference over event descriptions. Transactions of the Association for Computational Linguistics, 2014.
[48] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[49] Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. In CVPR, 2022.