UnIVAL: Unified Model for Image, Video, Audio and Language Tasks
Authors: Mustafa Shukor
Corentin Dancette
Alexandre Rame
Matthieu Cord
Venue: Accepted by TMLR ()

Abstract: Large Language Models (LLMs) have made the ambitious quest for generalist agents significantly far from being a fantasy. A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities. A promising solution is unification, allowing the support of a myriad of tasks and modalities within one unified framework. While few large models (e.g., Flamingo (Alayrac et al. 2022)), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text. The question that we ask is: is it possible to build efficiently a unified model that can support all modalities? To answer this, we propose UnIVAL, a step further towards this ambitious goal. Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into a single model. Our model is efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. UnIVAL shows competitive performance to existing state-of-the-art approaches, across image and video-text tasks. The feature representations learned from image and video-text modalities,  allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained on audio. Thanks to the unified model, we propose a novel study on multimodal model merging via weight interpolation of models trained on different multimodal tasks, showing their benefits in particular for out-of-distribution generalization. Finally, we motivate unification by showing the synergy between tasks. The model weights and code are available at: https://github.com/mshukor/UnIVAL.

Published in Transactions on Machine Learning Research (12/2023)
UnIVAL: Unified Model for Image, Video, Audio and Lan- guage Tasks
Mustafa Shukor
Sorbonne University MLIA, ISIR, Paris, Fronce
Corentin Dancette
Sorbonne University
MLIA, ISIR, Poris, France
Alexandre Rame
ddeeanndre. upme..
Sorbonne University
MLIA, ISIR, Paris, France
Matthieu Cord
matthieu. saatheee..or@@srroooomuvvvurrrse
Sorbonne University, MLIA, ISIR, Paris, Prance Valeo.ai, Paris, France
Reviewed on OpenReview: https: // openreview. net/ form?? id=
Abstract
Large Language Models (LLMs) have made the ambitious quest for generalist agents sig- nilicantly far from being fantasy. A key hurdle for building such general models is the diversity and heterogeneity of tasks and modalities. A promising solution is unification, allowing the support of a myriad tasks and modalitics within one unified framework. While few large models (e.g., Flamingo (Alayrac et al., 2022 ), trained on massive datasets, can support more than two modalities, current small to mid-scale unified models are still limited to 2 modalities, usually image-text or video-text. The question that we ask is: is possible to build efficiently unified model that support all modalities? To answer this, propose UnIV AL, step further towards this ambitious goal. Without relying on fancy datasets sizes or models with billions of parameters, the ~ 0.25B parameter UnIVAL model goes beyond two modalities and unifies text, images, video, and audio into single model. Our model efficiently pretrained on many tasks, based on task balancing and multimodal curriculum learning. UnIVAL shows competitive performance existing state-of-the-art approaches, across image and video-text tasks. The feature representations learned from image and video-text. modalities, allows the model to achieve competitive performance when finetuned on audio-text tasks, despite not being pretrained audio. Thanks to the unified model, we proposc novel study on multimodal model merging via weight interpolation of models trained on different multimodal lasks, showing their benefits in particular for out-of- distribution generalization. Finally, we motivate unification by showing the synergy between tasks. The model weights and code are available at: https:/ htips://gihnan htt.ghgttu.c..oo comuubbccmm/aee htt.uuuw....om/ssuoooor..ooo htt////sshuuuuu woooo//nssook//UAAAA UnIVAL
1 Introduction
The advent of Large Language Models (LLMs) (Brown et al. 2020 Rae et al. 2021 Chowdhery et al. 2022 Tay et al.) 2022) represents significant step towards the development of generalist models. Generally based on the Transformer architecture /Vaswani al. 2017) and a single next-token prediction objective, they continue to astound the world with their remarkable performances in text. understanding and generation.

---

Published in Transactions on Machine Learning Research (12/2023)
LM UnIVAL
Whtch repis "Porphe
Enerite?
The begg alrpiao sn nawvy
What ducc the tmap. doorthe?
rolkda pattes peopl
New tasks and modalities (not used during pretraining)
What
Caste
graing graee par surrwasde
Figure 1: UnIVAL model. Our sequence-to-sqquenc model unifies the architecture, tasks, saput/outpru format, and training objective (next token prodiction). UnlV pretrained on image and video-text tasks and can finetuned to tackle new modalities (andio-text) and tasks (text-to-image generation) that were not used during pretraining.
Nevertheless, their current limitation to single modality (text) restricts their understanding and interaction with the world. This highlights the need for robust multimodal models handling diverse tasks across numerous modalities. Recently, many works have tried to go beyond single modality, and build powerful multimodal models (Huang et al., 2023 Driess et al.) 2023 Li et al., 2023) that surpass previous task htte/moodltttssscciii approaches. However, most of these works focus on image-text tasks and only a handful of approaches aim incorporate more than two modalities, such image/ video-text. (Alayrac al.
2022 Wang et al. 2022b)
The prevailing approach for pretraining multimodal models revolves around training them on large, noisy image-caption datasets (Schuhmann et al.) 2021 Jia et al. 2021 Radford et al. 2021), where the model is tasked with generating or aligning image-captions through causal generation or unmasking. However, this approach encounters a significant challenge: it relies on extensive datasots to compensate for the inherent noise and the relatively simple task of caption generation. relatively small yet high-quality datasets presents an alternative solution to learn efficient models capable of
contrasl, multitask learning (Caruana] 1997) on
competing with their largo-scale counterparts (Alayrac ct al. 2022 Chen et al. 2022b
Reed al. 2022).
Current small to mid-scale (less than couple of hundred million parameters) vision-language models Li et al.
2019 Shukor et al. 2022 Dou et al. 2021 Li et al. 2022b) still have task-specific modules/heads many training objectives, and support a very small number of downstream tasks due to the different input/ /output format. Recently, the seatencc--o-sequeen
OFA (Wang et al. 2022c) and Unified-IO (Lu et al. 2022a have made noticeable step towards more unified systems that can support a wide range of image and image-text tasks, with more reasonable scales (e.g. can fit on user-grade GPU). These models are pretrained many good quality, public benchmarks. On video-text tasks, LAVENDER (Li et al. 2022c takes a similar direction by unifying the pretraining tasks Masked Language Modeling (MLM). Sequence-to-sequent unified models are particularly well-suited for open-ended text generation tasks and can readily incorporate recent LLMs. To guide the model in solving specific lask, a textual prompt resembling an instruction Raffel et al. 2020) added at the beginning the input sequence. They have the capability to unify tasks across different modalities, and thus easily supporting new tasks, by representing all inputs and outputs sequences of tokens, utilizing an unified input/ mput/output output format and vocabulary. These tokens can represent various modalities such text, image patches, bounding boxes, audio, video, any other modality, without.
2

---

Published in Transactions on Machine Learning Research (12/2023)
the need for task-specific modules/heads These strategies are straightforward to scale and manage, they involve single training objective and a single model.
However, existing works are still limited to downstream tasks with no more than 2 modalities (image-text or video-text). Providing unification across a larger number of tasks and modalities would offers additional advantages. First, we would benefit from the knowledge transfer across them, by harnessing their collaborative strengths. Second, once pretraining is done, the model can be finetuned on many different datasets: because of the wider range of more diverse pretraining data, unification across more tasks would enable better and more efficient generalization after finetuning novel tasks and modalities. In this paper, thus ask the following question.
Is it possible to efficiently build unified model that can support all modalities?
A positive answer to this question will pave the way for building generalist models that can potentially solve any task. To answer this question, we propose UnIVAL, step further towards generalist modality-agnostic models. UnIVAL (illustrated in Fig D goes beyond two modalities and unifies text, images, video, and audio into a single model.
Our contributions are multiple:
To the best of our knowledge, UnIVAL is the first model, with unified architecture, vocabulary, input/output format, and training objective, that is able to tackle image, video, and audio lan- guage tasks, without relying on large scale training or large model size. Our 0.25B parameter model achieves competitive performance to existing modality-custommzzed work. With comparable model sizes, we achieves new SoTA on some tasks (e.g. +1.4/ 1..4/+0.9/+0... points accuracy on REICCCCOReeCOCOO+ /Re//OO Visual Grounding, +3.4 CIDEr on Audiocaps).
We show the benefits multimodal curriculum learning with task balancing, for efficiently training the model beyond two modalities.
We show the importance of multitask pretraining, compared to the standard single task one, and study the knowledge transfer between pretrained tasks and modalities. In addition, we find that pretraining on more modalities makes the model generalizes better to new ones. In particular, without any audio pretraining, UnIVAL able to attain competitive performance to SoTA when finetuned on audio-text tasks.
We propose novel study multimodal model merging via weight interpolation (Txmailov et al. 2018 Wortsman et al. 2022 Rame et al. 2022). We show that, when weights are finetuned on different multimodal tasks from our unified pretrained model, interpolation in the weight space can effectively combine the skills of the various finetuned weights, creating more robust multitask models without any inference overhead. Thus, addition to multitask pretraining, averaging differently finctuned weights another way to leverage and recycle (Rame al., (2023a) the diversity of multimodal tasks, enabling their collaboration. This is the first study of weight interpolation showing its effectiveness with multimodal foundation models.
2 Related Work
We provide brief related work, further detailed in Appendix B
Multimodal pretraining. So far, most of the efforts to build multimodal models have been focused
vision-language pretraining. Contrastivo-based approaches [Radford ct al.) 2021: Jia et al.) 2021) try to learn a shared and aligned latent space by training on hundreds of millions of pairs. More data-efficient approaches (Shukor et al.) 2022 Li et al.) 20211a 2022b Dou et al. 2021 Singh et al. 2022) relied on additional multimodal interaction modules and varicty of training objectives such as image-text text matching,
masked language modeling and image-text contrastive (Chen et al.) |2020c Kim et al.) 2021 Lu al. 2019
Zhang et al. 2021 Tn the video-language community, similar approaches have tried to model the interaction
3

---

Published in Transactions on Machine Learning Research (12/2023)
between language and frames sequences (Cheng et al. 2022 Wang et al. 2023a Fu et al.
2021
Zellers et al.
2021: Yang et al. 2021a). Few works have targeted both image and video language pretraining 2022b
(Wang et al.
Machod
PT exsupplo (V)
Model Size
Param. init
PT Modaliiis V-T
Del Modalitis
V-T
A-T
Arch. 1/0 Tisks Objeeetive
Unitted
1.5R/1298
O. 0.7R/5.18
/DaVi
20240
30/150//1 177B
VIT
mI5
CoCa
Yu AM X72
4.S.
2.10
Random
Rondom
DailieFF77 207
201220
180M
0.283/0.888/2..88
Random
25
OrniVV
15,331 (2 SM)
0.20
T TimeSSormes
BERT
VIOLET
S.3M (162.5X))
0.21
VithoSwin
BERT
Mastoi
(9603))
0.313 /0.71
AAST
LAVENDER
19M (14-4M)
00.2
BERT
BLI1P-
129M+
EVA/CLIP
Flu TT/TT
Flauiing Plavan d 1 2022
2.58 (27M)
3,28//0 /0.01 /50B
CLIP
Chinchalla
OFA
6UM+
RowNet
BART
Gete
2.2R=
1,28
RoxNot
N/A
UnIVAL
21.43 (SM)
0.25B
MoNNe//eeNoN
BART
Table 1: Comparison of different foundation models. Our UnIVAL approach is pretrained on relatively small dataset,
tackles aange/video/addeee modalities, while unifying the different aspects explained in See 3 and in Appendix c unified model, input/output format, pretraining tasks, and training objective.
Unified models. Building unified systems has been explored first in the NLP community. Raffel et al. (2020) proposed the T5 transformer model, text-to-text framework that solves many NLP tasks, each one being described by a task-specific textual prefix. Since then, building general textual models has been heavily explored with LLMs (Brown et al.) 2020: Rae et al., 2021 Chowdhery et al.] 2022). This inspired other communities to build unified models. In the vision community, the work of Chen et al. (2022a), proposed a pixel-to-soquence framework to unify different vision tasks such as object detection and instance segmentation. For multimodal tasks, Cho et al. (2021) proposed to unify vision-language tasks as conditional lext generation. OFA WWagg et al. 2022c) then proposed large-scale ter-eence----equence framework and extended previous approaches to more image-text. tasks, including text-to-image generation. Similarly, Unified-TO (Lu et al., 2022a), in addition to image-text tasks, targets many visual tasks including dense prediction ones. The closest works to us are indeed OFA (Wang el. al. (022c) and Unified-IO (Lu el al. 2022a), however, we propose lo unify tasks across more modalities, with significantly smaller model and dataset sizes. The differences are clarified in Tab where we compare different foundation models involving unification.
Weight averaging across multimodal tasks. To combine multiple expert models with diverse special- izations, we leverage a simple yet. practical strategy: linear interpolation in the weight space. We follow Ilharco et al. (0223: Daheim al. (2023); Ortiz-Jimenez et al. (2023) suggesting that averaging networks in weights combine their abilities without any computational overhead. In particular, weight averaging
(WA) was shown useful in model soups approaches (Wortsman et al. 2022 Rame et al. 2022 to improve out-of-distribution generalization as an approximation of the more costly averaging of predictions (Laksh
minarayanan et al.
2017). Recent works extended WA to weights fine-tuned with different losses |Rae
et al. 2022 2023b Croce et al.) 2023) or on different datasets (Matena & Raffel 2022 Choshen et al. 2022
Don- Yehiya et al. 2023 Rame et al. (023a). In addition, some techniques try leverage the features learned
on different auxiliary tasks for a given target task. Fusing (Choshen et al., 2022) averages multiple auxiliary weights to serve an initialization for the unique finetuning on the target task. In contrast, ratatouille
[Raee et al. 2023a) delays the averaging after the multiple finetunings on the target tasks: each auxiliary model is finetuned independantly on the target task, and then all fine-tuned weights are averaged. These approaches consider classification lasks, for a given modality (usually images): interpolating weights of models trained on different multimodal tasks very little investigated. The most similar and concurrent work is the recent Sung et al. (2023) applying a complex architecturo-speeffc merging stratogy. This work differs from us, as we explore WA during finetuning on multimodal downstream tasks, where they merge models pretrained on different modalities.
4

---

Published in Transactions on Machine Learning Research (12/2023)
3 Pretraining of UnIVAL
Current multimodal models are pretrained on massive noisy datasets with limited number of tasks (e.g., image-conditooned text generation). We focus the challenge of achieving reasonable performance without many good-quality
relying on vast amounts of data. Our approach involves multi-task pretraining datasets. We hope that the quality mitigates the need for massive datasets, thereby reducing computational requirements, while enhancing the overall model capability. The adoption of such an approach has become increasingly easy due to the growing availability of public, datascts. UnIVAL is unified along the following axes (more detailed Appendix Ch model, input/outpin format, pretraining tasks, and training objective.
or automatically generated
3.1 Unified model
Our model's core is LM designed to process abstract representations. It is enhanced with lightweight modality-specific projections that enable the mapping of different modalities to shared and more abstract representation space, which can then be processed by the LM. We use the same model during pretraining and finetuning of all tasks, without any task-specific heads. We detail below key components of this architecture, that are further detailed in Appendix D.
Shared module. To tackle multimodal tasks at small to mid-scale, employ an encoder-decoder LM, its effectiveness for multinodal lasks has been demonstrated compared to decoder-only LMs (Wang et al. 2021), in addition to its superiority in zero-shot generalization after multitask training (Wang et al. 2022d]. Another advantage of this architecture the inclusion of bidirectional attention mechanisms in addition to umidirectional causal attention. This particularly beneficial for processing various non-textual modalities. Our model accepts a sequence of tokens representing different modalities as input and generates a sequence of tokens as output.
wiigtt specialized modules. To optimize data and compute requirements, it is crucial to map different modalities to a shared representation space, before feeding them into the encoder of the LM. To achieve this, we employ weght modality-specific encoders. Fach encoder extracts a feature map, which is then flattened to generate a sequence of tokens. These tokens are linearly projected match the input dimension of the LM. It is important to strike balance in the choice of encoder complexity. Using overly simplistic encoders, such as linear projections, may disrupt the LM, impede training speed, and necessitate larger datasets and then computational resources. Conversely, employing excessively complex encoders can hinder the benefits of learning a unified representation in the shared module. In our approach, we opt for CNN encoders as they scale effectively with reshurtsolution inputs, minimize the number of output tokens, and exhibit improved efficiency during both inference and training compared to transformers.
3.2 Unified
format
The input/output of all tasks consists of a sequence of tokens, where we use a unified vocabulary that contains text, location, and discrete image tokens.
3.3 Unified pretraining tasks
To train a single model on many tasks, a unified representation of these tasks is necessary. As our model's core is LM, we transform all tasks into a sequence-to-sequence format, where each task is specified by textual prompt (e.g., 'What does the video describe?" for video captioning).
For pretraining lasks, we pretrain only on relatively small public datasets, such as image captioning (COCO Lin et al.) 2014), Visual Genome (VG) (Krishna et al.) 2017b), SBU (Ordonez et al., 2011), CC3M (harma ct al. 2018) and CC12M (Changpinyo et al. (021)) (only in the first stage)), VQA (VQAv2 (Goyal et al.
2017), GQA (Hudson & Manning 2219)), VG (Krishna et al. 2017b)), Visual Grounding (VGround) and
referring expression comprehension (ReICOCO, ReICOCO+, RefCOCOg Yu et. al.] (016)), video captioning
5

---

Published in Transactions on Machine Learning Research (12/2023)
(WebVid2M [Bain et al. 2021 and video question answering (WebVidQA Yang et al. 2021a ) Note that
we only use the training sets during pretraining. Pretraining tasks are further detailed in A Appendix P
3.4
Unified training objective
We follow other approaches (Wang et al., 2022c Alayrac et al. 2022 and optimize the model for conditional next token prediction. Specifically, we use a cross-entropy loss.
3.5
Efficient pretraining
Besides unification across tasks and modalities, detail different techniques for efficient pretraining.
Multimodal curriculum learning (MCL). Other works train the model on all tasks and modalities simultaneously (Wang al. 2022c Li et al. 2022c). However, we have observed that models trained on more modalities tend to exhibit better gencralization to new ones. To capitalize on this, we employ different strategy wherein we gradually introduce additional modalities during training. This approach facilitates a smoother transition to new modalities by providing a better initialization. Furthermore, this paradigm siguificantly reduces computational requirements compared to training on the entire dataset at once. Previous studies [Wang et al.) 2022b) have demonstrated notable performance enhancements when employing this paradigm for shared visual encoders (applied to both images and vidoos). In our work, we extend this setting beyond shared visual encoders, and show its effectiveness for modality-specifie projections and unified models. This approach mainly yields gains in training efficiency. This is important as it allows uss to leverage existing pretrained multimodal models to incorporate new modalities. To validate the approach, model on imago-text and vidco-text data for 20 epochs using 2 training approaches; the one-stage approach where we train on all data from the beginning, and our 2-stage curriculum training where we start to train on image-text for 10 epochs then we contimue training on all data for the next 10 epochs. Tab 2 shows that the performance of both approaches are comparable. However, the 2-stage approach is more efficient in terms of training time (18% faster) and memory (25% less GPU memory).
train the same
Training Time
batch size (avg)
COCO
VQAv2 (Acc)
RefCOCO+ (Acc@0.5)
MSR-VTV (CIDEr)
MSRVTT-QA (Acc)
Method
(CIDEr)
One-stage
2h04m
4K
127.9
73.21
70.89
55.9
42.38
MCL
1h42m
3K
128
73.24
70.19
56.3
42.27
Multimodal Curriculum learning (MCL). We show that multi-stage training is more efficient than the one stage one and leads on par rosults. The training time is for one epoch the same number of GPUs.
Table 2:
Multimodal task balancing. Contrary to previous work (Wang et al. 2022c we find it more beneficial to balance the tasks in the batch, especially when using highly unbalanced datasets. Tab 3 shows some results. We compare models trained without balancing, where in each batch the number of examples for each task is proportional the corresponding dataset size, and with task balancing, where the tasks have similar number of examples. The results show consistent improvement after balancing especially with highly unbalanced datasets (e.g., when adding CC12M, the overall performance drops significantly (B+CC12M)).
Implementation details for pretraining. The architecture of the LM a typical encoder-decoder transformer initialized by BART-base (Lewis et al. 2020 with few modifications, following the implementation details of other work (Wang et al. 2022c). The speelity encoders
ResNet-101 pretrained on ImageNet as image encoder, 3D ResNext-101 (Hara et al., 2018b) pretrained on kinctics 400 as video encoder and PANN encoder pretrained for audio classification as audio encoder, we do not skip the last block as done by previous approaches (Wang et al. 2022c). We use Adam optimizer with weight decay 0.01 and linear decay schoduler for the learning rate starting from 4. All model parameters are pretrained 2 stages; first we train only on image-text tasks for 150k steps and batch size 3200, then we add video-text tasks and continue training (after removing CC12M) for 90K steps with batch size 4K (2k for each modality). At the end of the

---

Published in Transactions on Machine Learning Research (12/2023)
Data
Task Balancing
COCO (CTDEr)
VQA v2 (Acc)
RefCOCO+ (Acc@0.5)
B B+CC12M
127.0
72.93
66.03
126.8
72.79
68.04
B+VQA+ Ground..
129.9
74.43
78.78
B+VQA+
/
130.3
75.44
78.99
3-VVA--Ground-+CC122 -r01
129.9
75.21
78.85
131.3
75.34
79.47
Table 3: Multimodal task balancing. Task balancing significantly improve the performance, espocially when using datasets that largely differ in size (e.g., CC12M). The baseline (I3 consists of; VQAv2, IalCOCOO-/CCCM///////OO0 vQA; GQA/VG. Ground:: RelCOCO//4lCOCOOg
last stage, we train the model for additional epoch after increasing the resolution of images from 384 to 480 and the videos from 224 x 224 and 8 frames to 384 384 and 16 frames. More details in Appendix
Data exammlees
COCO
VQAx2
RefCOCO+
MSR-VTT
MSRVTT-QQ TA-QA
Data (Modality)
Method
(CIDEx)
(Ace)
(Ace13055
(CIDEr)
(Acc)
CC3M (I)
2.8M
117.3
69.5
55.2
CC12M (I)
10M
120.2
71.6
56.7
One-task pretraining
CCM--CCI2M (I)
12.8M
123.6
71.7
59.8
COC0-SBU+VG+CCAM (I)
5M
125.8
72.0
56.1
B (I)
5.6M
127.0
72.9
66.0
B+VQA ())
7.4MM
128.9
73.2
71.0
B+Grouud (I)
9.3M
Multitasic pretzaining
129.8
74.4
77.6
B+VQA+Ground
129.9
75.1
78.8
B+VVAA+CGouud-CCI2M
21.6M
130.0
75.2
78.9
D (I+V)
8.1M
128.8
73.2
70.1
54.6
B+ WebiVIA (I+V)
10.6M
Multitask protraining
128.0
73.2
70.2
56.3
42.3
B+VQA+ +eoviiAW (I+V)
13.9M
131.7
75.0
77.9
57.0
42.6
B+Gromd WWelVaad WebVidQA (I+V)
17.6M
131.1
75.1
78.1
56.2
42.5
Table 4: Knowledge transfer across tasks and datasets. We show the synergy between different tasks and datasets.
Multitask learning is efficient as it leverages the collaboration across tasks.. Models are trained longer I+V tasks.
Knowledge transfer across tasks and modalities. We investigate the knowledge transfer between lasks/modalities. We train for 10 epochs on image-text (I) datasets, followed by 10 epochs on image/video-tes (I+V) datasets. The results are shown T We first compare between single and multitask learning. For single task, the models are trained on different image captioning datasets. For multitask learning, the models are trained for several tasks such captioning, VQA or grounding. Overall, multitask learning is more efficient. as with comparable number of examples, it significantly outperforms models trained single task. Second, we investigate the synergy between tasks and datasets. For image-text pretraining, there is a clear benefit of multitask training. Specifically, training on VQA helps to get +1.9 points on Captioning and +5 points for Visual Grounding, Similarly training on VGround, we have larger improvements on captioning and VQA. For imago-text and vidco-text pretraining, VidcoQA helps Vidco Caption and interestingly, Image VQA helps video lasks. We noticed that large datasets like CC12M does not bring significant improvements, compared adding additional task with smaller number of examples. This also demonstrates that multitask learning is more efficient than large-scale single task learning.
We put in Appendix our experiments that study further the knowledge transfer across modalities.
4 UnIVAL on Downstream Tasks
In this section, we present the experimental results of UnIVAL following different setups; finetuning on downstream datasets and direct evaluation without finetuning (e.g. zero-shot). As UnIVAL is unified and targets more than two modalities, for fair comparison to, highlight other unified approaches in yellow and models targeting more than 2 modalities in red We did not highlight UnIVAL for clarity of presentation.

---

Published in Transactions on Machine Learning Research (12/2023)
4.1
Finetuning multimodal tasks
For downstream tasks, finetune standard image-text, video-text and audio-text benchmarks (Appendix contains more implementation details). To have fairer comparison with OFA, we finetune the author's released checkpoint (denoted as OFA Base using the same hyperparametres as UnIVAL.
G
4.1.1
Image-text tasks
RefCOCO
RefCOCO+
RefCOCOg
Model
val
testA
test.
val
frst.A
tesd
val-u
tees -
VL-T3
Cla el al
2021
71.3
UNITER Chen
2020le
81.11
87.01
74.17
81.15
66.70
74.86
75.77
VILLA
2720
87.48
74.84
76.17
81.51
66.81
76.18
76.71
MDETR
Kannah
20021
86.75
89.58
81.41
79.52
84.09
70.62
81.64
80.89
UniTAB
Yang of al.
21
88.59
91,06
83.75
90.97
85.36
71.55
81.58
81.70
OFA
Wang ct a..
2022c
88.48
90.67
83.30
$1.39
87.15
7/1.99
82,29
82.31
OFAg
92.04
94.03
88.44
87.
91.70
80.71
88.07
88.78
UnIVAL (ours)
89.12
91.53
85.16
82.18
86.92
75.27
84.70
85.16
Table 5:
Finetuning for Visual Crounding RefCOCO, RefCOCO+, and RefCOCOg datasets. UnIVAL achieves the SoTA results among comparable model sizes. We report Acc@0.5.
Visual Grounding. We evaluate the ability the model to localise spatially the text in the image. This task consists of predicting the coordinates of bounding box given an input text. The task is cast as sequence generation task, where the model outputs a sequence of 4 pixel locations corresponding the bounding box. Tab shows that achive new SoTA results all 3 benchmarks. Interestingly, scores are better than the reported OFA scores, which additionally pretrain for object detection.
the corners of
VQAr2
SNLI-VE test
Model
test-dev
lest-std
UNITER
Chen et al.
20200:
73.8
74.0
79.4
79.4
OSCAR
pl.
73.6
VILLA
21211
Crees- Entrope Optimisation METEOR
BLEUS4
CIDEs
SPICE
VinVI
212
76.5
76.6
UNIMO
ot
2022
75.0
75.3
81.1
80.6
VL-T5
Cloo
116.5
ALBEF
66 al
212
75.8
76.0
80.8
80.9
OSCAR
L et
20206
37.1
30.7
127.
23.5
METER
VICIA
Sbnkor et
Dou et
2021
2022
77.7
75.0
75.1
77.6
79.9
80.9
81.2
79.4
UniTAB
ViaVL Zane et al.
Ynng
2121
35.5
28.6
30.1
130.8
21.7
23.1
Text- gemeruttonn approscches
GIT-L
Nang et
2172a
42.0
30.8
138. 5
23.8
V1.:5
Cho
2021
OmnnVVL
Wabt
133.9
UnITBB
Yang
2121b
70.7
71.0
OFA
Wang
21/23c
42.5
30.6
138.1
23.7
GIT-L
Wain
75.5
Large-amie
pretraining
OmniVi
Wang et
21/2b
78.3
78.4
LEMON Hu ml. 20222
30.8
OFA
Bose
Wang
20/2
77.0
77.1
78.8
78.6
PaLM-E 848
SinVLMteree
Warre
2021
10.3
33.4
138.0
142.5
21.7
Large- preitrsiiiing
UnIVAL (ours)
42.0
30.5
137.0
23.6
SimVLMrarr [Wang al.
2021
79.3
79.6
85.7
85.6
Florence (Yua et al
2021
80.2
80.1
PaLM-E 84B (Driess et al
2023
80.5
UnIVAL (ours)
77.0
77.1
78.2
78.6
Table 6: Finetuning Image-Text understanding and generation tasks such VQAv2, SNLI-VE and Image Captioning. Our text-generation based approach is competitive with other SoTA, while using less pretraining data.
Multimodal understanding tasks. We evaluate on VQA and Visual entailment tasks, that we cast as text generation. Tab shows a comparison with other approaches. Despite pretraining on less data for less number of steps, our approach is on par with the previous unified model OFA (Wang al. (022cc finetuned from the author's relcased checkpoint (OFA) ) For comparable scale, we significantly outperform GIT, [Wang al.) 2022a) that uses CLIP-VITE as image encoder. Our model is competitive with other SoTA models trained on large datasets that casted the task classification. Note that, evaluate both model
8

---

Published in Transactions on Machine Learning Research (12/2023)
and OFA, with beam search for VQA, instead of all-candidate evaluation. For SNLI-VE, our approach uses only the image and the text hypothesis, without the text premise previously done in OFA (Wang et al.) 2022ch. The results on SNLI-VE suggest that unified models such OFA and our models underperform on the visual entailment task.
Multimodal gencration tasks. We evaluate the model for image captioning COCO dataset (Ln et al. 2014), and report the scores on the Karpathy test split. Tab shows that we are comparable with OFA. Compared to the previous OmniVL model (Wang et al. 2022b; that pretrain on both image and video text datasets, we largely outperform it by more than 3 points CIDEr. Our model is very close to other SoTA such as GIT-L and large-scale trained ones such as LEMON and PaLM-E 84B.
4.1.2 Video- Text tasks
Here we evaluate the model on different video-text tasks.
Method
# PT Images/videos
MSRVTT-QA
MSVD-QA
ClipEITT
Lei ct al.
2021
0.15M/-
37.4
JustAsk
Yang et
21215
-009M
11.5
46.3
ALPRO
21222
3M/2.5M
42.1
45.9
MERLOT
Zelers
2021
-/180M
13.1
nl. 22/21
3.3M/162M
All-in-one Wang ct al
20236
-/283M
46.8
48.3
GIT WWang et a.. 21725n
808M/-
43.2
56.8
Omm Wain
20226
14M/2.8M
44.1
51.0
LAVENDER [L.
20220
14M/14.4M
45.0
56.6
UnlV AL (ours)
14M/2.5M
43.48
19.55
Table 7: Finetuning for VideoQA on MSRVTT-QA and MSVD-QA datasets. The text-generation based UnIVAL model is competitive with SoTA models customized for videos or trained on significantly larger datasets. We report the accuracy.
Video question answering. We evaluate for VideoQA on MSRVIT-QA and MSVD-QA (Xu et al, 2017) datasets. Tab shows a comparison with other approaches. On MSRVTT-QA, we outperform large scale pretrained models like GIT, including models trained on more videos (MERLOT) and customised for VideoQA (JustAskk). We competitive with the unified video model LAVENDER with heavier vision encoder (Video Swin), trained on more videos (and restrict the generated answers to one word), and the ununified OmniVL targeting both images and videos. On MSVD-QA, we have competitive performance to previous work.
ActivityNet- ActivityNct-Captions
MSRVTT
Method
B03
B04
Method
#PT Image (Video) Data
BGt
II
DCEV
Krishns et
2017a
4.09
1.60
8.88
UniVI. Lt al
SwinBERd
2720
(136M)
42.2
25.2
61.2
49.5
BL-SS
Wang 16 ml
2018s
10.89
MV-CPT" Sea
CLIP4Captsa Tong BL
2105
2021
(G8M)
48.9
46.1
30.7
35.7
63.7
64.0
57.7
G0.0
MWSDEC
HACA
Wapg el
Rallmett e6
2018
2010
5.76
3.04
2.71
1.46
11.16
7:23
LAVENDER
MDVC
[adian & Railu
202216
1.46
7.23
BMT
Daxma Helhe
201
4.63
1.99
10.90
UnIVAL (ours)
14M (2.M
48.02
29.01
62.92
60.5
MV-CPT
Scp nle
6.84
12.31
UnIVAL AL (ours)
4.76
Table 8: Finetuning for Video Captioning on MSRVTT and ActivityNet--appeee Cattionss UnIVA AL competitive with other wesk/modality-cuttomiii SoTA that are trained on larger datasets. T, uses addition text transcript. For ActivityNee-Captioon we use ground-truth action proposals.
Video captioning. We evaluate our model for Video Captioning. Tab shows that our model is very competitive with other approaches customized for videos, trained on much larger datasets (LAVENDER) and use speech transcript as additional input (MV-GPT). On ActivityNect-aapiion with ground truth proposal, we outperform previous approaches by significant margin as per the B04 metric and we are competitive with the current SoTA MV-GPT.
9

---

Published in Transactions on Machine Learning Research (12/2023)
Dataset
Method
BLEU,
BLEU2
METEOR
CIDER
SPICE
Kim el al.
2019b
0.614
0.446
0.203
0.503
0.144
Xu et al.
2021
0.655
0.476
0.229
0.660
0.168
Audiocaps
Mci ct al.
2021
0.647
0.488
0.222
0.679
0,160
Lau et al.
2022
0.671
0.498
0.232
0.667
0.172
UnIVAL (ouns)
0.690
0.515
0.237
0.713
0.178
Takenchi et al.
2020
0.512
0.325
0.145
0.290
0.089
Koizuri el ml
2020
0.521
0.309
0.149
0.258
0.097
Chen al.
202
0.534
0.343
0.160
0.346
0.108
Clotho vl
Xu
2020
0.561
0.341
0.162
0.338
0.108
Eron & Scrt
2020
0.590
0.350
0.220
0.280
Xu et al.
202
0.556
0.363
0.169
0.377
0.115
Koh et al.
2022
0.551
0.369
0.165
0.380
0.111
UnIVAL (ours)
0.569
0.367
0.178
0.380
0.114
Table Finetuning on the new audio-text modality for audio-captioning. We compare UnIV AL to other audio-text models on Audiocaps and Clotho vl datasets. Despite not using audio-text during pretraining UnIV AL is very competitive with other customized SoTA. We compare with models that rely only audio as input. The best and next best scores are bolded and underlined respectively.
4.1.3 Audio- Texi Tasks
In addition to considering only modalities seen during pretraining, we explore if UnIVAL also works well for potential new ones. To this end, we evaluate the model on the new audio modality. We use an additional audio encoder pretrained on audio classification and finctune jointly the encoder and the core model pretrained on our image/video-text data.
Audio captioning. We evaluate the model on standard audio captioning datasets; Clotho vl and Audiocaps. Tab shows a comparison with other approaches that take solely the audio as imput. Interestingly, we significantly outperform other approaches on Audiocaps, and we are competitive with the current SoTA on the small Clotho vl dataset.
4.2 Evaluation without finetuning
Model
Lost-dey Ace
VQA+2
0000 Capáion
Val/Tist CIDES
Vil Acet00.5
++00000+
Uuitel Deyes
Moidl
GIEVQA
Vol Aec
VieWaa: Ave
NoChap ERDEY Orek donsi))
MBBY-T-QA The
MIVIAGA
Unilied-1ppus Lu 2022
61.8
101./
GEA (Ray
40.00
17.80
48:35
OFAgue
Wang
05.91
74.47/ 47/75.27
30445
LAVENDET
4.5
11.6
11.0
275
Un
50,07 99.04
70,11
UsIVAL
38:91
17.04
5M1
00.3
Table 10: Evaluation without finetuning. UnIVAL outperforms OFA and competitive with Unified-IO trained data.
Table 11: Zero-Shot Evaluation. Scores in gray means the dataset used during pretraining. UnIV AL is competitive with modality-specife models.
Evaluation on seen datasets. Following Lu et al. (2022a). we directly evaluate the representation learned during pretraining without task-specific finetuning, We compare our model to different baselines following the same setup, with the main difference that other baselines pretrain longer, on siguificantly larger datasets and more tasks. T0 shows that our approach outperforms the most similar baseline OFA on all tasks. Compared to Unified-IO,, we are significantly better on VQAv2, despite pretraining on less VQA datasets.
Evaluation on unseen datascts (zero-shot). We follow the same previous sctup, but we evaluate the model on new datasets, unseen during pretraining. Tab II shows a comparison with other models on several image and video-text datasets. Our model is very competitive to OFA, and close to Unified-IO (grayed scores) on OKVQA. However, Unificd-IO pretrains on both OKVQA and VizWiz. Compared the unified video-language model LAVENDER, we significantly outperform it on video tasks. Our approach attains close performance to the large-scale Flamingo-3B model OKVQA and MSVD-QA.
10

---

Published in Transactions on Machine Learning Research (12/2023)
4.3 Generalization to new tasks and modalities
In this section we investigate the importance of pretraining on different modalities for the generalization tasks and modalities. This is important in scenarios where want the model to do well also potentially novel tasks or modalities. Specifically, want to validate the following hypothesis; pretraining more modalilies, and thus more lasks, allows to learn more modalily and lask-ognoslic represenlation.
Modality
Multitask Audiocaps
Image-Text
X
54.4
Text
57.6
Method
CLIP score f
Text
53.2
Text
31.0
Tmage-Test Text
58.4
Image-Tese
31.6
Video- Text
Image Textt+ Vixeo+
Video Text
31.3
Image-Texte Video- Text
58.8
Table 13:
Finetuning for text-to-image gen- eration COCO dataset. Multimodal pretrain- ing improves the results when finetuning new text-to-image generation, task not during pretraining.
Table 12: Finetuning for Audio Captioning the Au- diocaps dataset. We compare different initialization (after pretraining TextesTeext (I), Videos-exxt (V), Text (T)) for audio captioning, Pretraining on modalities leads to better results when finetuning on audio captioning, a task not seen during pretraining.
Better initialization for new modalities: from vision-language to audio-language tasks. We finetune our model for audio captioning the Audiocaps dataset. To compare the effect of pretraining more tasks and modalities, we evaluate the same model with different initialization; pretraining on text (the model initialized from BART), pretraining on image-text (with and without multitask pretraining), pretraining on video-text and pretraining on both image and video-text. We pretrain for the same number of epochs. Tab [2 shows that pretraining on image- text and video-text data leads to better scores on Audiocaps, compared to the model pretrained on text. Interestingly, the model pretrained on both modalities attain the best scores. This support our underlying hypothesis. We also show the importance of multitask pretraining, by comparing two models trained image-text tasks; one with single task CC3M and CC12M (12.8M examples) and another one with multitask on COCO, VG, SBU, CC3M, VQAv2 and RefCOCO+ (5.6M examples). The results validates again the importance of multitasking in generalization to new modalities/tane stasks..
Better initialization for new tasks: from multimodal input to multimodal output. Here, we investigate our pretrained model can be good initialization to add new tasks. We experiment with a more challenging scenario; text-to-image generation. We finetune the model with different initialization on the COCO dataset and report the CLIP score (Wu et al. 2022)- Tab showw that pretraining on either image-text video-text data helps to get additional improvement, with more improvement coming from pretraining on image-text lasks.
5 Weight Interpolation of UnIVAL Models
Previously, we showed the synergy between tasks and modalities that results from multitask pretraining. Here, instead, we use interpolation the weight space leverage this synergy. We follow the literature on weight interpolation (Izmailov et al.) 2018 Wortsman al.] 2022 Rame et al., 2022) to merge models finetuned different multimodal tasks, without inference overhead. Our framework is ideal candidate for this investigation, due to the unified architecture and the shared pretraining (Neyshabur et al. 2020), which naturally enforces lincar mode connectivity (Frankle et al. 2020) and thus averegability (without requiring weight permutation as in Ainsworth et al. (2022)) across finetuned weights. We consider 4 image-texi (asks; Image Captioning (IC), VQA, Visual Grounding (VGround) and Visual Entailment (VE), and provide similar results for vidco tasks Appendix J Then, given two models with weights W, and W2 finctuned on 2 different tasks among those image-text tasks, we analyze the performance of a new model whose weights are W = A Wi (1 X). W2, where À E [0, 1]. We propose to study the following questions.
11

---

Published in Transactions on Machine Learning Research (12/2023)
Scenario 1: can trade off between 2 different multimodal tasks by weight interpolation? In Fig 2 we plot the performances of the model whose weights are defined by interpolation (A E [0, between different finetuned weights. All weights were initialized from the same "pretrain init", which performs badly on the considered tasks (grey star). By vanilla finctuning (blue stars at the endpoints of the blue line) on a target lask, we consistently improve results on the corresponding metric, yet at the cost of severe performance degradation on other tasks; this suggests that the different tasks are in tension, and optimizing one degrades another. Then, by weight interpolation between these two vanilla finetuned endpoints, we reveal convex front of solutions that trade-off between the different abilities, validating that can effectively combine the skills of expert models finetuned on diverse multimodal tasks. Actually, it is even possible to find an interpolating coefficient À so that the interpolated model outperforms the specialized one: e.g., in the first subplot, the CIDEr score of the model for À = 0.8 with weights 0.8 OCap + 0.2 QVQA is 138.26 vs. 136.10 for the captioning expert Cop corresponding to = 1. We speculate the interpolated model benefits from the synergy between different tasks.
184
Visual Grounding (Aco@i (.5)
Visual Grounding (Aco@0.3) 33
600-
Captioning (CIDEr)
129
Pretrain init
Pretrain init
Vanilla
Vanilla
102
Ratatouille
Ratatouille
25.0
225
Pretrain init Vanilla
72
200-
Ratatouille
20 VQA (Ace)
100
130
130
Captioning (CIDEr)
Visual Eniailment (Ace)
Figure 2: Weight interpolation between models trained on different: multimodal tasks.
Besides, we also experiment more costly finetuning approach, ratatouille (Rame al. 2023a where each finetuning leverages the other tasks as auxiliary tasks. For example, when considering VQA as the target task; (i) we first finctune weights on the auxiliary tasks (VGround, VE and IC); (ii) then we launch multiple finetunings on VQA from these auxiliary weights; (iii) we uniformly average all the weights finet on VQA to obtain W Similarly, to obtain Wic specialized on IC, we apply the same recipe but this time the final finetunings are on the target task IC. Then, as shown on the left subplot from Fig we plot the performances for W" = X- WV + (1 A)- WTe where [0, 1]. The obtained (yellow) front is the right and above the vanilla. (blue) front, validating the superior performances of ratatouille.
Scenario 2: given N models trained on different multimodal tasks, can we obtain a single model that is good on seen, and new unseen tasks? We consider N = 4 models with weights (W finetuned independently, one for each task. This time, we simply average them uniformly: X ==1/N, and then we consider the weight average (WA) N EN W; and plot its performance in Fig 3 First, we observe that this vanilla WA outperforms the "pretrain init", evaluated on training tasks (in-distribution or ID setting) but also when evaluated on new tasks Cout-of-distributie or OOD setting), without any additional training. Performances are improved when averaging uniformly ralatouille finetunings $ ΣN WT. We also consider the fusing (Choshen et al. 2022) strategy, which considers the average the auxiliary weights y E W, the initialization for a second step of finetunings on the target tasks to obtain {W/YN,, and then report fusing performs better than vanilla fine-tuning in ID, but not OOD. When comparing fusing and ratatouille, they perform similarly ID; yet, in the OOD setting, ratatouille outperforms fusing, validating the results from Rame et al. (2023a). In conclusion, these experiments show that uniform averaging can merge different finetuned models to get one general model that performs well on all seen and unseen tasks. Though, results could be improved with tailored selection of the interpolation coefficients X.
the performance for 1/2 Σ W:':
12

---

Published in Transactions on Machine Learning Research (12/2023)
VQA(Acc)
DKVQAAcc)
Pretrain Init
Pretrain Init
Vanilla WA
Vanilla WA
Fusing WA
Fusing WA
Ratatouille + WA
Ratatouille + WA
71 550
45.275
4.52
BTA 193
VE(Acc)
Cap(CIDEr)
(2
(ocDps(CIDEr)
ViZiz(AAc)
VG(Acc@O.55
ID Evaluation.
OOD Evaluation.
Figure 3: Finetuning for OOD. We uniformly averago the models finetuned on 4 image-text tasks and evaluate the resulting model on the same (ID) and new (OOD) tasks.
6 Discussion
Limitations and discussion. Despite the good quantitative results, we find that UnIVAL suffers from several limitations. First, UnIVAL can hallucinate; describe objects not present in the image (Rohrbach) ett al. 2018), prioritizing coherence its generation rather than factuality. In the case of VQA, the model can generate plausible response that can not answered given the image. similar challenge arises visual grounding, where UnIVAL may ground objects that are not mentioned in the text not present in the image. Nonetheless, in comparison to other large models like Flamingo (Alayrac et al. 2022), we show in Appendix K that UnIVAL demonstrates reduced hallucinations. The reason why models such as Flamingo hallucinate more might be due to using frozen LLMs, & component that is known to be susceptible to hallucinate (Zhang et. al. 2023 Shukor el al. 2023b). Second, it struggles in complex instruction following. We have observed that the model's performance is suboptimal to intricate instructions, such as identifying a specific object in the presence of similar alternatives, detecting small or distant objects, and recognizing numerals. Other limitations that are also important lo address are; social and other biases (hat can be learned from the datasets, toxic generations, and explainable generation. These limitations might not be solved by merely scaling the model, and might need different approaches Shukor et al. (2023b); Rame et al. (022b)). In Appendix K we provide a detailed discussion on these limitations and interesting future directions.
Conclusion. In this study, we introduce UnIVAL, the first unified model capable of supporting image, video, and audio-text tasks. We achieve this with a relatively small model with ~ 0.25B parameter on dalaset of relatively small sizes. Our unified system, pretrained with multitasking, offers several advantages. Tt. harnesses the synergies between diverse tasks and modalities, enables more data-efficient training, and exhibits strong generalization capabilities to novel modalitics and tasks. The unification aspect of our strategy paves the way to interesting techniques to merge models finetuned on different multimodal tasks: we demonstrate rate that, in addition to multitask pretraining, merging by weight interpolation can further exploit the tasks diversity. Ultimately, we aspire that our work inspires the research community and accelerates the progress toward constructing modality-agnostic generalist assistant agents.
7 Acknowledgments
This work was supprted by HPC resources of CINES and GENCL, The authors would like to thank the staff of CINES for technical support in managing the Adastra GPU cluster, in particular; Chrn-Christophe Penalva, Johanne Charpentier, Mathieu Cloirec, Jerome Castaings, Gérard Vernou, Bertrand Cirou and José Ricardo Kouakou. This work was also partly supported by ANR grant VISA DEEP (ANR-20-CHIA-0002))
13

---

Published in Transactions on Machine Learning Research (12/2023)
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. nocaps: novel object captioning at scale. In ICCV, 2019.
Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo
permutation symmetries. In ICLR, 2022.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems, 35:23716 2233666-3736, 2022.
Max Bain, Arsha Nagrani, Gül Varol, and Andrew Zisserman. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1728-1738, 2021.
Ali Furkan Biten, Lluis Gomez, and Dimosthenis Karatzas. Let there be clock the beach: Roducing object hallucination in image captioning. In Winter Conference Applicalions of Compuler Vision, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-99244 2020.
Rich Caruana. Multitask learning. Machine Learning, 28:41-75, 1997.
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training recognize long-tail visual concepts. In Proceedings of the IEEE/CVF Conference Computer Vision and Pottern Recognition (CVPR), pp. 3558 3568, 2021.
Kun Chen, Yusong Wu, Ziyue Wang, Xuan Zhang, Fudong Nian, Shengchen Li, and Shao. Audio captioning based on transformer and pro-trained emn. In Proceedings of the Detection and Classification of Acoustic Scenes and Events 2020 Workshop (DCASE2020), pp. 21 25, Tokyo, Japan, November 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. In International conference on machine learning, pp. 1597-1607. PMLR, 2020b.
Ting Chen, Saurabh Saxena, Lala Li, Tsung-Yi Lin, David J Fleet, and Geoffrey E Hinton. A unified sequence interface for vision tasks. Advances in Newral Information Processing Systems, 35:31333 31346, 2022a.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Gryener, Basil Mustafa, Lucas Beyer, et al. Pali: A jointly-scaled multilingual language-image model. arXiu preprint ar ari::2209.0007 2022b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on computer vision, pp- 104-120. Springer, 2020c.
Feng Cheng, Xizi Wang, Jie Lei, David Crandall, Mohit Bansal, and Godas Bertasins. Vindlu: recipe for effective hideo-and-language pretraining. arXiv preprint arXiv:2212, 25051,, 2022.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying vision-and-language tasks via text generation. In International Conference on Machine Learning, pp. 1981-1942. PMLR, 2021.
Leshem Choshen, Elad Venezian, Noam Slonim, and Yoav Katz. Fusing finetuned models for better pretraining. Xiv preprint ar Xiv:2200 03044, 2022.
14

---

Published in Transactions on Machine Learning Research (12/2023)
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXic preprint arXiv:220,.020: 2022.
Francesco Croce, Sylvestre-Alvise Rebuffi, Evan Shelhamer, and Sven Gowal. Seasoning model soups for robustness to adversarial and natural distribution shifts. In CVPR, 2023.
Nico Daheim, Nouha Dziri, Mrinmaya Sachan, Iryna Gurevych, and Edoardo M Ponti. Elastic weight removal for faithful and abstractive dialogue generation. arXiv preprint urXiu:2303.1757 2023.
Wenliang Dai, Zihan Liu, Ziwci Ji, Dan Su, and Pascale Fung. Plansible may not be faithful: Probing object hallucination in lision-language pre-training, arXiv preprinl arXiv:2210.07688 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. arXiv preprint Xrxe:22305.000 06500,, 2023a.
Wenliang Dai, Zihan Liu, Ziwei Ji, Dan Su, and Pascale Fung. Plausible may not be faithful: Probing object ballucination in vision-language pre-training. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2136-2148, Dubrovnik, Croatia, May 2023b. Association for Computational Linguistics. URL /tttps://cclanthoolggg. httphhhoooogyrrg222222223 eac.-3aaclmmmmllmme 156
Corentin Dancette, Spencer Whitehead, Rishabh Maheshwary, Ramakrishna Vedantam, Stefan Scherer, Xinlei Chen, Matthieu Cord, and Marcus Rohrbach. Improving selective visual question answering by learning from your peers. In CVPR, 2023.
Mostafa Dehghani, Josip Djolonga, Basil Mustafa, Piotr Padlewski, Jonathan Heek, Justin Gilmer, Andreas Steiner, Mathilde Caron, Robert Geirhos, Ibrahim Alabdulmohsin, et al. Scaling vision transformers to 22 billion parameters. arXiv preprinl arXiv:2302.05. 05442, 2023.
Ming Ding, Zhuoyi Yang, Wenyi Hong, Wendi Zheng, Chang Zhou, Da Yin, Junyang Lin, Xu Zou, Zhou Shao, Hongxia Yang, et al. Cogview: Mastering texi-to-image generation via transformers. NearIPS, 2021.
Shachar Don- Yehiya,, Elad Venezian, Colin Raffel, Noam Slonim, Yoav Katz, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning. In ACL, 2023.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhilang Sui. A survey for in-context learning. arXiv preprint orXiv:2301.0025; 2022.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Zicheng Liu, Michael Zeng, et al. An empirical study training end-to-end vision-and-lange man-uang transformers. arxiv preprint ar vxXie:2111.02288 2021.
Danny Driess, Fei Xia, Mehdi SM Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, et al. Palm-e: An embodied multimodal language model. arXiv preprint arXiv:2303.032 03378, 2023.
Konstantinos Drossos, Samuel Lipping, and Tuomas Virtanen. Clotho: An audio captioning dataset. In
ICASSP, 2020.
Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabeseu, and Anette Frank. Magma- multimodal augmentation of generative models through adapter- based finetuning. arXiv preprint arXiv:2112.05253, 2021.
Rahim Entezari, Hanie Sedghi, Olga Saukh, and Behnam Neyshabur. The role of permutation invariance in linear mode connectivity of neural networks. In ICLR, 2022.
Aysegül Özkaya Eren and Mustafa Sert. Audio captioning basod on combined audio and semantic embeddings. In 2020 IEEE International Symposium on Multimedia (ISM), pp. 41 41-48, 2020. doi: (w.1109/ISM... 2020.. 00014.
15

---

Published in Transactions on Machine Learning Research (12/2023)
Jonathan Frankle, Gintare Karolina Dziugaite, Daniel M. Roy, and Michael Carbin. Linear mode connectivity and the lottery ticket hypothesis. In ICML, 2020.
Tsu-Jui Fu, Linjie Li, Zhe Gan, Kevin Lin, William Yang Wang, Lijuan Wang, and Zicheng Liu. Violet: End- to-end video-language transformers with masked visual-token modeling. arxiv preprint arXiw:211.116881, 2021.
Zhe Gan,
Chen, Linjie Li, Chen Zhu, Yu Cheng, and Jingjing Liu. Large-scale adversarial training for vision-and- lisio--an-langguage representation learning. Advances in Neural Information Processing Systems, 33: 6616-6628, 2020.
Jort F. Gemmeke, Daniel P. W. Ellis, Dylan Freedman, Aren Jansen, Wade Lawrence, R. Channing Moore, Manoj Plakal, and Marvin Ritter. Audio set: An ontology and human-labeled dataset for audio events. In Proc. IEEE ICASSP 2017, New Orleans, L.A, 2017.
Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. In Proceedings of the IEEE conference computer vision and pallern recognilion, pp. 6904 6913, 2017.
Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. In CVPR, 2018.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cmns retrace the history of 2d cnns and imagenet? In CVPR, 2018a.
Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh. Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenel? In Proceedings of the IEEE conference on Compuler Vision and Pallern Recognilion, pp. 6546 6555, 2018b.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pp. 16000 16000-16009, 2022.
Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. Gans trained by two time-scale update rule converge to local nash equilibrium. NewrIPS, 2017.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hondricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. Xin preprind Xiv::22003 (l.:2203.1556,, 2022.
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106 206.006666965 2021.
Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision-language pre-training for image captioning. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17980 17989, 2022.
Shaohan Huang, Li Dong, Wenhui Wang, Yaru Hao, Saksham Singhal, Shuming Ma, Tengchao Lv, Lei Cui, Owais Khan Mohammed, Qiang Liu, et al. Language is not all you need: Aligning perception with language models. arXiv preprint 21ki::2302.144445 2023.
Yupan Huang, Hongwei Xue, Bei Liu, and Yutong Lu. Unifying multimodal transformer for bi-directional image and text generation. In ICM, 2021.
Drew A Hudson and Christopher D Manning. Gqa: A new datasct for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp- 6700 6709, 2019.
16

---

Published in Transactions on Machine Learning Research (12/2023)
Vladimir lashin and Esa Rahtu. A better use of audio- cues: Dense video captioning with bi-modal transformer. arXiv preprint ar arXiv:205500277, 2020a.
Vladimir lashin and Esa Rahtu. Multi-modal dense video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognátion Workshops, pp. 958 558-959, 2020b.
Gabriel Ilharco, Mitchell Wortsman, Samir Yitzhak Gadre, Shuran Song, Hannaneh Hajishirzi, Simon Kornblith, Ali Farhadi, and Ludwig Schmidt. Patching open-vocabulary models by interpolating weights. In Newr/PS, 2022.
Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannanch Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. Tn ICLR, 2023.
Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Averaging weights leads to wider optima and better gencralization. In UAI, 2018.
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajanugen Logeswaran, Moontae Lee, Kyungjae Lee, and Minjoon Seo. Exploring the benefits of training expert language models over instruction tuning. arxiv preprial ar Xix:23302 03202, 2023.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904 4916. PMLR, 2021.
Keller Jordan, Hanie Sedghi, Olga Saukh, Rahim Entezari, and Behnam Neyshabur. REPAIR: Renormalizing permuted activations for interpolation repair. In ICLR, 2023.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Aldetr-modulateed detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1780-1790, 2021.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint Xrv: 06950, 2017.
Chris Dongjoo Kim, Byeongchang Kim, Hyummin Lee, and Gunhee Kim. Audiocaps: Generating captions for audios in the wild. In NAACI-HITT, 2019a.
Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. AudioCaps: Generating captions for audios in the wild. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Compulational Linguisties: Human Language Technologies, Volume (Long and Short Papers), pp. 119-132 132, Minneapolis, Minnesota, June 2019b. Association for Computational Linguistics. doi: 1038653/11/119.//
URL
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: sssion-and-languaaee transformer without convolution or region supervision. In International Conference on Machine Learning, pp. 5583 5594. PMLR, 2021.
Andrew Koh, Xue Fuzhao, and Chnng Eng Siong. Automated audio captioning using transfer learning and reconstruction latent space similarity regularization. In ICASSP 2022-2022 IEFE Internalional Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 7722 7726. IEEE, 2022.
Jing Yu Koh, Ruslan Salakhutdinov, and Daniel Fried. Grounding language models to images for multimodal generation. XrXiv preprint arXiv:2301. 13823, 2023.
Yuma Koizumi, Ryo Masumura, Kyosuke Nishida, Masahiro Yasuda, and Shoichiro Saito. A transformer-basel audio captioning model with keyword estimation. Proc. Interspeech 2020, pp. 1977-1981 2020.
Qiuqiang Kong, Yin Cao, Turab Iqbal, Yuxuan Wang, Wenwu Wang, and Mark D Plumbley. Panns: Large-scale pretrained audio neural networks for audio pattern recognition. ACM, 2020.
17

---

Published in Transactions on Machine Learning Research (12/2023)
Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles. Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision, pp. 706-715, 2017a.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Ti, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32 73, 2017b.
Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. Simple and scalable predictive uncertainty estimation using deep ensembles. In NeurIPS, 2017.
Jie Lei, Linjie Li, Luowei Zhou, Zhe Gan, Tamara L Berg, Mohit Bansal, and Jingjing Liu. Less is morc: Clipbert for video-and-languuage learning via sparse sampling. In Proceedings of the IEEE/CVP Conference on Computer Vision and Pottern Recognition (CVPR), pp. 7331-7341 2021.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameler-efficien prompt luning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 1045-3059) 2021.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequenco-to-soquonee pre-training for natural language generation, translation, and comprehension. In ACL, 2020.
Dongxu Li, Junnan Li, Hongdong Li, Juan Carlos Niebles, and Steven CH Hoi. Align and prompt: Video- and-language pre-training with entity prompts. In Proceedlings of the IEEE/CVF Conference on Compuler Vision and Pattern Recognition (CVPR), pp. 4953 4963, 2022a.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems, 34, 2021a.
unified vision-language understanding and generation. arXiv preprinl arXiv:2201. 12086, 2022b. Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. arXiv preprint arXhv:2001 2023.
Linjie Li, Zhe Gan, Kevin Lin, Chung-Ching Lin, Zicheng Liu, Ce Liu, and Lijuan Wang. Lavender: Unilying video-language understanding as masked language modeling. arXi preprint orXiv:2206.0700 07166, 2022c.
Liunian Harold Li, Mark Yalskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. Visualbert: A simple and performant baseline for vision and language. orXiv preprint arXiv: 03557, 2019.
Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wa, and Haifeng Wang. Unimo: Towards unified-modal understanding and generation via cross-modal contrastive learning, arXic preprint arXiv:2012.1544 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020b.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language- image pre-training paradigm. arXie preprint XrXiv:21 10. 05208, 2021b.
Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. Jointly localizing and describing events for dense video captioning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 7492-75000 7500, 2018.
Sheng Liang, Mengjie Zhao, and Hinrich Schütze. Modular and parameter-cfficien multimodal fusion with prompting. Tn Findings of the Association for Compulational Linguistics: ACL 2022, pp. 2976 2985, 2022.
18

---

Published in Transactions on Machine Learning Research (12/2023)
Kevin Lin, Linjie Li, Chung- Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 17949 17958, 2022.
Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Europewn conference on computer vision, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning. ar preprint OrXiv2304.084888 2023a.
Xubo Liu, Xinhao Mei, Qiushi Huang, Jianyuan Sun, Jinzhong Zhao, Haohe Liu, Mark D Plumbley, Volkan Kilic, and Wenwu Wang. Leveraging pre-trained bert for audio captioning. In 2022 30th European Signal Processing Conference (EUSTPCO), pp. 1145-1149. IEEE, 2022.
Yiheng Liu, Tianle Han, Siyuan Ma, Jiayue Zhang, Yuanyuan Yang, Jiaming Tian, Hao He, Antong Li, Mengshon He, Zhengliang Liu, et al. Summary of chatgpt/gpt-4 rescarch and perspective towards the future of large language models. arXie preprinl urXi::230,01882. 2023b.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for vision-and-languaag tasks. Advances in neural information processing systems, 32, 2019.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. arXiv preprint arXiv:2206. 220.0220000816, 2022a.
Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song- Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. In NewrIPS, 2022b.
Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXio:2002.066 06353, 2020.
Oscar Mañas, Pau Rodriguez, Saba Ahmadi, Aida Nematzadeh, Yash Goyal, and Aishwarya Agrawal. Mapl: Parameter-efficient adaptation of uninodal pre-trained models for vision-languaae few-shot prompting. arXiv preprint orXi:22210.07172 2022.
Kenneth Marino, Mohammad Rastegari, Ali Farhadi, and Roozbeh Mottaghi. Ok-vqa: A visual question answering benchmark requiring external knowledge. In CVPR, 2019.
Michael Matena and Colin Raffel. Merging models with Fisher-weighted averaging. In NewrIPS, 2022.
Xinhao Mei, Xubo Liu, Qiushi Huang, Mark David Plumbley, and Wenwu Wang, Audio captioning transformer. In Workshop on Detection and Classification of Acoustic Scenes and Events (DCASE 2021), 2021.
Jack Merullo, Louis Castricato, Carsten Eickholf, and Ellie Pavlick. Linearly mapping from image to text. space. arXiv preprint orXiv:2209.11622 2022.
Margaret Mitchell, Simone Wu, Andrew Zaldivar, Parker Barnes, Lucy Vasserman, Ben Hutchinson, Elena Spitzer, Inioluwa Deborah Raji, and Timnit Gebru. Model cards for model reporting, In Proceedings of the conference on fairness, accounlabilily, and transparency, 2019.
Behnam Neyshabur, Hanie Sedghi, and Chiyuan Zhang. What is being transferred in transfer learning? NewrIPS, 2020.
Alexander Quinn Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob Megrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. Tn TCML, 2022.
19

---

Published in Transactions on Machine Learning Research (12/2023)
Vicente Ordonez, Girish Kulkarmi, and Tamara L Berg. Im2text: Describing images using million captioned photographs. In Proceedings of the 24th International Conference on Neural Information Processing Systems, NIPS'11, pp. 1143 1131,,,,,1111 Red Hook, NY, USA, 2011. Curran Associates Inc. ISBN 9781618395993.
Guillermo Ortiz-Jimencz, Alessandro Favero, and Pascal Frossard. Task arithmotic the tangent space: Improved editing of pre-trained models. arXiv preprint arXiv:2305.12877 2023.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8788 8763. PMLR, 2021.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXie preprint 0rXi:2111211444 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-tert transformer. The Journal of Machine Learning Research, 21(1):5485 5551, 2020.
Tanzila Rahman, Bicheng Xu, and Leonid Sigal. Watch, listen and tell: Multi-modal weakly supervised dense event captioning, In Proceedings of the IEEE/CVE international conference on computer vision, pp- 1908-8917) 2019.
Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gallinari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. In NewrIPS, 2022.
Alexandre Rame, Kartik Ahuja, Jianyu Zhang, Matthieu Cord, Téon Bottou, and David Lopez-Paz. Model Ratatouille: Recycling diverse models for out-of-distribution generalization. In ICML, 2023a.
Alexandre Rame, Guillaume Couairon, Mustafa Shukor, Corentin Dancette, Jean-Baptiste Gaya, Laure Soulier, and Matthieu Cord. Rewarded soups: towards pareto-optimal alignment by interpolating weights fine-tuned on diverse rewards. arXiv preprint arXiv:2306. 04440,8060004888,,, 2023b.
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scout Gray, Chelsea Voss, Alec Radford, Mark Chen, and Tly Sutskever. Zero-shot text-to-image generation. In ICML, 2021.
Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth- Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. A generalist agent. arXiv preprint arXiv:2205.061 2022.
Anna Rohrbach, Lisa Anne Hendricks, Kaylee Burns, Trevor Darrell, and Kate Saenko. Object hallucination in image captioning. In EMNLP, 2018.
Tim Salimans, Tan Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. NewrIPS, 2016.
Christoph Schuhmann, Richard Vencu, Romain Beaumont, Robert Kaczmarezyk, Clayton Mullis, Aarush Katta, Theo Coombes, Jenia Jitsev, and Aran Komatsuzaki. Laion-400m: Open dataset of clip-filterdd 400 million image-text pairs. arXie preprint or Xiv:211 11112111.111.11 14, 2021.
Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. End-to-end generative pretraining for multimodal video captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition (CVPR), pp. 17959 17968, 2022.
Piyush Sharma, Nan Ding, Scbastian Goodman, and Radu Soricut. Conceptual captions: A cleaned, hypernymed, image alt-lext dataset for automatic image captioning. In ACL, 2018.
Mustafa Shukor, Guillaume Couairon, and Matthieu Cord. Efficient vision language pretraining with visual concepts and hierarchical alignment. In 33rd Brilish Machine Vision Conference (BMVC), 2022.
20

---

Published in Transactions on Machine Learning Research (12/2023)
Mustafa Shukor, Corentin Dancette, and Matthieu Cord. ep-alm: Efficient perceptual augmentation of language models. orXiv preprint Xrvv:230031 1 1400, 2023a.
Mustala Shukor; Alexandre Rame, Corentin Dancelte, and Matthieu Cord. Beyond task performance: Evaluating and reducing the flaws of large multimodal models with in-context learning. arXiv preprint arXiu:2310.00647 2023b.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision aligument model. In Proceedings of the IEFE/CV Conference on Computer Vision and Pattern Recognition (CVPR), pp. 15638-1565) 2022.
Tejas Srinivasan, Xiang Ren, and Jesse Thomason. Curriculun learning for data-efficient vision-language aligmment. arXiv preprint arXiv:2207114525, 2022.
Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang- Yan Gui, Yu-Xiong Wang, Yiming Yang, et al. Aligning large multimodal models with factually auginented rlhf. arXiv preprint arXiv:2309. 14525, 2023.
Yi-Lin Sung, Linjie Li, Kevin Lin, Zhe Gan, Mohit Bansal, and Lijuan Wang. An empirical study of multimodal model merging. arXio preprint arXim:2304. 14933, 2023.
Daiki Takeuchi, Yuma Koizumi, Yasunori Ohishi, Noboru Harada, and Kunio Kashino. Effects of word- frequency based pre-and post-processings for audio captioning. arXiv preprint or 1XX::2009.14436, 2020.
Mingkang Tang, Zhanyu Wang, Zhenhua Liu, Fengyun Rao, Dian Li, and Xiu Li. ClipAcaption: Clip for video caption. In Proceedings of the 29th ACM International Conference on Multimedia, pp. 4858 4862, 2021.
Yi Tay, Mostafa Dehghani, Vinh Q Tran, Xavier Garcia, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Neil Houlsby, and Donald Metzler. Unifying language learning paradigms. arXiv preprint trXiv:2205.05134 2022.
Hugo Touvron, Matthicu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and Hervé Jégou. Training data-efficient image transformers & distillation through attention. In International Conference on Machine Learning, pp. 10347-10357. PMLR, 2021.
Hugo Touvron, Thibaut Lavril, Gautier lzacard, Xavier Martinet, Marie- Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2902. 13971, 2023.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in newral information processing systems, pp- 5998 6008, 2017.
Jianseng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. GIT: A generative image-to-text to-text transformer for vision and language. Transactions on Machine Learning Research, 2022a. ISSN 2835-8856. URL https::/ net//oorummddddda//M0000
Jingwen Wang, Wenhao Jiang, Lin Ma, Wei Liu, and Yong Xu. Bidirectional attentive fusion with context. gating for dense video captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 1190-7198 2018a.
Jinpeng Wang, Yixiao Ge, Rui Yan, Yuying Ge, Kevin Qinghong Lin, Satoshi Tsutsui, Xudong Lin, Guanyu Cai, Jianping Wu, Ying Shan, et al. All in one: Exploring unified video-language pre-training, In Proceedings of the IEEE/CVP Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6508-6608, 2023a.
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gan Ga-G Jiang, and Lu Yuan. Omnivl: One foundation model for image-language and video-language tasks. Xiv preprint or Xix:2209.007266 2022b.
21

---

Published in Transactions on Machine Learning Research (12/2023)
Liyuan Wang, Xingxing Zhang, Hang Su, and Jun Zhu. A comprehensive survey of continual learning: Theory, method and application. arXiv preprint or arXiv:2302.004 00487, 2023b.
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through & simple learning framework. In ICML, 2022c.
Thomas Wang, Adam Roberts, Daniel Hesslow, Teven Le Scao, Hyung Won Chung, Iz Beltagy, Julien Launay, and Colin Raffel. What language model architecture and pretraining objective works best for zero-shot generalization? In ICML, 2022d.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image foreign language: Beit pretraining for all vision and vision-language tasks. 2022c.
Xin Wang, Yuan-Fang Wang, and William Yang Wang. Watch, listen, and describe: Globally and locally aligned cross-modal attentions for video caplioning. In Proceedings of the 2018 Conference of the North American Chopter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), pp. 795-801, New Orlcans, Louisiana, June 2018b. Association for Computational -125
Linguistics. doi: (0.18853/v1/// /v//N18-2122 URL
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint orXiv:2108. 10904, 2021.
Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roclofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. In ICML, 2022.
Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. Nüwa: Visual synthesis pre-training for neural visual world creation. In ECCV, 2022.
Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine- ane-grained image 2019.
understanding, arXie preprint
Dejing Xu, Zhou Zhao, Jun Xiao, Fei Wu, Hanwang Zhang, Xiangnan He, and Yueting Zhuang. Video question answering via gradually refined attention over appearance and motion. In Proceedings of the 25th ACM International Conference on Multimedia, MM '17, pp. 1645-1653, New York, NY, USA, 2017. Association for Computing Machinery. ISBN 9781450349062. doi: 10. 10.145/3122296.312322 URL btttp://doo..rgg: 10. 1111...1111113226633333322 31234477
Jun Xu, Tao Mei, Ting Yao, and Yong Rui. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), June 2016.
Xuenan Xu, Heinrich Dinkel, Mengyue Wu, and Kai Yu. A crnn-gru based reinforcement learning approach to audio captioning. 2020.
Xuenan Xu, Heinrich Dinkel, Mengyue Wu, Zeyu Xie, and Kai Yu. Investigating local and global information for automated audio captioning with transfer learning. In ICASSP 2021-2021 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 005-909. TEEE, 2021.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multiintrruct: Improving multi-modal zero-shot learning via instruction tuning. arXiv preprint (xXx:2211..0071 2022.
Pratoek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. Resolving interference when merging models. Xin preprint ar 2006:22306.01708, 2023.
22

---

Published in Transactions on Machine Learning Research (12/2023)
Antoine Yang, Antoine Miech, Josef Sivic, Ivan Laptev, and Cordelia Schmid. Just ask: Learning to answer questions from millions of narrated videos. In Proceedings of the TEEE/CVF International Conference Computer Vision, pp. 1686 1697, 2021a.
Antoine Yang, Antoine Miech, Josef Sivie, Ivan Laptev, and Cordelia Schmid. Zero-shot video question answering via frozen bidirectional language models. In NewrIPS 2022-36th Conference on Neural Information Processing Systems, 2022.
Zhengyuan Yang, Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Crossing the format boundary of text and boxes: Towards unified vision-language modeling. arxiv preprint ar Xiv:21111. 12085,, 2021b.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are imago-text foundation models. arXiv preprint "XXi:2205.00999 2022.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context in referring expressions. In European Conference on Computer Vision, pp. 69 85. Springer, 2016.
Lu Yuan, Dongdong Chen, Yi-Ling Chen, Noel Codella, Xiyang Dai, Jianfeng Gao, Houdong Hu, Xuedong Huang, Boxin Li, Chunyuan Li, et al. Florence: A new loundation model for computer vision. arXiv preprint xrXie:2111.11440 11432, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310-12320. PMLR,
2021.
Rowan Zellers, Ximing Lu, Jack Hessel, Youngjae Yu, Jae Sung Park, Jize Cao, Ali Farhadi, and Yejin Choi. Merlot: Multimodal neural script knowledge models. Advances Neurol Information Processing Systems, 1123634-23651. 2021.
Rowan Zellers, Jiasen Lu, Ximing Lu, Youngjae Yu, Yanpeng Zhao, Mohammadreza Salehi, Aditya Kusupati, Jack Hessel, Ali Farhadi, and Yejin Choi. Merlot reserve: Neural script knowledge through vision and language and sound. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 16375 16387, 2022.
Pongchuan Zhang, Xiujun Li, Xiaowci Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEFE/CV Conference on Computer Vision and Pottern Recognátion CaciiiionnCPP (CVPR), pp. 5579,5588, 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language modcls. arXic preprint ar 2iv:
2022.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, et al. Siren's song in the ai ocean: A survey on hallucination in large language models. arXiv preprint arXio:2399001233 2023.
Guangxiang Zhao, Wenkai Yang, Xuancheng Ren, Lei Li, Yunfang Wu, and Xu Sun. Well-classffied examples are underestimated in classification with deep neural networks. In AAAI, 2022.
Xueyan Zou, Zi-Yi Dou, Jianwei Yang, Zhe Gan, Linjie Li, Chunyuan Li, Xiyang Dai, Harkirat Behl, Jianfeng Wang, Lu Yuan, al. Generalized decoding for pixel, image, and language. In CVPR, 2023.
23

---

Published in Transactions on Machine Learning Research (12/2023)
Appendix
The Appendix is organized as follows:
Section model card.
Section B
detailed discussion about related work.
Section background on unified models and different unification
Section details about model architecture.
Section B
image and video-text pretraining tasks.
Section
illustration and details about multimodal curriculum learning.
Section
datascts and implementation details.
Section
finctuning only the lincar conncction (Paramctor- Finctuning).
Section
ablation study including knowledge transfer across modalities and training efficiency.
Section
additional quantitative results.
Section
K
discussion of several limitations and future directions.
Section qualitative results of several image-text tasks.
A
Model Card
In the following table, we detail our model card Mitchell et al. 2019
Model Details
Model Date
July 2023
Model Type
Transformer encoder-decoder pretrained on text and trained end-to-end be conditioned on image, video and audio input. Modality-specifie encoders based convnets and pretrained from classification public benchmarks. All input tokens are concatenated and fed to the encoder. The text generation is conditioned on other modalities via cross-attentes (See Section for details.)
Intended Uses
Primary Intended Uses
The primary use is research unified multimodal models that span wide range of applications such as; image/ image/videdt rideo/audio captioning, image/video question answering, grounding srounding/detectioon detectioo and image generation. In addition, the study of the limitation and biases such kind of model, and novel approach for efficient training and adaptation. Other similar multimodal applications ions can also be considered, like multimodal dialogue, and text- gext-guided robotics applications.
Primary Intended Users
The research community. The model will be made public.
24

---

Published in Transactions on Machine Learning Research (12/2023)
Out-of-Scope Uses
Any downstream applications that can cause harm to society, or without mitigation of associative safety measures.
Factors
Card Prompts Relevant Factor
The model is trained on english and based on BART (Lewis et al. 2020) language model. The model should not be used any downstream application without propoer factor analysis.
Card Prompts Evaluation Factors
The model inherits the biases and risks of the pretrained language model (Lewis et al. 2020). It may also hallucinates some infor- mation not present in the conditioned modality. On some tasks we constraints the text generation predifined set of answers, however, generally, there is mechanism that force it to not produce toxic or racist oulput on all tasks.
Metrics
Model Performance Measures
The performance using standard metrics to evaluate the model per- formance on several public benchmarks, such as; Visual Question Answering (accuracy on VQAv2, OKVQA, VizWiz, MSVD-QA and MSRVTT-QA) Visual Grounding (IoU>0.5 on Re(COCO, Re- fCOC0+ and RefCOCOg), Image Captioning (CIDEr, METEOR, BLEU, SPICE on MSCOCO, MSR-VTT, Audiocaps and Clotho vl) and Text to Image Generation (CLIP score on MSCOCO).
Decision thresholds
N/A
Approaches to Uncertainty and Vari- ability
The relatively costly pretraining prevent from doing several runs, however the different ablation study and the evaluation on many datasets validate the overall performance of the model.
Evaluation Data
Datascts
Check Tab. 15 for more details.
Motivation
The datasets span different standard benchamrks across image, video and audio modalities. This show the overall capability of the model to process different modalities.
Preprocessing
Text is process with BPE tokenizers, audio is transformer to mel spectorgram and we randomly sample some frames from videos. Some addition data augmentation techniques are used during training.
Training Data
25

---

Published in Transactions on Machine Learning Research (12/2023)
Datasets
We only use public datasets, such as image captioning (COCO
(Lin et al. 2014), Visual Genome (VG) (Krishna et al. 201 7b)
SBU (Ordonez et al. 2011 CC3M (Sharma et al. 2018 ) and
CC12M (Changpinyo et al.
2021 (only in the first stage)). VQA
(VQAv2 GGoyal et al. 2017):
7) GQA (Hudson & Manning 2019)
VG (Krishna al.) 2017b)): Visual Grounding (VGround) and
referring expression comprehension (RelCOCO, RefCOCO+, Ref-
COCOg Yuu et al. 2016)), video captioning (WebVid2M (Bain et al. 2021 1 and video question answering (WebVidQA (Yang
et al. 2021a We only use the training sets during pretraining.
Quantitative Analyses
Unitary Results
Our unified model is competitive state of the art approaches customized for less modalities. It attains state of the art results on Visual Grounding and Audio Captioning. Please check Sec 4.1 for more details.
Intersectional Results
N/A.
Ethical Considerations
Data.
We only public benchmarks, however benchmarks are not filtered from racist, sexist or otherwise harmful content.
Human Life
The model is not intended to be used for safety critical applica- tions.
Mitigations
Constrained text generation can be adapted for some tasks. How- ever, for open-ended generation post processing or some enginecred prompts might mitigate some the biases. Overall, filtering the pretraining data can be ver effective approach.
Risks and Harms
We use public datascts. Not all of them are filtered from from toxic and personal data.
Use Cases
Forcing the model (finetuning prompting) to generate harmful or racist text. Other cases regarding general language models are also relevant.
Table 14: UnIVAL Model Card, We follow the framework of Mitchell al. 2019
B
Related Work
Unimodal pretraining Pretraining on large uncurated datasets has been substantial ingredients in the vision and NLP communities develop powerful models that generalize to wide range of tasks. For vision models, supervised (Tourron et al., 2021 Dehghani et al., 2023) and self supervised (Chen et al. 2020b)
Caron et al. 2020 Zbontar et al. 2021 He et al.) 2022 techniques have extensively investigated, while for
NLP, the widely used training objective is next token prediction (Brown et al. 2020
Hoffmann et al. 2022
Touvron et al. 2023
Recently, these domains started to converge on a simple training paradigm; joint scaling the pretraining data, model size and compute, while using unified architecture and training objective. Surpassing a certain
scaling threshold has elicited new emergent capabilities, especially in LLMs (Brown et al.) 2020 Chowdhery
et al. 2022). that allows such models to solve new reasoning tasks that were out of reach few years ago. Once
26

---

Published in Transactions on Machine Learning Research (12/2023)
such models are available, they can be scamlessly adapted without retraining, via prompting such zero-shot or few-shot In Context Learning. Scaling vision transformer models (Dehghani et al. 2023) lead to be more robust and aligned to human object recognition.
While being very successful, training such models is hard, extremely costly and need dedicated infrastructure. However, the public release of many of these models allow to leverage them for variety of tasks. In this work we leverage unimodal pretrained models for multimodal tasks.
Multimodal pretraining. So far, most of the effort to build multimodal models have been focused on vision-language pretraining. Contrastive based approaches (Radford et al. 2021 Jia et al.) 2021) try to learn shared and aligned latent space by training on hundred of millions of data. More data efficient approaches
(Shukor et al. 2022: Li et al. 2021a 2022b;
Dou et al.) 2021: Singh et al.) 2022), have relied on additional
multimodal interaction modules and variety of training objectives such as image-text matching, masked language modeling and image-text contrastive (Chen et al., 2020c Kim al.
2021 Lu et al.,
2019 Zhang
2021 In the video-language community, similar approaches have been mildly adapted to model the et al.
interaction between language and frames sequences (Cheng et al. 2022 Wang et al.
2023a: Fu et al. 2021
Zellers et al.
2021 Yang et al. 2021a). Few work have targeted both image and video language pretraining 2022b
Wang et al.
These works have been following the scaling trend as unimodal pretraining. Scaling the model went from
couple of billions of parameters YYu et al. 2022 Wang et al. 2022e a) to tens of billions (Chen et al., 2022b
Alayrac et al. 2022).
Unified models Building unified systems has been triggered first in the NLP community. (Raffel et al, 2020) proposed the T5 transformer model, a text-to-text framework, where the same pretrained model is used to solve many NLP lasks, each one described by task-specific textual prefix. Since then, building general textual models has been heavily investigated by LLMs (Brown et al. 2020: Raa et al., 20211 Chowdhery et al.) 2022). The success of unified Language models, have inspired other communities.. In the vision community, (Chen et al. 2022a) proposed pixel-to-sequence framework to unify different vision tasks such as object detection and instance segmentation. For multimodal tasks, (Cho et al.) 2021) proposed to unify vision-language tasks, including discriminative ones, as conditional text generation. This was followed by Yang et al. 2021b), which targets also grounded tasks and does not rely on an object delection model. OFA Wang et al. 2022ch then proposed a large scale to-seqeece framework, and extended previous approaches to more imago-text tasks, including text to image generation. Similarly, Unificd-OO (Lu al., 2022a), in addition to image-lext lasks, largets many visual tasks including dense prediction such as depth estimation and image segmentation. The most closest to us is the work of OFA and Unified-10, however, we propose to unify tasks across many modalities, and use smaller model and dataset sizes.
Efficient multimodal learning The current paradigm in training multimodal models is to train all model parameters, even when using pretrained models (Chen et al. 2022b Wang et al.] 2022c Li et al. b022b). Despite attaining SoTA, these approaches are extremely costly to train. To overcome this, recent approaches showed that pretrained models, generalize well to multinodal tasks, where it is possible to use a frozen LM with powerful multimodal encoder such as CLIP, and train only handful of parameters, such as the vision encoder (Eichenberg et al. 2021). the vision connector (Merullo et al.] 2022 Mañas ct al. 2022 Koh et al. 2023 Li et al. 2023) or additionally the Adapters (Eichenberg et al. 2021 Yang et al. 2022). This paradigm was then generalized in (Shukor et al.) 2023aa,, , to other modalities, such video and audio, where the authors showed that is even possible train only a linear projection layer to adapt pretrained unimodal encoder (e.g.,
pretrained on ImageNet) and a language decoder to do multimodal tasks.
Another line of research, is data-efficient approaches, recent work shows that it is possible to get comparable results by training on significantly less data, by designing better training objectives (Shukor et al. 2022) ) data augmentation (Li et al. 2021b) and curriculum learning (Srinivasan et al. 2022). In this work, we focus on parameter-efficient finetuning, especially, training only the linear connection.
Weight interpolation and mutltimodal tasks. Our strategy enable the training of multiple expert models with diverse specializations. To combine them, we leverage simple yet practical strategy: linear
27

---

Published in Transactions on Machine Learning Research (12/2023)
interpolation in the weight space, despite the non-linearities in the network's architecture. This weight averaging (WA) strategy is in line with recent findings on linear mode connectivity (Frankle et al. 2020
Neyshabur et al. 2020] LMC): weights fine-tuned from a shared pre-trained initialization remain linearly
connected. Recent works (Ilharco et al,, 2023 Dahcim et al. 2023 Ortiz-Jimencz et al. 2023) suggest
that averaging networks in weights can combine their abilities without any computationa overhead; for
instance, the average of an English summarizer and an English-to-French translator will behave as French sunmarizer (Jang et al.; 2023). Model soups approaches (Wortsman et al., 2022: [ame al., 2022) improve out-of-distribution generalization and show that weight averaging actually approximates predictions averaging Lakshminarayanan et al. 2017) when the LMC holds. The LMC was extended to weights fine-tuned with
different losses (Rame et al. 2022 Croce et al.; 2023 Rame et al. 2023b) or on different datasets Mateea
& Raffel 2022 Ilharco et al.
2022 Choshen et. al. 2022]
Don- Yehiya el al., 2023 Rame et al. 2023a ?).
Moreover, several other merging approaches (Matena & Raffel
Yaday et al. 2023) have been proposed, linear interpolation. For example, (Matena &
2022
though with arguably minor empirical gains over the simpler
resolve updates conflicts across weights. Neuron permutations strategies (Entezari et al. 2022 Ainsworth al. 2022 Jordan et al. 2023 address the
Raflel 2022 considers the Fisher information; (Yadav et al.
2023
ambitious challenge of enforcing connectivity across weights with different random initializations, though so far with moderate empirical results. Most of exisiting WA approaches consider very similar tasks, such as image classifications from different datasets or text classification/ ggeneration. Interpolating weights of models finetuned on different multimodal tasks, is little investigated, with no work exploring this technique in multimodal foundation models. The most similar and concurrent work is the recent Sung et al. (2023) applying a complex architecture--peeiffc merging strategy involving weight averaging for models pretrained different modalities. Another difference, is that we explore WA for multimodal downstream tasks.
iaaasffiattionggenraaiiionn
C Unified Foundation Models: 4 Unification axes
While many previous works have attempted to build unified models, they still have some customization in terms of architectures and tasks. Our work tries to unify most aspects of the model, following recent line of work [Wang et al. 2022c). In the following, detail the 4 unification axes that distinguish our work from previous ones.
Unified input/output. To have a unified model, it is important to have the same input and output format. across all tasks and modalities. The common approach is to cast everything to sequence of tokens as in language models. Multimodal inputs, such as images, videos and audios can be transformed to tokens by patchifying or using shallow mopelity-speeific projections. Multimodal outputs can also be discritized, by using VQ-GAN for images and discrete pixel locations for visual grounding. A unified vocabulary is used when training the model.
Unified model. The unified input/outppe representation allows to use a single model to solve all tasks, without the need to any adaplation when transitioning from the pretraining to the finetuning phase (e.g., no need for task-specific heads). In addition, the current advances in LLMs, especially their generalization to new tasks, make good choice to leverage these models to solve multimodal tasks. The common approach is to have language model the core model, with light-weight modality-specifie input projections.
Unified tasks. To scamlessly evaluate the model new unseen tasks, it is essential to reformulate all tasks in the same way. For sequence-to-sequene frameworks, this can be done via prompting, where each task is specified by particular textual instruction. In addition, discriminaive tasks can be cast to generation ones, and thus having only sequence gencration output.
Unified training objective. Due the success of next token prediction in LLMs, it common to use this objective to train also unified models. An alternative, is to use an equivalent to the MLM loss. The same loss is used during pretraining and finetuning.
28

---

Published in Transactions on Machine Learning Research (12/2023)
foutball
Decoder
Encoder
000
CA
SA
SA
are some guys playing in a ground?
N
X
Figure 4: UnIVAL architecture. We use a typical encoder-decoder transformer, in addition to weight CNN-based modality encoders.
D Model Architecture
To tackle multimodal tasks at small to mid-scale, we employ an encoder-decoder LM (Vaswani et al, 2017 Lewis et al., 2020) (shown in 4)g as its effectiveness for multimodal tasks has been demonstrated compared to decoder-only models (Wang et al. 2021), and their superiority in zero-shot generalization after multitask training (Wang et al. [022d]. The encoder consists of stack of blocks of Self-Attention (SA), Layer Normalization (I.N), GELU activations and Feed Forward Network (FFN) layers. The decoder blocks contains additionally cross-attention (CA) layers to attend to the encoder last layer tokens. Specifically, the output tokens of the encoder are considered keys and values in the CA, while the text generated in the decoder is considered as queries. Following other approaches (Wang et al. 2022c), and to stabilize the training, we add LN layers after the SA and the FFN, and head scaling to the SA. We use independent absolute and relative position embeddings for text, images, videos and audios. We add different modality token embeddings distinguish text. from other modalities. The model parameters are initialized from BART-base model (Lewis et al. 2020
For each modality, we use fight-weight convolution architectures (e.g., the encoders in orange and green in 4. For images, we follow other work (Wang et al.
2022ch and use ResNet-101 trained on ImageNet. For videos, we use 3D ResNext-101 (Hara et al.) 2018a trained on Kinetics-400 (Kay et al. 2017), and for audio, we use PANN-CNN14 (Kng et al.] 2020
2021
trained on AudioSet (Gemmeke et al. 2017). We do not
skip the last block in the encoders (Wang et al. 2022c), as we find that it reduces the number of tokens and accelcrate the training (sce Tab
Each modality is encoded in the modality projection (for text we linear embedding layer), and then concatenated to form sequence of tokens (e.g., textual and visual) before being passed to the encoder (for some tasks such as VQA, we pass also the question to the dccoder). After encoding, the output of the encoder interact. with the decoder via cross-allention. The decoder generates the response auto-regressively starting from a special BOS token.
E Pretraining Tasks
We pretrain UnIVAL on the following
tasks:
Image Captioning. The model takes as input an image and 'what does the image describe?" as text and generate textual description of the image.
29

---

Published in Transactions on Machine Learning Research (12/2023)
Visual Question Answering (VQA). The model takes input an image and question and generates textual answer based on the image.
Visual Grounding (VGround.). The model takes an image and "Which region does the <lext> describe?" as text and the model generates the coordinates of the bounding box described by the <text>.
Grounded Captioning (GC). This is similar to image captioning, but the model should generate a description of specific region in the image. Specifically, the model takes image and 'what. does the region describe? region: <xl, yl, x2, y2> text and generates a caption of the region. <xl, yl, x2, y2> are coordinates of the region bounding box.
Image-Tess Text Matching (ITM). The model takes an image and a text and should predict the text corresponds to the image. For a given image we randomly sample a caption as negative text and consider the original caption as positive. The input text "Does the image describe <lext>?' and the output either "Yes" or 'No".
Video Captioning. Similarly to image captioning, the model takes video and "what does the video describe?" and generates video description.
Video Question Answering (VideoQA). The model takes a video and question and should answer the question based on the video.
Video-Texx Matching (VTM). The model should predict if text corresponds to given video or not.
Stage
Stage 2
Stage
Stage N
Text data
Text data
Image data
Text data
Image data
Video data
Modality agnostic models
0
LM
ILM
IVLM (r.p: UnIVA))
Modality:
Modalities:
Typical
Modalities: LM trained on
3 Modalities:
ILM is treined on image-text and video toxt data
model trained on the new modality and all
training
image-lext heee-ltxtteeeeet data
corpus
N-1
Figure 5: Multimodal Curriculum Learning. We pretrain UnIVAL in different stages. (1) The first pretraining is a typical training for language models on corpus of text. (2) Then, the model is trained on image and text data to obtain an Image-Language Model (ILM). (3) In the third stage, the model is trained additionally on video-text data obtain a Video-Image- Sarragg--anguggg-MMon (VILM). To obtain modality agnostic models the model should be trained on many modalities. Following this setup, UnIVAL can be used to solve image/video/ auxio-text tasks.
F Multimodal Curriculum Learning
Training on many tasks and modalities is computationally expensive, especially when considering long videos ore audios. To overcome this, propose multistage curriculum training approach (depicted in Fig in which we progressively add more modalities. In stage 1, the model is trained on large corpus of text following typical next loken prediction or other LM training. Thanks to the many open sourced pretrained language models, it is easier to leverage and initialize from existing LMs (e.g., BART (Lewis et al., 2020) as in our case). In stage 2, the model trained on many tasks of images and texts. Afterwards, vidco-text datasets are added and the model is trained on both image-text and video-text data. This is general paradigm efficiently train multimodal models on many modalities. Training many tasks is more efficient, however,
30

---

Published in Transactions on Machine Learning Research (12/2023)
the standard training image-text alignment image captioning can be also considered. Note that, to keep good performance on unimodal tasks, it is better to add also unimodal data.
While this training scheme is more efficient than training on all data from the beginning, using more efficient approaches from the contimual learning community (Wang et al. 2023b) is extremely useful in this context, limit the number of examples as we add more modalities, especially if the objective is obtaïn modality agnostic models. Training only on the new modalities will make the model forget about previous ones.
G Data and Implementation Details
Table 15: Downstream Lasks and datasels. We show the size of different splits used in our work.
Dataset
Modality
Size (Train/Val/Tooe /Val/Test)
COCO
Lio et al.
2014
Image-Tean
Image Captioning
113K/5K/GK
Rоcарe
2019
Image-Tehe Text
Image Captioning
/1.K/
VQAv2
Guval et al
201
Image-Tess
VQA
443K/21 4214//22//// /15KK
OKVQA
Marino el
2019
Imigee Text
VQA
-/KK/-
VizWix
Guram e5 al
Image-Text Text
VQA
- -4..3K
SNLI-VE
X3e al:
2011
Image-Tu Text
Visual Entailment
30K/ /IK/KKKKKKK
RefCOCO
Yu
2016
Image-Tes Text
Visual Grounding
120K//KK /GK/
RefCOCO
Yu et
2016
Image-Test Text
Visual Grounding
120K/GK BKKK/GK/SK
RefCOCOg
Yu et ml
2016
Image-Texd
Visual Grounding
80K, /KK, '10K
COCO [Lin ot ol. 2014
Image Text
Texx
Text to Imaaee Generation
80K/64K /30K
MSR-VTT Xu et al.
2016
Viden- Viden-Text
Video Captioning
L.5K/O.5K/3K /3K
ActivityNet-Caption KKishha 05
2017a
Video- Text
Video Captioning
37.5K/ /17K
MSRVTT-QA Xu
2017
Video-Text
VideoQA
156K 12K /7KK
MSVD-QA Xu el al.
Video-Test Text
VideoQA
30K/6K/122
Andiocags Kim
2019h
Audio-Tess
Audio Captioning
47K /0.5K//KKK IIK
Clotho vl
Trossos ml. 2020
Audio Text
Audio Captioning
17.5K /1K/-
G.1 Implementation details of downstream tasks.
For image-lext lasks, we keep the hyperparameters during finetuning close to those in OFA (Wang el al. 2022ch. The downstream datasets are detailed in Tab 15
VQA. We finetune VQAv2 dataset and cast the task as text generation. The model is trained for 5 epochs with a batch size of 256 using Adam optimizer. We use a learning rate of le - 4 with linear decay and label smoothing of 0.1. The image resolution is increased to 480 and we use expomential moving average with 0.9999 decay. We use Trie based search to constraint the generated answers to the top 3.1k answers. We freeze the encoder and decoder embeddings during finetuning. The question passed both the encoder and decoder as prompt.
Image Captioning. We finetune on MSCOCO karpathy split and report standard captioning metrics. The model is trained for 4 epochs with a batch size of 128. The image resolution is set to 480 and the learning rate to le - 5 with linear decay. We use an encouraging (Zhao el al. 2022) cross entropy loss with label smoothing of 0.1. We freeze the encoder and decoder embeddings during finetuning.
Visual Grounding. We finetune on RefCOCO, RefCOCO+ and RefCOCOg for 8 epochs with batch size of 256. The images are resized to 512 and the learning rate start with 5e - 5 and decreases lincarly. We train with cross entropy and label smoothing of 0.1. We limit the generation length to 4 and report the 5.c@0.5.
Visual Entailment, The model trained for 4 epochs with batch size of 256 and learning rate of 5e-5 that deacrcases lincarly. The image resolution is set 480. The model takes only the image and the text hypothesis, without the text premise, and the generation is constrained to yes ges/maybe/no using Shie- search. The text. is passed to both the encoder and decoder as prompt.
31

---

Published in Transactions on Machine Learning Research (12/2023)
VideoQA. The model is trained for 25 epochs MSRVTT-QQ QA and 40 epochs MSVD-QA with batch size of 128 and learning rate of le - 4 that decreases linearly. We sample randomly 8 frames with resolution 384. We train with cross entropy with encouraging loss and label smoothing of 0.1. We use exponential moving averaging model pass the question to both the encoder and the decoder. The answer generation is constrained to the set of possible answers via Trie-based search. We freeze the encoder and decoder embedding layers.
Video Captioning. We train on MSR-VTT for 15 epochs and a batch size of 256 with a starting learning rate of le- that decreases linearly. We randomly sample 16 frames with resolution 384 and train with encouraging cross entropy loss and label smoothing of 0.1. We freeze both the encoder and the decoder embedding layers.
Audio Captioning. We train for 10 epochs on Audiocaps and Clotho v1 with batch size of 128 and starting learning rate of le -4 4 (5e - 5 for clotho vl). The mel bins is set to 64 and the hop size to 200. We train with encouraging cross entropy loss with label smoothing of 0.1 and freeze the encoder and decoder embedding layers.
Text-to-Image Generation. We follow previous work [Wang et al. 2022e and finetune the model on the train set of MSCOCO and evaluate on 30K images from its validation set. We use the discrete tokens of VQ-GAN that are then passed to frozen VQ-GAN decoder to generate the images. We start by training with cross-entropy loss for 50K steps and batch size of 512 (~ 60 epochs) and Ir 1e-3, followed by CLIP score optimization for 5K steps and batch size of 128 and Ir 1e-6. When evaluating the model we select the best image, among 24 generations based on CLIP score. We report Inception score (IS) (Salimans et al. 2016). Fréchet Inception Distance (FID) Heusel et. al. 2017) and CLIP simliarity score (CLIPSIM) (Wu et. al. 2022
H Parameter Efficient Fine- Tuni-T (PEFT): Training only the Linear Connection.
Method
PTT modality
Model size
COCO
VQA v2 vll
MSR MSRVTT-QA
Audiocaps
PromptFise Liang et al.
2022
Tex
0.22B
31.1
ProscaIiLM M (ang
Video-Text
0.801
47.0
6P-ATM Shikkor et n.
2023
Text
2.8
97.2
53.3
50.7
36.7
63.6
UnIVAL [owrs]
Image- (81)
0.255
71.6
39.8
47.5
UnIVAL louss]
Iuage+ Text (82)
0.25B
132.7
71.6
51.8
33.6
49.5
Table 16: Finetuning only the linear connection different hatgge//vvo////udiioe tasks. Despite the significantly smaller size of AL, the model can achieve reasonable performance when finetuned on new modalities. Scores gray for models pretrained the same target modality.
Once we have powerful pretrained models, it becomes important to develop highly efficient approaches that can be adapted to various tasks and modalities. Recent studies (Shukor et al., 2023a Merullo et al. 2022) have demonstrated the possibility of efficiently adapting unimodal pretrained models to multimodal tasks, by training only linear layer. The key idca is to project modality-spcciffc tokens onto the input text space of language model, effectively transforming them into textual tokens, while keeping all the pretrained parameters frozen. While this approach has proven effective with large models containing billions of parameters, in this section, we explore this setup with smaller models comprising several hundred million parameters. Following UnIVAL pretraining, we train only the linear projection responsible for mapping the output of the modality-specifie encoders to the input of the LM encoder.
As shown in Tab UnIVAL achieves reasonable performance on new tasks and modalitics despite the smaller parameter count. However, these results suggest that achieving competitive performance with only the linear connection may require larger models training on larger datasets.
32

---

Published in Transactions on Machine Learning Research (12/2023)
Pretrain Modality
COCO
VQA v2
R+CCOCO+
MSR VTT
MSRVTT-Q4
37.9
62.1
6.4
47.7
23.0
128.0
73.1
70.5
17.3
29.0
96.6
68.1
24.3
5-15
41.9
I+V
73.2
70.2
56.3
Table 17:
Knowledge transfer across modalities. Training on images helps significantly the vidco tasks. However, training on videos does seem to have a significant effect on image tasks.
Ablation Study
Knowledge transfer across modalities. Here we investigate the knowledge transfer between modalities, in other words, how learning modality affect the performance of the model other modalities. We test the following hypothesis; pretraining more modalities should improve the overall performance all lasks.
Tab 17 shows that in general learning a new modality, improves the performance on other modalities. Besides, it significantly helps to solve the downstream tasks of the same modality. Compared to model initialized from scratch, training solely on image-texi. datasets help VideoQA. In addition, training on video-text datasets (V) significantly helps image-text tasks on VQAv2, COCO and RefCOCO+. Finally, training both image and video-text datasets improve the performance on video-text task (w.r.t to pretraining on video) and did not degrade the performance image-text tasks.
Efficiency during training. Another important aspect of our approach is the significantly shorter training time. In Tab 18 we compare the training time (finetuning for one epoch) with the previous unified model OFA (Wang et al. 2022ch. Compare to OFA, our training time is significantly reduced, especially with tasks requiring high image resolution (e.g., 512x512 with RefCOCO+). This mainly due to the small number of visual tokens passed to the LM, that results from using additional convolution block in the image encoder. In addition to the training time, AL requires a 20 GB/GPU memory when finetuned with batch size of 16. During inference (on 1 GPU AMD Instinct M1250X with batch size of 1), the model requires 1.2GB of GPU memory and ~ 0.015 seconds to generate COCO caption.
COCO
Method
COCO
VQA
RefCOCO+
OFA
5.7
UnIVAL
3.1
Table 18: Estimated finetuning time in GPUh for one epoch training. UnIVAL significantly more efficient than OFA, especially with tasks using high image resolution. The total time is divided by the number of training GPUs.
J Additional Results
J.1 Text-to-image generation
Model
Model Size
Pretrain
FIDL
CLIPSIM!
1St
DALLE
Ramesh
2021
12B
27.5
Cog NonVVeew
Ding al.
4B
27.1
33.3
18.2
GLIDE
Nechol
M122
3.5B
/
12.2
Unifying
[lunn
2021
0.2B
X
29.9
30.9
NUWA
Wi al.
2122
0.9B
12.9
34.3
27.2
OFA Dose
Wang al.
X022e
0.2B
/
13.9
31.0
26.7
UnIVAL
(ours
0.2B
X
15.4
33.6
25.7
Table 19: Text-to-image generation on MSCOCO. Pretrain: image generation is included during pretraining.
We finetune UnIVAL on MSCOCO train set and compare the performance with other approaches. Tab 19 shows that model is competitive with previous approaches, despite being significantly smaller in size
33

---

Published in Transactions on Machine Learning Research (12/2023)
and does rely image generation during pretraining. Compared to OFA, we have very close performance, especially w.r.t the CLIPSIM score.
J.2 Linear interpolation of weights
To complement our study from the main paper, show in Fig more results when interpolating weights finetuned on different multimodal tasks. These results (on both image-text and video-text tasks) confirm those previously reported in the main paper.
80
Visual Grounding (Acc@ .)
Video Captioning (BLEU@4)
60-
40
Pretrain init Vanilla Ratatouille
Pretrain init Vanilla
20
Ratatouille
40
50
20 VideoQA (Acc)
VQA (Acc)
(a) Visual Grounding and VQA.
(b) Video QA and Video Captioning.
Figure 6: Addition weight interpolation results.
J.3
Finetuning for OOD generalization.
explores the generalization abilities of different finelunings strategies after zero-shot evaluation. We evaluate the OOD performances on 3 datasets that were not seen during pretraining nor during finetuning: OKVQA (a VQA task), VizWiz (a VQA task) and nocaps (an IC task). We use the model trained on VQAv2 for OKVQA/VizWas and on COCO Captioning for nocaps. While fusing outperforms on VizWiz the vanilla finetuning (on VQAv2), it lags behind on the other 2 evaluation datasets. Ratatouille, significantly outperforms both vanilla. and fusing finetuning on all OOD datasets, which echos the observation from Rame et al. (002aa). The reason being that features diversity (promoted by delayed averaging in ratatouille) increases robustness, and thus helps OOD generalization.
Fig 20
OKVQA
VixWiz
NoCaps CIDEr (out-domain)
Model
Val
Val Ace
Vanilla
38.06
13.57
94.39
Fusing
35.12
15.63
93.58
Ratarouille
38.97
18.48
95.28
Table 20: Zero-shot evaluation of different finetunings strategy.
K Discussion
In this section discuss some of the limitations and interesting future directions.
34

---

Published in Transactions on Machine Learning Research (12/2023)
K.1 Limitations
Hallucinations, abstention and other biases. We find that model suffers from different kind of hallucinations (Fig however, similar to OFA, it is less inclined to hallucinate (especially after finetuning) compared to other multimodal models (Tab including larger models like Flamingo (Alayrac et al. 2022 (Fig 7. Reducing hallucinations remains an ongoing challenge within the research community, which has become more prominent with the emergence of large-scale multimodal models. While certain recent studies [Bieen et al. 2022 Dai et al. 2022) have proposed partial solutions to address this problem, an effective approach for mitigating hallucinations largo-scale pretrained models has yet to be established. Additionally, refraining from generating answers (Dancette al. 2023 or visual grounding can be promising directions to enhance factuality and diminish hallucinations. Nevertheless, despite the progress made by the rescarch community, there is still much work to be done in this area. Other biases and limitations that are crucial to address, and have not been covered in our work are; social biases, toxic generation, and explainable generation. As shown (Shukor et al. 2023b), these limitations might not be solved by merely scaling the models and more dedicated approaches [Rame et al. 2023b Sun et al. 2023 might be useful to address some these issues.
Method
CIDER
CHAIRs 4
CHAIR, +
OSCAR Base
Base
et al.
2020b
117.6
13.0
7.1
VinVL Larg
Zhang
al.
2021
130.8
10.5
5.5
BLIP Large
LL et al.
2022b
136.70
8.8
4.7
OFA (Wang et al.
2022c
75.27
4.36
3.98
UnIVAL
91.04
4.44
3.64
OFA Ft (Wang et al.
2022c
138.1
3.06
2.03
UnIVAL Ft
137.0
3.26
2.20
Table 21: Hallucinations. Comparison with different foundation models. So'IA results from
(Dai et al. 2023b
Q: what is on the phone screen? Q: what can you see out the window?
Q: whom is the person texting?
Flamingo:
A: text message from friend
A: a parking lot
A: the driver
UnIVAL:
nothing
A: restaurant
A:
Figure 7: Hallucinations with open-ended VQA. UnIVAL is less prone to hallucinate compare to Flamingg-808
Alayrau
eb al. 2022
Complex instructions following. UnIVAL exhibits good performance when presented with straightfor- ward instructions commonly encountered in standard benchmarks. However, it encounters difficulties when faced with complex instructions, such delivering intricate image descriptions or providing explanations for memes. To overcome this challenge, finetuning the model using substantial number of diverse instructions
serve potential solution (Xu et al. 2022 Liu et al. 2023a
Dai et al. 2023a
35

---

Published in Transactions on Machine Learning Research (12/2023)
you know this painting?
what image describe details?
image describe?
funny about image?
na
group of people riding skis down snow covered street
A: man loads up his car with clothes
A: how many cookies are there
Figure 8: Limitations of UnIVAL in following instructions. UnIVAL unable follow complex instructions,
Unimodal tasks. We noticed that training solely on aligned multimodal tasks can degrade the performance of the model in tackling unimodal ones. This problem is usually addressed by adding unimodal data, such
corpus of text or image, during pretraining (Singh et al. 2022 Lu et al. 2022a Wang et al. 2022c
Zero-shot evaluation and efficient finetuning. The ideal scenario is for the model to demonstrate strong performance and generalization across multiple tasks following the pretraining phase. However, we have observed that refraining from finetuning or solely training the linear connection (Shukor et al. 2023a results unsatisfactory performance compared to SoTA approaches. This issue can be lackled by training larger models on a greater number of instructions /tasks or by employing alternative parameter-effcient
finetuning techniques (Hu et al. 2021 Lester et al. 2021
K.2 Future Directions
Model scaling and better LM initialization. In this study, we conduct experiments using a relatively small BART-initialized encoder-deooder transformer. Nonetheless, numcrous intriguing language models have recently been introduced (Rallel et al. 2020 Zhang al. 2022 Touvron et al., 2023), which could potentially enhance performance when fine-tuned for multimodal tasks. Another aspect involves reasonably scaling the model size and training it on larger datasets, which could unveil more capabilities like In-Context Learning
Dong et al. 2022) and the ability to tackle more complex tasks (Lu et al. 2022b
More modalities and tasks. Our study demonnttraee the feasibility of training a unified model capable of addressing tasks involving image, video, audio, and text modalities. As a result, we posit that incorporating additional modalities, either during the pretraining phase or solely during finetuning,
be accomplished straightforwardly Furthermore, expanding the scope of tasks within each modality, such as incorporating broader range of visual tasks (Lu et al. 2022a Zou et al. 2023) or tasks necessitating complex reasoning abilities (Liu et al. (023aa,, represents a natural extension of this work. Ideally, we hope that in the future, there will be modality-agnosth models, bridging the gap between domains and modalities.
Towards embodied and generalist multimodal assistant agents. Modality-agnostii models hold the potential to facilitate the development of embodied agents capable of addressing real-world challenges, including navigation and robotics manipulation, which demand the simultaneous handling of multiple modalities. Furthermore, while there has been notable progress in the NLP community regarding the construction of generalist agents, such as chatbots (Liu et al.) 2023b). these advancements remain constrained in terms of their ability to accept diverse input modalitics and generate outputs beyond tcxtual form.
Better training schemes for multitask multimodal training. While growing the number of tasks and modalities, it is important to devise new efficient training schemes to better leverage the collaboration between
36

---

Published in Transactions on Machine Learning Research (12/2023)
tasks, and continually support more modalities. We believe that there is more efficient approaches than our multimodal curriculum learning, to continually add more modalities while avoiding forgetting previous ones.
L
Qualitative Results
37

---

Published in Transactions on Machine Learning Research (12/2023)
broken accordion sits on the floor
close up a doughnut with ketchup on it
a park with benches and a fire hydrant
the woman wearing blue
the family photo
the Tokyo Skytree
Q: Is the woman wearing green happy? A: no
Where people eating? A: restaurant
Q: Whom is the person texting? A: man
Figure 9: Limitations of UnIVAL. We show the limitations different image-text tasks; (row 1) objects hallucinations (Image Captioning), (row 2) inability to capture nuanced description, object hallucinations, and struggle with far/small objects (Visual Grounding) and (row answer hallucination (VQA).
38

---

Published in Transactions on Machine Learning Research (12/2023)
Which region does the "the man sitting on the ground" describe?
Which region does the "the girl with blonde hair" describe?
Which region does the" bicycle behind the bench" describe?
Which region does the" bedside lamp above the bed" describe?
Which region does the "the man sitting the ground" describe?
Which region does the "bedside lamp above the bed" describe?
Which region does the "the snow on the mountains" describe?
Which region does the "the red vehicle" describe?
Which region does the "the sheep standing the right' describe?
Figure 10: Visual Grounding. Image from COCO val 2014 set. Texts constructed manually.
39

---

Published in Transactions on Machine Learning Research (12/2023)
Q: what does the image describe?
Q: what does the image describe?
what does the image describe?
small bird sitting on top of a skateboard
close up of banana and a bagel
two laptops sitting top of white box
Q: what does the image describe?
Q: what does the image describe?
Q: what does the image describe?
baseball player holding a bat on a field
wooden table topped with three vases filled with flowers
group of young men playing a game of soccer
Q: what does the image describe?
Q: what does the image describe?
Q: what does the image describe?
a man riding skis down a snow covered slope
herd of sheep grazing on lush green field
man sitting on bench surrounded by pigeon
Figure 11:
Image Captioning. Image from COCO val 2014 set.
40

---

Published in Transactions on Machine Learning Research (12/2023)
Q: why the people are happy?
is there basketball hoop in the image?
Q: this a vegetarian plate?
A: they're graduating
A: yes
A: no
Q: what is the woman watching?
Q: what is the man doing?
Q: what the woman wearing black doing?
A: polar bear
riding motorcycle
A: walking
Q: how many players are the court?
Q: hat does the street sign say?
Q: what is the colour of the man's shirt?
A: 2
A: walk
A: orange
Figure 12: VQA. Image from COCO val 2014 set. Question constructed manually.
41


Reviews:
[{"review_id": "VzEJZynl8p", "reviewer": "Reviewer_wPdr", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The proposed method achieves competitive results with more modality-specialized methods without using web-scale data or extremely large - The authors conduct significant testing of the model on different tasks, modalities, and types of seen or unseen data. It is overall comprehensive and convincing of the method's effectiveness.\n- The authors demonstrate that multimodal curriculum learning and task balancing can help improve efficiency when training multi-modal models, a useful finding for others training similar models in the future.\n- The paper is well-written and easy to follow. The appendix contains significant depth that was able to answer most questions I had from the prose-heavy main paper.\n\n**Weaknesses**\n- Some statements made by the authors are lacking in justification. See the first set of requested critical changes below.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "In this paper, the authors propose UnIVAL, a transformer encoder-decoder based model presenting unified input/output and modeling across varying tasks and modalities (**I**mages, **V**ideo, **A**udio, and **L**anguage). UnIVAL casts tasks to natural language inputs (e.g., “What does the video describe?” for video captioning) and connects modality-specific encoders to a shared representation space via a process of flattening, tokenization, and projection. Unlike other work, this allows UnIVAL to provide a single task and modality-agnostic training objective: next token prediction. And, unlike many contemporary multi-modal models, UnIVAL utilizes curated task-specific datasets instead of web-scale training and requires significantly fewer parameters. With this model, the authors demonstrate that multimodal curriculum learning (MCL) with task balancing can improve pretraining efficiency by sequentially adding new modalities, instead of beginning with all available, and balancing the occurrence of different tasks within each batch. The authors demonstrate that UnIVAL with MCL has competitive performance on various image-text, video-text, and audio-text tasks, without audio ever being seen in the pretraining data. Finally, the authors show that weight interpolation can efficiently merge the skills of different fine-tuned UnIVAL models.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "**Critical**\n- Multiple statements need further justification. I don’t expect major revisions, but some addition is needed.\n  - *Section 3.1 “To tackle multimodal tasks at small to mid-scale, we employ an encoder-decoder LM, **due to its effectiveness for multimodal tasks and zero-shot generalization** after multitask training.”* There is insufficient justification for this, and it is not necessarily common knowledge. Please provide some support either in citation, intuition, and/or experimentation.\n  - *Section 3.1 “Another advantage of this architecture is the inclusion of bidirectional attention mechanisms in addition to unidirectional causal attention. **This is particularly beneficial for processing various non-textual modalities**.”* There is insufficient justification for this, and it is not necessarily common knowledge. Please provide some support either in citation, intuition, and/or experimentation.\n  - *Section 6 “This distinction can be attributed to using **smaller LM, a component that is known to be particularly susceptible to this issue when scaled**.”* There is insufficient justification for this, and it is not necessarily common knowledge. Please provide some intuition or a citation behind this.\n  - *Section 3 “We hope that the quality mitigates the need for massive datasets, thereby reducing computational requirements, **while enhancing the model’s generalization capabilities to novel tasks**.”* Similar issue to the above. This one is not critical, as it’s speculation about the effects of the author's proposed method, rather than a direct claim.\n- What exactly is the “synergy” between tasks and modalities? It is referenced as something separate from knowledge transfer between tasks in the third bullet point under contributions. Either different language should be used, or the term should be further specified. For example, Table 4 shows “the synergy between different tasks and datasets.” What specifically does it mean to show “synergy” in this case?\n\n**Non-Critical**\n- What is the training procedure used for evaluating multimodal task balancing in section 3.5? If it Is the same as the standard training procedure described further on, briefly specifying this would be appreciated.\n- Move some discussion of social bias impacts from the appendix into the main paper body, as it is not obvious that they are sufficiently discussed from the main paper alone.\n- Typos:\n  - Section 6, “attributed to using smaller LM”: should be “attributed to using **a** smaller LM”\n  - Appendix, “Multimodal pretraining”: “try to learn shared and aligned latent space” should be “try to learn **a** shared and aligned latent space”", "broader_impact_concerns": "I do not have major broader impact concerns for this work. The authors include a detailed model card and discussion of bias implications in the appendix. Including some of the bias discussion in section 6 (Discussion) would be appreciated but is not critical to my decision.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vtyx2vDxV4", "reviewer": "Reviewer_hRUp", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength: this model is a useful contribution to the public domain. There are not many models available today that combine multiple modalities and are small enough to be easily accessible. Do a description of the training methodology and of the various approaches to fine-tuning across tasks provide interesting data points to the lore of training large multimodal models. The paper is easy to follow and well illustrated.\n\nWeaknesses: to fully characterizes the model, it would be very useful to provide a more comprehensive systems view of the model. Statistics about entrance speed. Statistics about fine-tuning speed. High watermark memory consumption. One could argue that this is largely a systems paper assembling known techniques and mostly relying on the implementation details and careful benchmarking as the novelty.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposes a new transformer model which supports multiple modalities including video audio images of text. The model is relatively compact and exhibits strong performance on a number of benchmarks. The model in code is expected to be open source. The paper describes a numbers of approaches to training which enhanced the ultimate outcome. It also explores different approaches to fine-tuning which enables model to perform well across multiple fine-tuned downsteam tasks jointly. Exploration of weights interpolation techniques such as ratatouille to enhance fine-tuning across tasks and modalities provide validation that these can be made practical beyond SOTA in a wide range of multimodal settings. The paper provides a detailed description of the data sources and of the limitations of the model in appendix.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "Please provide some systems characterization of the model in terms of speed and memory.", "broader_impact_concerns": "No specific concerns. The model card and examples provide it describe the models limitations well.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "QuEpUKgtz3", "reviewer": "Reviewer_HzJS", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n- The proposed model achieves better performance than previous works for various tasks/datasets and competitive performance on other tasks/datasets.\n- The proposed model should be extendable to include more tasks/datasets without many changes.\n- The paper shows the importance of multimodal multitask pretraining, including task balancing and multimodal curriculum learning.\n\n\n**Weaknesses:**\n\n- My biggest concern is with the overall takeaway from the paper. The conclusions from the experiments seem a bit blurry to me after reading the paper so clarification from the authors would be very helpful. Overall, the only consistent takeaway from the experiments section seems to be that UnIVAL is better than other previous methods. But, the section fails to present a convincing argument for what exactly makes UnIVAL a better method than other works. Below are some specific cases that lead me to this high-level weakness: \n    - I think the authors need to discuss explicitly and in detail the technical takeaways from this paper. Some key questions to ask — Should I just use UnIVAL over other methods for all text-vision-audio tasks? What are the cases where it is better to use UnIVAL over other methods and what are the cases where choosing other methods might be better?\n    - The final goal seems to be a bit blurry to me. Is the goal to achieve a final best text-vision-audio model? If so, isn’t it better to pretrain on all available datasets? Why not also use the datasets used for zero-shot evaluation as well audio captioning and text-to-image generation (Sections 4.2 and 4.3) to obtain the best final model?\n    - What are the technical differences (in terms of network architecture, loss function, etc.) from closely related works such as OFA and Unified-IO?\n    - For audio captioning, Table 9, the comparison is with methods that are trained specifically for audio captioning, right? But, all of them have only seen a very small amount of data, i.e., the training dataset of the corresponding audio captioning dataset, which becomes specially relevant because these datasets are very small in size (could the authors please report the sizes?). I appreciate the novelty in applying the proposed method for a novel modality, but the proposed method is also not compared with other unified methods to compare the adaptation capability. In the current state of this section, it’s not clear what to take away from these results.\n    - Table 5, Visual Grounding results, what are the results for models that are not of comparable sizes? They should also be included in the table. Also, the pretraining data is different for different models, right? Are all the compared methods pretrained on at least the data UnIVAL is pretrained on? Without this information, I am unsure what to take away from these results.\n    - Why is Unified-IO not included for comparison for image-text tasks (Section 4.1.1)?\n    - Why is OmniVL not included for video captioning in Table 8?\n\n\nOther concerns/questions:\n- The model merging across multimodal tasks seems very separate from the rest of the paper and it seems to have expected results (please correct me if the authors think otherwise). I agree that it is still useful to validate known hypotheses through experiments, but I don’t think this section is adding any contributions to the paper.\n- How does merging models finetuned on different mutlimodal tasks compare to merging models pretrained on different multimodal tasks / modalities (Song et al., 2023)? I understand that it is a concurrent work but I think it should still be included, the current results seem to tell half the picture.\n- “*validating the superior performances of ratatouille.*” Again, the takeaway message from this section is confusing. Why is it relevant for this paper to show that ratatouille merging is better than vanilla averaging? Same comment applies to the comparison between Fusing and Ratatouille in the first paragraph of page 12.\n- For text-to-image generation task, how do you generate an image as the output? \n- Why use beam search for VQA? Are the results different with all-candidate evaluation?\n- Table 12: why are the two image-text multitask pretraining numbers different?\n- “*Tab.13 shows that pretraining on either image-text or video-text data helps to get additional improvement*” The numbers in the table don't show that pretraining on video-text data helps.\n- There are various minor issues that are enough to break the readability of the paper. Some are listed below:\n    - “*training on VQA helps to get +1.9 points on Captioning and 4 points for Visual Grounding*”. Should this be 5 points for Visual Grounding if I’m reading the table correctly?\n    - “*This setup is similar to the zero-shot evaluation, except that the evaluation tasks are seen during pretraining*”. This is very confusing — since the tasks are seen during pretraining, this setup is far from zero-shot. Please rephrase.\n    - “*the CIDEr score of the model for λ = 0.8 with weights 0.8*Cap+0.2*VQA is 138.51 vs. 136.52*” It should be Visual Grounding instead of VQA, right?\n    - “*(ii) then we launch multiple finetunings on VG from these auxiliary weights; (iii) we uniformly average all the weights finetuned on VG to obtain W1*” It should be VQA instead of VG?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new unified multimodal model called UnIVAL. It is a ~0.25 billion parameter model, the core of which is pretrained on multiple vision-text and video-text tasks using small scale datasets. After pretraining, it can be used for various multimodal tasks including text, images, videos and audio such as image captioning, video captioning, audio captioning, visual question answering (VQA), video QA, visual grounding and text-to-image generation. The model achieves competitive performance to existing state-of-the-art methods across these tasks. The paper also studies merging of model weights finetuned on different multimodal tasks.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "I have included all my questions that I would like the authors to answer in the weaknesses section above. The answers to all these issues are crucial to my final recommendation.", "broader_impact_concerns": "Given that the proposed method can be extended to various tasks and datasets without requiring a lot of data, I think the authors should include a Broader Impact section to discuss these concerns. They should also point out that the model might have learnt various biases present in the datasets that were used for pretraining.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The proposed UnIVAL method is a unified architecture for text grounded image, video, and audio processing tasks that is trained with relatively modest computationally and data resources compared to many large-scale multimodal models. This is enabled by careful curriculum and multitask learning techniques ablated in the paper. Despite its relatively smaller size and lower data/compute budget, UnIVAL achieves competitive results with many modality-specific models. During the reviewing process, reviewers raised several requests for clarification that authors have responded to and made appropriate updates to the paper. There are several comparisons to other multimodal models that would have improved the work but do not invalidate the central claims. Reviewers generally were positive about the lower data requirements and model size the proposed work offers to the community. As such, the open sourcing of code and model weights promised in the manuscript will be important.
Decision: Accept (poster)