Prismer: A Vision-Language Model with Multi-Task Experts
Authors: Shikun Liu
Linxi Fan
Edward Johns
Zhiding Yu
Chaowei Xiao
Anima Anandkumar
Venue: Accepted by TMLR ()

Abstract: Recent vision-language models have shown impressive multi-modal generation capabilities. However, typically they require training huge models on massive datasets. As a more scalable alternative, we introduce Prismer, a data- and parameter-efficient vision-language model that leverages an ensemble of task-specific experts. Prismer only requires training of a small number of components, with the majority of network weights inherited from multiple readily-available, pre-trained experts, and kept frozen during training. By leveraging experts from a wide range of domains, we show Prismer can efficiently pool this expert knowledge and adapt it to various vision-language reasoning tasks. In our experiments, we show that Prismer achieves fine-tuned and few-shot learning performance which is competitive with current state-of-the-arts, whilst requiring up to two orders of magnitude less training data. Code is available at https://github.com/NVlabs/prismer.

Published in Transactions on Machine Learning Research (01/2024)
Prismer: A Vision- Model with Multi-Task Experts
Shikun Liu 1.2+
Linxi Fan?
Edward Johns'
Zhiding Yu?
Chaowei Xiao233
Anima Anandkumar24
Imperiall College London
NVIDIA
University of Wisconsin, Madison
Caltech
Reviewed on OpenReview: https: // openreveewwwww.. net/ formm? id- R7H43YD66O
Abstract
Recent vision-language models have shown impressive multi-modal generation capabilities. However, lypically they require training huge models on massive datasels. As a more scalable alternative, we introduce Prismer, a data- and parameter-effiient vision-language model that leverages an ensemble of task-specific experts. Prismer only requires training of a small number of components, with the majority of network weights inherited from multiple readily-availabbe pre-trained experts, and kept frozen during training. By leveraging experts from a wide range of domains, show Prismer can efficiently pool this expert knowledge and adapt it to various vision-language reasoning tasks. In our experiments, we show that Prismer achieves fine-tuned and few-shot. learning performance which is competitive with current ttate-of-the-arss, whilst requiring up to two orders of magnitude less training data. Code is available petps://githnn com//Vlaabs/prrsmer
1 Introduction
Large pre-trained models have demonstrated exceptional generalisation capabilities across a wide range of tasks. However, these capabilities come at hefty cost in terms of computational resources required for training and inference, as well as the need for large amounts of training data. In the language domain, models with hundreds of billions of learnable parameters typically require compute budget the yottaFT.PP scale
[Chowdhery et al. 2022 Brown et al. 2020 Black et al. 2022. Rae et al. 2021).
The problems in lasion-l learning are arguably more challenging. This domain is a strict super-set of language processing, whilst also requiring extra skills unique to visual and multi-modal reasoning. For example, many image captioning and visual question answering problems require the model to be capable of fine-graine fine-grained object recognition, detection, counting, and 3D perception (Antol et al., 2015 Chen al., 2015). A typical solution to use a massive amount of image-text data to train one giant, monolithic model that learns to develop these task-specific skills from scratch, simultancously, and within the same generic architecture.
Instead, we investigate an alternative approach to learning these skills and domain knowledge via distinct and separate sub-networks, referred to as "cxperts". As such, each expert can be optimised independently for specific task, allowing for the use of domain-specific data and architectures that would not be feasible with a single large network. This leads to improved training efficiency, as the model can focus on integrating specialised skills and domain knowledge, rather than trying to learn every thryyhhnng& at once, making an effective way scale down multi-modal learning.
To achieve this, we propose Prismer visually conditioned autoregressive text generation model, trained to better use diverse pre-trained task erperts for open-ended lision-language reasoning tasks. Prismer's key design elements include i) powerful vision-only and language-only models for web-scale knowledge to construct our core network backbones, and ii) multi-lask vision experts encoding multiple types of visual information,
* *Corresponding Author: shikun www.:///mmppraal....mmmm uk. Work done during an internship NVIDIA. The model name "Prismer" draws from the analogy to optical prism which breaks white light into spectrum colours, and here we break down single reasoning task into diverse domain-specific reasoning.

---

Published in Transactions on Machine Learning Research (01/2024)
OCR DETECTION
SEGMENTATION
BISEBBLL
PRRSON
IMAG CAP TINNING
TO SOC
mmn playing
baseball in a field.
VISUA QUESTION ANSWERING
Q: What's doing?
PRISMER
A: Playing baseball.
What's number player? A: 21.
DEPTH
EDGE
Figure 1: Prismer model overview. Prismer is a data-efficient vision-language model that leverages diverse pre-trained experts through its predicted multi-task signals. It perform vision-language reasoning tasks such as image captioning and visual question answering. The analogy is with an optical prisi: Prismer splits a single reasoning task into diverse domain-specific reasoning.
including low-level vision signals such depth, and high-level vision signals such as instance and semantic labels, form of auriliory knowledge, directly from their corresponding network outputs. All expert models are individually pre-trained and frozen, and are connected through some lightweight trainable components which contribute to roughly 20% of the total network parameters.
Despite Prismer being trained on only 13M publicly available image/alt-text data examples, it shows strong multi-modal reasoning performance tasks such as image captioning, image classification, and visual question answering, competitive with many thte-rtf vision-language models (Alayrac al. 2022 Wang et al., trained with in-depth analysis of Prismer's learning behaviours and observe some encouraging properties. For example, i) Prismer exhibits strong robustness against the inclusion of noisy experts, and ii) the learning performance also scales favourably with increases in both the quantity or quality of experts.
2022a 2021). that
two orders of magnitude more data. Finally, y, we conduct
2 Related Work
Vision-Language Models (VLMs) Inspired by the breakthrough of transformers in the language domain
Vaswani et al. 2017 Devlin et al., 2019), early works aimed to model the vision-language relationship using shared network based on transformers
single-stream design (Li al. 2020a Chen et al.) 2020 Li
et al. 2020b: Su al. 2020). These works usually leverage a pre-trained object detector, encoding images sequences of visual words, parameterised by object- region-level features. Prismer takes slightly different approach by using pre-trained models to provide their output predictions as auxiliary signals, whilst still relying on the original images to encode visual features.
Another line of works encodes vision and language features in separate networks in a dual-stream design, where the vision-only and language-only fcatures are aligned through contrastive learning (Radford et al.] 2021 Zhai al.) 2022, Jia al., 2022 Li al. 2021). These works typically focus on close-ended multi-modal alignment tasks such image-text classification and retrieval. In contrast, Prismer's vision encoder also
2

---

Published in Transactions on Machine Learning Research (01/2024)
aligns its vision features with the language embedding through pro-training with contrastive learning, but with a greater emphasis on multi-modal generation tasks.
Both single- and dual-steam VLMs in the past years have often been pre-trained with a combination of multiple objectives, such as masked language modelling, masked region modelling, word-region alignment, visual grounding and more (Li et al.] (2020a; Cho et al. 2021 Li al. 2022 2021 Lu et al. 2019). These
multiple objectives can make the training process more complex and require careful balancing of the different losses. Prismer adopts a different approach, aligning with recent developments in VLMs that focus language
generation, and only require a single autoregressive training objective (Wang al. 2022a 2021 2022). Despite the reduced complexity, training these large-scale VLMs is data intensive and computationalyy demanding, often requiring billions of training data. To overcome these challenges, Prismer leverages powerful pre-trained task-specific expert models for data-efficient training. Unlike another set of works that prioritise in-context capability by conditioning on a large frozen language model with no task-specific fine-tuning [Eichenberg et al.) 2021 Tsimpoukelli et al. 2021 Alayrac et al.) 2022), Prismer focuses on fine-tuned performance with an emphasis on parameter efficiency, using smaller but diverse pre-trained experts.
Hu et al.,
Multi-task and Auxiliary Learning Multi-task learning and auxiliary learning aim to train models to predict multiple outputs (such as semantic segmentation, object detection, and depth cstimation) from a single input, thereby improving the performance across one or multiple tasks. This is oflen achieved through the design of effective multi-task networks that balance task-shared and task-specific features {Liu ct al. 2019b Misra et al.] 2016 Sun et al.] 2020 Xu et al. 2018), or through the explicit modelling of task relationships (Liu et al.] 2019a 2022 Navon et al. 2021 Zamir et al., 2018 Fifty et al. 2021). Recently, multi-task learning has been further generalised to unily vision-only, language-only, and vision-language tasks by considering them within a requence-to-sequeee framework (Wang et al.) 2022b Lu et al. 2022 Zhu et al. 2022 Prismer also employs multiple tasks, specifically in the vision domain, similar to these methods, but uniquely uses them solely as input, serving as auxiliary knowledge. Prismer is more related to works such as [Bachmann et al. 2022: Ghiasi et al. 2021), which utilise pre-trained experts to create pscudo labels for multi-task self-training. However, whilst those methods focus on learning task-agnostic features through multi-task supervision, Prismer focuses purely on multi-modal reasoning with a single-tast objective.
Unifying Pre-trained Experts The utilisation of diverse pre-trained domain experts for multi-modal reasoning has been investigated in previous studies. Socratic models [Zng et al. 2022) use language as a one-way communication interface to connect different pre-trained experts. ViperGPT (uuís et al 2023) and Visual Programming (Gupta & Kembhavi 2023) harness the in-context learning capabilities of large language models, breaking down complex multi-modal rcasoning into modular programs, which are then solved sequentially by leveraging pre-trained vision experts through APIs. The aforementioned methods excel at modular problem decomposition and establishing connections among pre-trained experts, and thereby being limited to zero-shot multi-modal reasoning within the domains which the experts were pre-trained, and errors predicted by previous experts can be carried forward to future experts. However, Prismer stands out with a distinct objective by aiming to better bridge these pre-trained experts through a unified architecture design. As such, Prismer aims to create more seamless collaboration between these experts, ultimately optimising multi-modal reasoning in a more integrated manner, and more robust to non-optimal experts.
Finally, we would like to highlight the distinction between the concept of "experts" defined in "Mixture of Experts (MoE))"(Rquull)) (Riquelme et al.) 2021 Nguyen & Chamroukhi 2018 Masoudnia & Ebrahhmpour 2014)) and in Prismer. In MoE, the "experts" are sub-modules in single network, interconnected through their corresponding gating networks, encoding émplicit knowledge guided by shared training objective. On the other hand, in Prismer, the "experts" are independently pre-trained models, encoding explicit knowledge based on their pre-trained tasks or domains.
3 Prismer: Open-ended Reasoning with Multi- Task Knowledge
In this section, we introduce the Prismer model, a type of vision-language generative model that takes multi-task signals as input, and outputs free-form text.
3

---

Published in Transactions on Machine Learning Research (01/2024)
A bear is sitting on the grass 

LayerNorm
Prediction
Feed-Forward Block
Feed-Forwar Block
*
*
Adaptor
Adaptor
Bi-Directional
Cross
Attention Block
Attention Block
X N
Causal *
Experts Resampler
Attention Block
RGB
X M
Patchify
Depth Patchify
Normal Patchify
Word Embedding
A bear is sitting on the grass
Vision Encoder
Language Decoder
Figure 2: Prismer architecture design overview. Prismer has two main trainable components: the Experts Resampler that converts variable multi-task signals to fixed number of outputs, and the Adaptor that enhances the model's expressivity for vision-language reasoning. To ensure that the model takes advantage of the rich domain-specific knowledge encoded in the pre-trained experts, the majority of network weights are frozen during training, as represented by $
3.1 Model Overview
The design of the Prismer model illustrated Fig. 2 Prismer is an encoder-decoder transformer model (Vaswani al. 2017) that leverages library of existing pre-trained experts. consists of a vision encoder and an auto-segressive language decoder. The vision encoder takes an RGB image and its multi-task labels input (e.g. depth, surface normal, segmentation labels, predicted from the frozen pre-trained experts). and outputs sequence of RGB and multi-task features. The language decoder is then conditioned on these multi-task features via cross attention, and produces a sequence of text tokens.
Prismer is designed to leverage pre-trained experts whilst keeping the number of trainable parameters to a minimum. To do this, the network weights of the pre-trained experts are frozen to maintain the integrity of their learned knowledge and prevent catastrophic forgetting (Kemker al.) 2018 Kirkpatrick al. 2017). To link the multi-task knowledge as well as the vision and language parts of Prismer, we insert two parfieter-efffiient trainable components: Experts Resampler and Adaptor. The Experts Resampler is used in the vision encoder to map variable length of multi-task signals to a sequence of multi-task features with a fixed length. The Adaptors are inserted in each transformer layer of the vision and language parts of the model to better adapt the pre-trained experts to new tasks and modalities.
Prismer generative model, and we re-formulate all vision-language reasoning tasks as a language modelling or prefiz language modelling problem. For example, given the input image along with its multi-task tokens (predicted with the multi-task experts) and question the prefix, the model generates the answer for the
4

---

Published in Transactions on Machine Learning Research (01/2024)
visual question answering task; given the input image along with its multi-task tokens, the model generates its caption for the image captioning task. Once we have a prefix prompt, we may either sample the output text in an autoregressive manner, as in an open-ended setting; or we may rank the log- logliiklihood from fixed set of completions, as in a closed-ended setting.
3.2 Pre-trained Experts
In Prismer, we include two types of pre-trained experts:
Backbone Experts The vision-only and languago-only pre-trained models, which are responsible for encoding images and texts into a meaningful sequence of tokens. Both models are required to be based on the transformer architecture |Vaswani et al. 2017), so that can easily connect them with a few trainable components of similar designs. To preserve their rich domain-specific knowledge encoded in the network parameters, the majority of the weights are frozen during pre-training.
Task Experts The models that produce multiple task-specific labels, depending on their training datasets, are treated as black-boz predictors. These task experts can be designed either as single multi-task expert or an ensemble of multiple task-specific experts, and their prodicted labels are utilised as input for the Prismer model. Consequently, all network weights of the task experts are frozen, and they can have any design. Prismer, we incorporate up to task-specific experts, all within the vision domain. These experts encode three low-level vision signals (depth, surface normals, and edges) and three high-level vision signals (object labels, segmentation labels, and OCR labels). Our selection of these 6 vision experts is based tasks commonly studied in the multi-task learning community (Zamir et al. 2018 Standley et al. 2020 Liu et al. 2022)) which have demonstrated varying levels of benefits in learning generalised visual representations. Additionally, these expert models are relatively lightweight, incurring minimal additional training and inference costs with simple model parallelism.
We apply task-specific post-processing on these predicted labels, transforming them a R# tensor (here H, W, C represent image height, width and channels respectively. e.g. C C= for depth and edge labels, and C' = 3 for surface normals label). For all expert labels encoding high-level signals, we tile each pixel with its corresponding text embodding from a pre-trained CLIP text model (Radford et al. 2021] and then we apply PCA to down-sample the dimensionality to C = 64 for elficient training. The detailed descriptions of all task experts, including their pre-trained datasets and the architecture design, are listed in Appendix A
3.3 Key Architectural Components
Task-Specific Convolutional Stem All expert labels are first processed with randomly initialised convolution layers to map them to the same dimensionality. Specifically, apply 5 convolutional layers and each is composed of small [3 x 3] kernel, which is shown to perform better than single convolutional layer but with a larger kernel in the original Vision Transformer design (Dosovisskiy et al. 2020). consistent with the finding in (Xiao et al.] 2021). The convolutional stem is designed to be task-specific, which we have found to yield superior performance in comparison to a shared design in a multi-lask learning setting (Liu et al. 2019b Misra al. 2016).
For high-level semantic labels such as those in object detection, semantic segmentation, and OCR detection, down-sample the resolution by factor of 4 to conserve running memory. Furthermore, for each object instance, we add a trainable and randomly sampled embedding to distinguish among different object instances. The size of this instance embedding is set to 128, which corresponds to the maximum possible number of object instances to be present in a single image. For RGB images, we simply process with the pre-trained convolutional stem defined by the original vision backbone. All task expert embeddings, including RGB, are then added with a pre-trained positional embedding before being further processed by transformer layers.
Exports Resampler The computational complexity of self-attention is quadratically proportional to the number of input tokens. As such, the vision encoder can easily require tremendous memory when including a large number of task experts. To address this, we propose Eaperts Resampler, which takes a variable number
5

---

Published in Transactions on Machine Learning Research (01/2024)
of expert labels imput and outputs fized number of tokens, illustrated in Fig. 3 Left. Such design produces conslant memory for the self-attention in the vision encoder, as well as the vision-text cross at aetetioon in the language decoder (shown in Fig. 2 independent of the inclusion of a different number of experts. Inspired by the design in the Perceiver Resampler (Jeegle et al., 2021) and the Flamingo model (Alayrac et al. 2022 the Experts Resampler learns a pre-defined number of latent inpul queries, to cross-at a flattened multi-lask features. The resampler then compresses the multi-task features into much smaller number of tokens equal to the number of learned latent queries, as form of auziliary knowledge distillation. We design keys and queries to be concatenation for both multi-task features and the learned latent queries, which is shown to be more effective, consistent with the design in the Flamingo model (Alayrac et al. 2022
Lightweight Adaptor We insert one lightweight adaptor into cach transformer layer of both vision and language backbones in order to improve Prismer's expressivity and conditioning on multi-task features, illustrated in Fig. Right. The adaptor has an encoder-decoder design, which has proven to be successful for efficient transfer learning in the NLP domain (Houlsby et al., 2019 Pfeiffer et al. 2020). It first down-projects the input features into smaller dimension, applics a non-linearity, and then up-projects the features back to the original input dimension. We choose the non-linean linearity function to be squared ReLU (So et al., (2021) a simple and parameter-free function that delivers strong training stability. With the residual connection, we initialisc all adaptors with near-zero weights to approximate the identity function. Combined with a standard cross attention block in the language decoder, the model is able to smoothly transition from the domain-specific vision-only and language-only backbones image-text data.
Experts Resempler
Adaptor
Add & Norm
Feed-Forward
Up Projection
OI-Directional Attention Block
Squared ReLU
Ktt
Down Projection
Multi-modal Features
Learned Latents
Figure 3: Adaptor. Left: The Experts Resampler takes multi-task features with variable length as imput, and outputs a fixed number of tokens via cross attention. Right: The Adap- tor has residual connection to the input and two fully- connected layers, that down-projects the input features to a smaller bottleneck dimension and then up-projects back to the original dimension.
Design details in Experts Resampler and
vision-language model during pre-training with paired
The model performance, memory usage and time complexity for other design choices systematically evaluated and ablated in Appendix B
3.4 Training Objective
For simplicity, we train Prismer with single objective to predict the next text autoregressively. Following the standard encoder-decoder architecture, the vision encoder predicts the multi-task features 2, and the language decoder learns to maximise the conditional likelihood of the paired text caption y under the forward autorcgressive factorisation:
-E logp(ya (ey.peeee ().,
In practice, one-time pre-processing step of collecting multi-task expert labels is computationally cheap and fast with data and model parallelism. The single generative objective then only requires forward pass to compute gradients, which is significantly more efficient and streamlined than many other VLMs that may require a multi-stage and/or multi-step pre-training (Li el al. 2022 2021 Wang el al.) 2022b Dou et al. 2022 Chen et al.) 20200), with multiple objectives and data sources. However, because our model only focuses on multi-modal language generation, it less suitable for multi-modal discriminative tasks such image-text retrieval and visual entailment, which are the focus of other types of VLMs (Gan et al. 2020
Chen al.] 2020 Jia et. al. 2021

---

Published in Transactions on Machine Learning Research (01/2024)
Resampler
Vision Encoder
Language Decoder
Trainable Params.
Total
Layers
Width
Backbone
Layers
Width
Backbone
Layers
Width
Params.
prismergass
768
VIT-B/16
768
RoBERTagASE
768
160M
980M
Prismer
4
1024
VIT-L/14
24
1024
ROBERTaLARGE
24
1024
360M
1.6B
Prismer 7BASE
ViT-B/16
768
RoBERTaSASP
768
105M
275M
HissmrrZaaae
ViT-L/14
1024
ROBERTaLARCE
1024
270M
870M
Table 1: Prismer and PrismerZ architecture details. We report the backbone we choose for cach architecture size, along with its corresponding number of layers and width. We also report the number of trainable parameters and total parameters for each architecture. We count the total parameters required for data inference, which include the additional 6 task experts with a combined parameter size of 654M parameters in our Prismer model.
4 Experiments
4.1 Prismer Model Variants
In addition to Prismer, we also introduce model variant named PrismerZ, which solely relies the power of strong backbone experts and is trained with zero task experts. PrismerZ has the same architectural design as the original Prismer but without the Experts Resampler. PrismerZ simplifies the data inference process as it only requires RGB images, making it more efficient and applicable to a wider range of applications. Prismer is less efficient in data inference due to the need for data processing on expert labels, but as we will show, it has better predictive performance.
Both Prismer and PrismerZ utilise ViT (Dosovitskiy et al. 2020 pre-trained by CLIP (Radford et al. 2021 as the frozen vision encoder, and RoBERTa (Liu et al. 2019c as the frozen language decoder. We have also tried using two other popular open-sourced decoder-only autoregressive language models: OPT (Zhang et al. 2022) and BLOOM (Scao et al. 2022 but early experiments showed that they did not perform as well.
We experiment with two model sizes, BASE and LARGE. The BASE model is based on ViT-B/16 and RoBERTasasE; and the LARGE model is based on ViT-L/14 and ROBERTaLARCE: In Prismer, we apply the same Experts Resampler with roughly 50M parameters in both model sizes. The architecture details are further summarised in Table 1
4.2 Training and Evaluation Details
Pre-training Datasets We construct our pre-training data from the following datasets: two in-domain datasots: COCO [Lin et al. 2014) and Visual Genome (Krishna et al. 2017]; and thrce web datascts: Conceptual Captions (Sharma et al. 2018). SBU captions (Ordonez et al. 2011). and a much noisier Conceptual 12M (Changpinyo et al. 2021 The web datasets are pre- filterrd and re-captioned by a pre- trained image captioner (Li ct al.] 2022). The pre-training datasets include 11M unique images or 12.7M
image/altes alx-lext pairs All dalasets are available publicly and have been widely used for pre-training many
VLMSs (Li et al. 2021
2022
Chen et al. 2020
Optimisation and Implementation All our models are trained with AdamW optimiser (Loshchilov & Hutter 2019) with weight docay of 0.05. Since only a small proportion of the model paramcters are trainable, model sharding only applied during fine-tuning on large-reoolution images. Specifically, we employ ZeRO Stage 2 technique (Rajbhandari et al.) 2020), which enables the sharding of optimiser states and parameter gradients across all GPU instances. Additionally, we also apply Automatic Mixed Precision (AMP) with fp16 precision to further reduce training time. For more details on our data processing techniques and hyper-parameter choices, please refer to Appendix B An analysis of training and inference costs compared Lo other vision- languae models can be found in Appendix C
Thhis slightly less than the theoretical number which should 14M unique images. because some image URLs in the web datasets are valid during the time we downloaded the datasets.
7

---

Published in Transactions on Machine Learning Research (01/2024)
Pre-train Pairs)
COCO Caption
NoCaps
VQAv2
B 1
M
C
In
Near
Out
Overall
test-dey test-std
OSCARlaa
Li
6.5M
36.5
123.7
23.1
83.4
77.6
73.2
VinVTaas
Zhang et
202
8.9M
38.2
30.3
129.3
23.6
103.7
95.6
83.8
94.3
76.0
76.1
GiTaa
Wang et a
2022a
10M
40.4
30.0
131.4
23.0
100.7
97.7
89.6
96.6
72.7
BLiFPaase
al.
129M
39.7
133.3
111.8
108.6
111.5
109.6
78.3
78.3
L.MON
2022
200M
40.3
30.2
133.3
23.3
107.7
106.2
107.9
106.8
PrismerZas
12.7M
39.7
31.1
133.7
24.1
108.7
107.8
105.8
107.5
76.6
Prismersue
12.7M
40.1
31.1
135.1
24.1
108.8
108.3
111.7
109.1
76.8
77.0
OSCA
[Ti al. 20206
6.5M
37.4
30.7
127.8
23.5
85.4
84.0
80.3
83.4
73.4
73.8
VinVLass
Zhang et 6
2021
8.9M
38.5
30.4
130.8
23.4
76.5
76.6
Wang et al 20226
20M
42.0
30.8
138.5
23.8
107.7
107.8
102.5
75.5
BL LANgE
2022
129M
40.4
136.7
114.9
115.3
LEMONuse
Hu
2022
200M
40.6
30.4
135.7
23.5
116.9
113.3
111.3
113.4
PrismerZu.mm
12.7M
40.0
31.2
135.7
24.2
112.3
111.2
112.8
111.8
77.5
Prismerlace
12.7M
40.4
31.4
136.5
24.4
114.2
112.5
113.5
112.9
78.4
78.5
LEMON
Hu ct al.
2022
200M
41.5
30.8
139.1
24.1
118.0
116.3
120.2
117.3
SimVLM
Waiy
2021
1.8B
40.6
33.7
143.3
25.4
113.7
115.2
112.2
80.0
GIT Wang
0.8B
44.1
31.5
144.8
24.7
129.8
124.1
127.1
125.5
78.6
78.8
GIT-2
Wang et
2022a
12.9B
41.1
31/4
145.0
24.8
126.9
125.8
130.6
126.9
81.7
81.9
CoCa
Tu aL. 2022
4.8B
40.9
33.9
143.6
24.7
122.4
82.3
82.3
PaL.1
Chen et al.
2022
1.6B
149.1
127.0
84.3
84.3
Table 2: Fine-tuned performance on COCO Caption (Karpathy split), NoCaps (validation set)
and VQAv2. Both Prismer and Prismer'Z achieve superior performance in all three datasets compared to other VLMs with similar model sizes. Prismer can achieve competitive performance on par with VLMs that are trained with orders of magnitude more data. (B04, M, C, S} refer to BLEU@4, METEOR, CIDEr, SPICE respectively. {In, Ncar, Out} refer to in-domain, near-domain and out-of-domain respectively. i evaluates the VQAv2 dataset in a generative setting. All other models evaluate the VQAv2 dataset in a closed-ended discriminative setting.
Evaluation Setting We evaluate the performance of our models through generative language modelling, which is a challenging lask than discriminative learning (particularly in VQA tasks), and aligns with that used in other vision-language generative models (Li et al. 2022 Alayrac et al. 2022; Wang et al. 2022a Chen et al. 2023). For example, the model must accurately generate all text tokens for a question (which is average 2.2 tokens per question in the VQAv2 dataset (Antol et al. 2015) as reported in (Wang et al, 2022a)), ), rather than just one correct prediction as required in discriminative models.
Specifically, we evaluate image captioning tasks in an open-ended generative setting, and we apply beam search with a beam size of for text generation. A prefix prompt of "A picture of" is added to the input text for fined-tuned image captioning tasks, similar to previous studics in (Wang et al. 2021 Li et al. 2022 Radford et al. 2021), which have shown to improve the quality of image captions. We evaluate both VQA and image classification tasks in a close-ended generative setting, by ranking the per-token log-likelihood from a pre-defined answer list.
4.3 Results on Vision-Language Benchmarks
Fine-tuned Performance on COCO Caption, NoCaps and VQAv2 We fine-tune our models on COCO Caption dataset (Chen et al. 2015) on a widely adopted Karpathy split (Karpathy & Fei-Fei 2015), with the standard cross-entropy loss, and without metric-speeific optimisation (Vedantam et al.|| Wo evaluate the fine-tuned models on the COCO Caption Karpathy test. split and NoCaps (Agrawal et al. 2019 validation set. We also evaluate our models the VQAv2 dataset (Antol al. 2015). with additional training samples from Visual Genome (Krishna et al., 2017) following (Li et al.] 2022). We compare our models with prior state-of-the-art VLMs that are mostly pre-trained on image-text data for a fair comparison. We sort all VLMs by their model sizes and report the results in Table 2
The results show that both Prismer and PrismerZ achieve superior performance considering their model sizes, which suggests that the strong backbone experts are primarily responsible for good generalisation. However, the task experts provide additional boost in performance, particularly in image captioning tasks
8

---

Published in Transactions on Machine Learning Research (01/2024)
NoCaps
80
Top-I Ace. (%)
70
VL. (Jin et al.
2022
47.7
9.1
60
MetaLM Hao al.
2022
58.7
8.6
VLKD
Dai al.
2022
63.6
12.8
50
PrismerBEE
SimVLMgaa
[Wang
2021
101.4
Prismen.uux
BLIP-2 (Vicuna-7BB
Li ct
2023
107.5
40
Flamingo
VIT-B/16
BLIP-2 (Vicuna-13B)
et a.
2023
103.9
30
GIT
VIT-1,114
Prismergase
87.5
13.0
4
16
PrismerJa3GE
107.9
14.8
shols per class
Figure 4: Results on zero-shot image captioning and few-shot ImageNet classification, Left: Prismer achieves state-of-the-art zero-shot image-captioning results on NoCaps (validation set), with similar performance to SimVLM and BLIP-2, despite being trained on 140 times and 10 times less data respectively. Prismer significantly improves few-shot, performance compared to its corresponding vision backbone. However, Prismer still underperforms GIT and Flamingo which are trained on significantly more data.
Right:
(such as a 6 CIDEr score increase in the NoCaps out-of-domain set in the BASE model) and in the LARGE model variant (such as a 1 VQAv2 accuracy increase in the LARGE model). Both Pгismers4нE and Prismer achieve comparable image captioning performance to BLIP (Li et al. 2022 and LEMON (Hu et al. 2022), despite being trained on and 20 times less data, respectively. Additionally, the PrismeTLARGE model has achieved VQAv2 accuracy comparable to GIT (Wang et al. 2022a). despite being trained on 60 times less data. Whilst acknowledge noticeable performance gap between Prismer and the current. state-of-the-arl. VLMs (such as CoCa (Yu et al. 2022), GIT-2 (Wang et al. 2022a) and PaLl (Chen et al. 2023 ) these models require substantially higher training costs and access to large-scale private training data.
Zero-shot Performance on Image Captioning Our generative pre-training approach allows for zero- shot generalisation, where the models can be directly applied to image captioning tasks without additional fine-tuning. In Fig. 4 Left, we show that Prismer achieves state-of-the-art performance on the NoCaps datasot similar SimVLM (Wang al. 2021) and BLIP-2 (Li al., 2023), whilst using significantly smaller network parameter size and trained with 140 times and 10 times less data respectively. Additionally,
notice that the zero-shot performance of Prismer models even surpasses the fine-tuned performance of certain VLMs
such as OSCAR (Li et al. 2020b and VinVL Zhang et al. 2021 as shown in Table 2
We present a list of example captions generated by Prismer in Table 3 The results show that both PrismeTgASE and PrismerLARGE are capable generating captions that are semantically coherent and aligned with the visual content of the images. Notably, PrismOTLARGE generates captions of higher quality compared to PrismeтBaSE, exhibiting a deep understanding of fine-grained object semantics such as brand recognition (e.g. Mercedes, CK One), and cultural concepts (e.g. vintage drawing, tango), indistinguishable to human-write captions.
Few-shot Performance on ImageNet Classification Finally, we fine-tune and evaluate Prismer on ImageNet (Deng et al.) 2009) in a few-shot setting. Following the approach outlined in (Radford et al.) 2021), we convert the classification task into a language modelling problem by mapping each category a template caption: "A photo of [CLASS NAME ", and we then score captions using the likeliill estimated by our model. Unlike Flamingo (Alayrac et al., 2022) which performs few-shot classification via in-context examples without gradient updates, we perform few-shot classification via lightweight fine-tuning following (Wang et al. 2022a). This is more similar to the standard linear probe setting, by considering the entire language decoder as an image classifier. Accordingly, we also compare with the few-shot linear probe performance of Prismer's 's original vision backbones ViT-B/16 16 and ViT-L./14 (Dosovitskiy et al. 2020 as
reported in (Schuhmann et al. 2022 Radford et al. 2021).
From the results shown in Fig.
Right, we observe that Prismer underperforms GIT (Wang et al. 2022a and
Flamingo (Alayrac et al. 2022 which both have stronger and larger vision backbones and are pre-trained
9

---

Published in Transactions on Machine Learning Research (01/2024)
Ground-Truth
Prismerasse
Prismer
A clear bottle of cologne is full of liuuid.
ck
bottle of alcohol sitting computer keyboard.
A bottle of ck nent computer keyboord.
The bottle of perfume is made by Calvin Klein,
statue has large purple headdress on
A vavveeuuiin dressed in black dress with feathers her head.
The woman is wearing a black dress.
2. woman decorated in fashioned clothing and relics.
The Next 125!
1. A new white car with the door open is in showroom full of people.
A white car display at show.
A white mercedes on display at an auto show.
2. A shiny white mercedes car is displog.
Large piece meat with slices of pincupple with chervics being held with toothpicks blue and white plate.
Pineapple upside down cake blue and white plate.
Pineapples plate.
2. cake has several slices of pinenpple and cheries them.
A and woman is dancing crowd watches them in the distance.
couple of people that standing the dirt.
A couple dancing tango front crowd.
2. woman in red dress dancing with bold man wearing black.
1. Two illustrations of lobster colors are shown as Fig. 21 and Fig. 22.
A vintage illustration of lobsters from the 19th centurg.
Colored drawing of two lobsters on pink paper.
2. drawing of a lobster and lobster.
Man in skyddiinn gear giving two thambss up with skydseers in the sky bekind him.
man wenring a helmet and goggles with parachates in the background.
Man wearing blue and purple jacket.
Person giving double thumbss up sign while others parachating background.
Table 3: Visualisation of zero-shot image captioning NoCaps. Prismer produces more detailed and semantically coherent captions than PrismereaB., showing an understanding of fine-graines object recognition and abstractions. Results are not cherry-picked.
10

---

Published in Transactions on Machine Learning Research (01/2024)
significantly more data. However, Prismer still outperforms its original vision backbones ViT-B and ViT-L by large margin, especially in a very few-shot setting, despite having the exact same representation space. This suggests that Prismer's generalisation abilities are enhanced by the multi-modal training data and expert labels, and its performance can likely be improved further by using an even stronger vision backbone.
5 Additional Analysis
We now include a comprehensive evaluation of Prismer, characterised by meticulous and fine-grained analysis of its learning strategy. We delve into various aspects of Prismer's performance, examining its behaviour with different types of multi-task experts (as discussed in Sec 5.1 Additionally, we explore the individual utility of each expert in addressing domain-specific reasoning tasks, allowing us to gain insights into the specific strengths and contributions of each expert (as discussed in S.2
5.1 Intriguing Learning Strategy of Prismer
To speed up training, all experiments are conducted with the BASE model on a combined dataset of the Conceptual Captions and SBU, consisting of total 3M data. All experiments are evaluated on the VQAv2 test-dev split in a smaller [224 224] resolution.
More Experts, Better Performance We observe that the performance of Prismer improves with more task experts, as shown in Fig. 5a This is intuitive because more experts provide greater diversity of domain knowledge to the model. However, we also note that the performance of the model eventually plateaus, which suggests that additional task experts beyond a certain number do not provide any extra gains.
Better Experts, Better Performance To evaluate the impact of expert quality on Prismer's performance, construct corrupted depth expert by replacing certain number of predicted depth labels with random noise sampled from Uniform Distribution. As shown in Fig. 5b Prismer's performance improves as the quality of the depth expert improves. This is intuitive as better experts provide more accurate domain knowledge, allowing the model to perceive more accurately.
Robustness to Noisy Experts Our results also demonstrate that Prismer maintains performance even when including experts that predict noise, as shown in Fig. 5c Interestingly, adding noise can even result in non-trivial improvement compared to training on RGB images alone, which can be considered as form of implicit regularisation. This property allows the model to safely include many experts withoul degrading the performance, even when the expert is not necessarily informative. Therefore, Prismer presents a more effective learning strategy than the standard multi-task or auxiliary learning methods, which either require exploring task relationships (Liu al.
2022
Fifty et al. 2021 Zamir et al. 2018 or designing more advanced
optimisation procedures (Liu et al.) 2019a
Navon et al.
2021
5.2 Utility of Task Experts
In this experiment, we conduct a comprehensive evaluation to assess the utility of each task expert within Prismer concerning different types of reasoning tasks. To accomplish this, we employ Prismer which was trained on the VQAv2 dataset, and evaluate its zero-shot performance combination with cach individual lask expert on two specific domsin-specific reasoning lasks: i) Visual Spatial Reasoning (VSR) (Liu al.) 2023): This task evaluates VLM's spatial reasoning ability. It involves classifying image-caption pairs either true or false, indicating whether the caption correctly describes the spatial relation in an image. ii) Text-VQA (Singh et al. 2019): This task assesses a VLM's ability to understand and reason about text within an image. It. involves comprehending and answering questions related to text in an image.
The results presented in Table 4 demonstrate that Prismer consistently outperforms several competitive basclines, such as VisualBERT (Li et al.) 2020aa, LXMERT (Tan Bansal 2019), and ViLT (Kim al. 2021 in the VSR dataset, all without requiring dataset-specific fine-tuning required by these methods. Prismer
11

---

Published in Transactions on Machine Learning Research (01/2024)
Acc. (%)
Ace. (%)
Acc.
(%)
73.0
73.0
72.79
72.82
72,64
72.58
72.52
72.5
72.5
72.44
72.5
72:17
72.17
72.17
72.0
72.0
72.0
IGB
-2 Exps
+44 Exps
+6 Exps
RGB
+Depth
+Depth
+Depth
RGB
+Noise
+
+Depth
-Depth & Noise
25% N.
10% N.
No
(a) with More Experts
(b) with Better Exports
(c) with Noisy Experts
Figure 5: Prismer's VQAv2 accuracy with different types and the number of experts. Prismer has shown that its performance improves with an increase in the number and quality of task experts. Additionally, Prismer also demonstrates its strong robustness to noisy experts, making it a practical and effective multi- modal learning strategy.
Baselines (fine-tned)
Prismer (xero-shot)
VisualBERT LXMERT
VILT
Depth
Normal
+ Edge
+Seeg.
OCR Dec.
+Obj. Det. No Experts +-6 Experts
51.0
61.2
63.0
68.4
68.3
67.8
68.4
67.2
68.3
65.6
(a) VSR
Baselines (xcro-shot)
Prismer (xero-shot)
OFA
BLIP-2
Flamingo
+ Depth
+ Normal
+ +Edge
+Seg.
OCR Det. +Obj.
Det. No Experts 4-6 Experts
18.3
15.7
35.0
27.4
28.0
28.2
27.8
28.4
28.4
22.6
28.8
(b) Text-VQA
Table 4: Zero-shot accuracies in VSR (zero-shot split) and Text-VQA (validation split) datasets, considering various types of experts. These results shed light on the valuable contributions of individual experts for domain-specifie reasoning tasks, offering insights into the versatility and adaptability of Prismer across different domains and problen-solving scenarios. The colour green represents the most helpful experts, while the colour red represents the least helpful experts.
also surpasses BLIP-2 (OPT 2.7B] (Li et al. 2023) and OFASuCE (Wang al. 2022b), despite employing smaller backbone network and significantly less pre- pre-training data respectively.
Furthermore, Prismer's utility analysis offers valuable insights into the contributions of individual experts in addressing specific reasoning lasks. For example, the "object detection" expert identified as crucial in both the VSR and Text-VQA tasks, highlighting the significance of object recognition capability general visual reasoning problems. Additionally, the "depth" and "OCR detection" experts are recognised as key contributors to Prismer's 's performance in spatial reasoning and text reasoning, respectively, aligning with human intuition depth information enhances 3D spatial understanding, whilst OCR detection directly improves text reading capability.
Finally, the substantial performance improvement observed (compared to general reasoning tasks presented in Table 2 when comparing Prismer to Prismer'Z (with no experts) underscores the pivotal role played by the experts in domain-speciffc reasoning tasks. This highlights the tangible benefits of incorporating experts within the Prismer architecture, particularly when tackling tasks that require specialised knowledge and reasoning capabilities.
6 Conclusions, Limitations and Discussion
In this paper, we have introduced Prismer, a vision-language model designed for reasoning tasks. Prismer is parameter-officient and utilises small number of trainable components to connect an ensemble of diverse, pro- trained experts. By leveraging these experts, Prismer achieves competitive performance image captioning, VQA, and image classification benchmarks, comparable to models trained up to two orders of magnitude
12

---

Published in Transactions on Machine Learning Research (01/2024)
more data. For full transparency, now discuss some limitations of Prismer during our implementation and explore potential future directions for this work.
Multi-modal In-context Learning Zero-shot in-conlext generalisatiinn is an emergent property that only exists in very large language models (Brown et al., 2020 Wei et al. 2022). In this work, we build Prismer on top of a small-scale language model with the main focus on parameter-offiiiott learning. Therefore, it does not have the ability to perform few-shot in-context prompting by design.
Zero-shot Adaptation on New Experts We experiment with inference on a pre-trained Prismer with a different segmentation expert pre-trained on different dataset. Although we apply the same language model to encode semantic labels, Prismer shows limited adaptability different expert with different set of semantic information, which leads to notable performance drop.
Free-form Inference on Partial Experts Similarly, discover that Prismer entangles its multi-task features from all experts we include during pre-training. Therefore, only having partial number of experts during inference will lead to a notable performance drop. We attempt to use a different training objective such as masked auto-encoding (Bachmann et al. 2022), ; to design Prismer to reason on an arbitrary number of experts, but eventually leads to a degraded fine-tuned performance.
Representation of Expert Knowledge In our current design of Prismer, we convert all expert labels into an imago-like 3-dimensional tensor via task-spocific post-procsssing for simplicity. There are other efficient methods represent expert knowledge, such as converting object detection into a sequence of text tokens [Chen et al. 2021 2022). This may lead to stronger reasoning performance and more stable training landscape in future works.
References
Harsh Agrawal, Karan Desai, Yufei Wang, Xinlei Chen, Rishabh Jain, Mark Johnson, Dhruv Batra, Devi Parikh, Stefan Lee, and Peter Anderson. Nocaps: Novel object captioning at scale. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katic Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for fow-shot learning. In Advances in Neurul Information Processing Systems (NeurIPS). 2022.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the International Conference Computer Vision (ICCV), 2015.
Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. MultiMAE: Multi-modal multi-task masked autoencoders. In Proceedings of the European Conference on Compuler Vision (ECCV), 2022.
Gwangbin Bae, Ignas Budvytis, and Roberto Cipolla. Estimating and exploiting the aleatoric uncertainty in surface normal estimation. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, et al. Gpt-neox-200 20b: An open-source autoregressive language model. arXiv preprint ar Xiv:2200 06745, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
Holger Cacsar, Jasper Uijlings, and Vittorio Ferrari. Coco-stuff: Thing and stuff classes in context. In Proceedings of the IEF. Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
13

---

Published in Transactions on Machine Learning Research (01/2024)
Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
Ting Chen, Saurabh Saxena, Lala Li, David J Fleet, and Geoffrey Hinton. Pix2seq: A language modeling framework for object detection. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Ting Chen, Saurabh Saxena, Lala Li, Tsung- Yi Lin, David J. Fleet, and Geoffrey Hinton. A unified sequence interface for vision tasks. Tn Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems (NeurIPS), 2022.
Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, et al. Pali: jointly-scaled multilingual languago-image model. In Proceedings of the International Conference on Learning Representations (ICLR), 2023.
Xinlei Chen, Hao Fang, Tsung-Yi Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco captions: Data collection and evaluation server. arXiv preprint 1r00v 00325, 2015.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, The Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In Proceedings of the European Conference on Computer Vision (ECCV), 2020.
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE Conference Compuler Vision and Pottern Recognition (CVPR), 2022.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal, Unifying aasion-and-language tasks via text generation. In Proceedings of the International Conference Machine Learning (ICML), 2021.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Scbastian Gchrmann, et al. Palm: Scaling language modcling with pathways. arXic preprint arXiv:2204. 02311, 2022.
Ekin D Cubuk, Barret Zoph, Jonathon Shlens, and Quoe V Le. Randaugment: Practical automated data augmentation with a reduced search space. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
Angela Dai, Angel X Chang, Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, and Pascale Fung. Enabling multimodal generation on clip via vision-language knowledge distillation. In Proceedings of the Associalion for Compulational Linguistics (ACL), 2022.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2009.
Jacob Devlin, Ming-Wo- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pro-training of deep bidiroctional transformers for language understanding. In Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies (Long and Short Papers), 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Mindcrer, Georg Heigold, Sylvain Gelly, et al. An image worth 16x16 words: Transformers for image recognition at scale. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
14

---

Published in Transactions on Machine Learning Research (01/2024)
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and-languuage transformers. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Constantin Eichenberg, Sidney Black, Samuel Weinbach, Letitia Parcalabeseu, and Anctte Frank. Magma- multimodal augmentation of generative models through adapler-based finetuning. arXiv preprinl arXiv:2112.05253, 2021.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. In Advances in Neural Information Processing Systems (NeurIPS), 2021.
Zhe Gan, Yen-Chun Chen, Linjic Li, Chen Zhu, Yu Cheng, and Jingjin Liu. Largo-scale adversarial training for vision-and -ad--anndddnaaee representation learning. Advonces in Neural Information Processing Systems (NewrIPS), 2020.
Golnaz Ghiasi, Barret Zoph, Ekin D Cubuk, Quoc V Le, and Tsung- Yi Lin. Multi-task self-training for learning general representations. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Tanmay Gupta and Aniruddha Kembhavi. Visual programming: Compositional visual reasoning without training. In Proceedings of the IEEE Conference on Computer Vision and Pottern Recognition (CVPR), 2023.
Yaru Hao, Haoyu Song, Li Dong, Shaohan Huang, Zewen Chi, Wenhui Wang, Shuming Ma, and Furu Wei. Language models are general-purpose interfaces. arXiv preprint arXie:2206.0633 2022.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-cffciint transfer learning for nlp. In Proceedings of the International Conference on Machine Leurning (ICML), 2019.
Xiaowci Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, and Lijuan Wang. Scaling up vision- languae pre-training for image captioning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognation (CVPR), 2022.
Andrew Jaegle, Felix Gimeno, Andy Brock, Oriol Vinyals, Andrew Zisserman, and Joao Carreira. Perceiver: General perception with iterative attention. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoe Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In Proceedings of the International Conference on Machine Learning (ICML). 2021.
Woojeong Jin, Yu Cheng, Yelong Shen, Weizhu Chen, and Xiang Ren. A good prompt is worth millions of parameters: Low-resource prompt-based learning for vision-languag models. In Proceedings of the Association for Computational Linguistics (ACL), 2022.
Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Iodar 2015 competition on robust reading. In International Conference on Document Analysis and Recognition (ICDAR), 2015.
Andrej Karpathy and Li Fei-Fei. Deep visual-semanntc alignments for generating image descriptions. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2015.
Ronald Kemker, Mare McClure, Angelina Abitino, Tyler Hayes, and Christopher Kanan. Mcasuring catastrophic forgetting in neural networks. In Proceedings of the National Conference on Artificial Intelligence (AAAT), 2018.
15

---

Published in Transactions on Machine Learning Research (01/2024)
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-and-languaae transformer without convolution region supervision. In Proceedings of the International Conference on Machine Learning (TCMT.), 2021.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnicszka Grabska- Barbinsk,a et al. Overcoming catastrophic forgelting in neural networks. Proceedings of the national academy of sciences, 2017.
Ranjay Krishma, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and Li Fei-Fei. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International Journal of Computer Vision (IJCV), 2017.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4. International Journal of Computer Viston (IJCV), 2020.
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Proceedings of the International Conference on Machine Learning (ICML), 2022.
Junman Li, Dongxu Li, Silvio Savarese, and Steven Hoi. BLIP-2: bootstrapping language-image pre-training with frozen image encoders and large language models. In Proceedings of the International Conference on Machine Learning (ICML), 2023.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai- Wei Chang, What does BERT with vision look at? In Proceedings of the Association for Computational Linguistics (ACL), 2020a.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semantics aligned pre-training for vision-language tasks. In Proceedings of the Europeon Conference on Computer Vision (ECCV), 2020b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proceedings of the Europeon Conference on Computer Vision (ECCV), 2014.
Fangyu Liu, Guy Edward Toh Emerson, and Nigel Collier. Visual spatial reasoning. Transactions of the Association for Computational Linguistics, 2023.
Hanxiao Liu, Zihang Dai, David So, and Quoc V Le. Pay attention to mlps. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vanghan (ods.), Advances in Neural Information Processing Systems (NewrIPS), 2021.
Shikun Liu, Andrew J Davison, and Edward Johns. Self-supervised generalisation with meta auxiliary learning. In Advances in Neural Information Processing Systems (NeurIPS), 2019a.
Shikun Liu, Edward Johns, and Andrew Davison. End-to-end multi-task learning with attention. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019b.
Shikun Liu, Stephen James, Andrew Davison, and Edward Johns. Auto-lambda: Disentangling dynamic task relationships. Transactions on Machine Laerning Research (TMLR), 2022.
Wei Liu, Chaofeng Chen, and Kwan- Yee Wong. Char-net: A character- aware neural network for distorted scene text. recognition. In Proceedings of the National Conference on Artificial Intelligence (AAAT), 2018.
16

---

Published in Transactions on Machine Learning Research (01/2024)
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint Xr:i::1907. 10007..1.99... 2019c.
Tly Loshchilov and Frank Hutter. Decoupled weight decay regularization. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Loc. Vilbert: Pretraining task-agnostic visiolinguistic repre- sentations for daiion-and-language tasks. Advances in Neural Information Processing Systems (NeurIPS), 2019.
Jiasen Lu, Christopher Clark, Rowan Zellers, Roozbeh Mottaghi, and Aniruddha Kembhavi. Unified-io: A unified model for vision, language, and multi-modal tasks. In Proceedings of the International Conference on Learning Representations (ICLR), 2022.
Saeed Masoudnia and Reza Ebrahimpour. Mixture of experts: a literature survey. The Artificial Intelligence Review, 2014.
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
Aviv Navon, Idan Achituve, Haggai Maron, Gal Chechik, and Ethan Fetaya. Auxiliary learning by implicit differentiation. In Proceedings of the International Conference on Learning Representations (ICLR), 2021.
Gerhard Neuhold, Tobias Ollmann, Samuel Rota Bulo, and Peter Kontschieder. The mapillary vistas dataset for semantic understanding of street scenes. In Proceedings of the International Conference on Computer Vision (ICCV), 2017.
Hien D Nguyen and Faicel Chamroukhi. Practical and theoretical aspects of mixture-of-cxperts modeling: An overview. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 2018.
Vicente Ordonez, Girish Kulkarni, and Tamara Berg. Im2text: Describing images using 1 million captioned photographs. Advances in Neural Information Processing Systems (NewrIPS), 2011.
Jonas Pfeiffer, Andreas Rücklé, Clifton Poth, Aishwarya Kamath, Ivan Vulié, Scbastian Rudor, Kyunghyun Cho, and Iryna Gurevych. Adapterhub: A framework for adapting transformers. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2020.
Xavier Soria Poma, Edgar Riba, and Angel Sappa. Dense extreme inception network: Towards a robust enn model for edge detection. In IEEE Winter Conference on Applications of Computer Vision (WACV), 2020.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.114466 2021.
Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, 2020.
René Ranftl, Alexey Bochkovskiy, and Vladlen Koltun. Vision transformers for dense prediction. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
Carlos Riquelme, Joan Puigeerver, Basil Mustafa, Maxim Neumann, Rodolphe Jenatton, André Susano Pinto, Daniel Keysers, and Neil Houlsby. Scaling vision with sparse mixture of experts. Advances in Neural Information Processing Systems (NeurIPS), 2021.
17

---

Published in Transactions on Machine Learning Research (01/2024)
Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilié, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, Frangois Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. arXiv preprint arXiv:2211.05100, 2022.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LA1ON-5b: An open largo-scale datasot for training next generation imago-text models. In Advances in Neural Information Processing Systems (NewrIPS), 2022.
Shuai Shao, Zeming Li, Tianyuan Zhang, Chao Peng, Gang Yu, Xiangyu Zhang, Jing Li, and Jian Sun. Objects365: A large-scale, high-quality dataset for object detection. In Proceedings of the International Conference on Computer Vision (ICCV), 2019.
Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu Sorieut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning, In Proceedings of the Association for Computational Linguisties (ACL), 2018.
Amanprect Singh, Vivek Natarjan, Moet Shah, Yu Jiang, Xinlci Chen, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
David So, Wojcioch Manke, Hanxiao Liu, Zihang Dai, Noam Shazcer, and Quoe V Lc. Searching for efficient. transformers for language modeling. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. Which tasks should be learned together in multi-task learning? In Proceedings of the International Conference Machine Learning (ICML), 2020.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic visual-linguistic representations. In Proceedings of the International Conference on Learning Representations (ICLR), 2020.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems (NeurIPS), 2020.
Didac Surís, Sachit Menon, and Carl Vondrick. Vipergpt: Visual inference via python execution for reasoning. In Proceedings of the International Conference on Computer Vision (ICCV) 2023.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers. Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
Maria Tsimpoukelli, Jacob L Menick, Serkan Cabi, SM Eslami, Oriol Vinyals, and Felix Hill. Multimodal few-shot learning with frozen language models. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Ramakrishna Vedantam, C LLawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. Transactions on Machine Laerning Research (TMLR), 2022a.
18

---

Published in Transactions on Machine Learning Research (01/2024)
Peng Wang, An Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to- sequence learning framework. In Proceedings of the International Conference on Machine Learning (ICML), 2022b.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvlm: Simple visual language model pretraining with weak supervision. In Proceediiggs of the International Conference Learning Representations (ICLR), 2021.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. Emergent abilities of large language models. Transactions on Machine Laerning Research (TMLR), 2022.
Tele Xiao, Mannat Singh, Eric Mintun, Trevor Darrell, Piour Dollár, and Ross Girshick. Early convolutions help transformers see better. Advances in Neural Information Processing Systems (NeurIPS), 2021.
Dan Xu, Wanli Ouyang, Xiaogang Wang, and Nicu Sebe. Pad-net: Multi-tasks guided prediction-anc distillation network for simultaneous depth estimation and scene parsing. In Proceedings of the IEFF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. Transactions on Machine Laerning Research (TMLR), 2022.
Amir R Zamir, Alexander Sax, William Shen, Leonidas J Guibas, Jitendra Malik, and Silvio Savarese. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Andy Zeng, Adrian Wong, Stefan Welker, Krzysztof Choromanski, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, et al. Socratic models: Composing zcro-shot multimodal reasoning with language. arXiv preprint arXim:2204.00500 2022.
Xiaohua Zhai, Xiao Wang, Basil Mustala, Andreas Steiner, Daniel Keysers, Alexander Kolesnikov, and Lucas Beyer. Lit: Zero-shot transfer with locked-image text tuning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2022.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Proceedings of the IEEE Conference Computer Vision and Pottern Recognition (CVPR), 2021.
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Lú, Xi Victoria Lin, et al. Opt: Open pre-trained transformer language models. arXic preprint ar 2205 01068, 2022.
Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühl. Simple multi-dataset detection. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
Xizhou Zhu, Jinguo Zhu, Hao Li, Xiaoshi Wu, Hongsheng Li, Xiaohua Wang, and Jifeng Dai. Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks. In Proceedings of the IEEE Conference on Computer Vision and Pallern Recognilion (CVPR), 2022.
19

---

Published in Transactions on Machine Learning Research (01/2024)
A Detailed Description of Experts
In Table 5
we present detailed list of task experts we have included in the Prismer model. We report cach expert's 's pre-trained domain, as well as its pre-trained dataset, parameter size, and post-processing techniques.
Task
Datasct
Model
Params.
Post-Proccssing
In-paint each pixel with its corresponding label
Semantic Segmentation
Mask2Former Cheng
COCO-Stuff
(Caesar al.
2018
215M
et al. 2022
parametrised by CLIP language
embedding.
In-paint each pixel with its corresponding label parametrised by CLIP language embedding. The labels for the overlapping pixels are further determined by the depth expert.
COCO
Lin al.
2014
Objects365
Shao
2019
Uni (Zhou et
Object Detection
OpenInages
Kuznelsova
2020
2022
120M
Mapillary
Neuhold ot
In-paint each pixel with its corresponding text parametrised by CL.IP language embeddingg
CharNet Liu et al.
Text Detection
ICDAR 2015
Karatzas et al.
2015
89M
2018
DPT
Ranftl al.
Depth Estimation
MIX-6
Ranftl et al.
2021
123M
Re-normalised to [-1, 1].
2021
NL..-AngMF
Bae
Surface Normal
ScanNet
Dai et
2017
72M
Re-normalised to [-1, 1].
al.
2021
DexiNed
(Poma
Edge Detection
BIPED
Poma al.
2020
35M
Re-normalised to [-1, 1].
et al. 2020
Table The detailed description of task experts. We provide a detailed description of each task expert including its pro-trained dataset, paramcter size, model name and type and its post-proccssing strategy.
20

---

Published in Transactions on Machine Learning Research (01/2024)
B Detailed Training Strategy and
All Prismer model variants are pre-trained with resolution of [224 224] and evaluated on three types of vision-language reasoning tasks: image captioning, visual question answering, and image classification. To improve performance, the models are fine-tuned with larger resolutions: [384 x 384] for image classification, and [448 x 448) for image captioning and visual question answering tasks. Automated data augmentation (Cubuk et al., 22200 is applied during both pre-training and fine-tuning. A list of the hyper- paper-parameters used in the experiments can be found in Table G
For language gencration, beam scarch is applicd with beam size of 3 for all image captioning datasets. Per-token log-likeling regliikei is used to rank answers for the VQAv2 and ImageNet datasels. A prelix prompt of A picture of is added to the input text for image captioning tasks, similar to previous studies such as in Radford et al. )021), as shown to improve the quality image captions.
Wang et al.
2021
Li et al. 2022
Pre-training
COCO NoCaps
VQAv2
ImageNet
Optimiser
AdamW Cosine annealling to zero 0.05
LR. Schedule
Weight Decay
Warmup Steps
2000
0
Initial LR
3/1 - 10-4 (B L)
5 10
5 10-5
5 10-5
Resolution
224
480
480
384
Epochs
20
3
10
20
Batch Size
1024
256
512
64
Table 6: The detailed list of hyper-paramcters and training strategy. To ensure reproducibility, we have included list of all syper-parameters used in our experiments. These same hyper-parameters are applied to both the BASE and LARGE model variants.
C Comparison of Training Cost
Prismer is highly efficient in terms of training and inference costs. Here, the training costs are defined by the costs exclusively involved in constructing the Prismer models, excluding all original gre-training data implied in the original expert models. This definition aligns with the conventions the VLM community. The largest model variant, PRISMETLARCE; only requires days of training on 32 NVIDIA V100 GPUs. This is significantly more efficient than previous state-of-the-art. VLMSs such SimVLM (Wang et al. 2021 which requires days of training 2048 TPUv3, GIT-2 (Wang et al. 2022a) which requires 1.5 months of training on 1500 NVIDIA A100s, and Flamingo (Alayrac et al. 2022 which requires 2 weeks of training on 1536 TPUv4. A detailed breakdown of the pre-training and inference costs can be found in Table 7
Model Params.
Pre-training Data (# Image Text Pairs)
Pre-training Cost (# PFlops Days)
Inference Cost (# TFlops)
BLIP LARGE
583M
129M
22.2
0.17
LinVLMane
1.4B
1.8B
66.91
0.40
GIT
681M
0.8B
45.8
0.37
PaLI
17B
2.3B
450
5.7
Plamingo
80B
2.3B
1.4Kt
23
GIT-2
5.1B
12.9B
5.5kt
2.6
Prismergass
980M
12.7M
0.66
0.20
PrismerLANG,
1.6B
12.7M
1.9
0.38
Table 7: Training and inference cost of vision-languaae models. We compare the training and inference cost of Prismer with several other vision-language models using the approximation method from (Brown) et al. 2020). The symbol represents the training cost estimated by (Chen et al. 2023), and $ represents the training cost estimated by All inference costs are estimated by with input of 256 image tokens and 30 text tokens.
21

---

Published in Transactions on Machine Learning Research (01/2024)
D Ablation Study on Architecture Design and Training Details
To perform the ablation studies, we use the PrismergasE model and train it on the Conceptual Captions and SBU with total of 3M training data. The results of the ablation studies are presented in Table S
Params. (Rel.
Step Time (Rel.)
VQAv2
Ablated Component
Our Setting
Changed Setting
(Acc.)
Prismeruas (our setting with reduced training)
1.00
1.00
72.79
Residual MLP x2
1.04
1.02
72.36
(i) Adapter Design
Residual MLP
Gated Residual ML.P
1.03
1.03
70.34
1/2 1/4
0.95
0.96
72.52
(ii) Adapter Bottleneck Dim. 1
0.93
0.93
71.66
Random Sampling
0.91
0.96
72.24
(ii) Resampler Design
Experts Perceiver
Full Perceiver
1.00
0.90
65.05
Dual Perceiver
1.08
1.02
71.56
1
0.94
0.93
70.61
(iv) Resampler Layers
4
2
0.96
0.96
72.39
6
1.04
1.01
72.78
32
1.00
0.95
72.44
(v) Resampler Latents
64
128
1.00
1.01
70.28
256
1.00
1.06
68.07
Freeze Vision Only
1.00
1.07
70.49
(vi) Pre-training
Freeze Vision and Lang. Freeze Lang. Ouly
1.00
1.05
67.77
All Parameters
1.00
1.15
68.13
Freeze Vision and Lang.
1.00
1.00
71.36
(vil) Fine-tuming
Freeze Vision
Freeze Lang. Only
1.00
1.00
70.37
All Parameters
1.00
1.00
68.69
Table 8: Ablation studies for architecture components and training strategies. We perform ablation studies to evaluate the impact of different architectural components and training strategies on the VQAv2 test-dev performance. We compare the performance of our default setting to other design and training options. The number of parameters and pre-training step time of the changed setting relative to the default setting are reported. To ensure a fair comparison, all experiments are evaluated using a reduced amount of training data and task experts: depth, normal and segmentation.
Adaptor Design: Single and Simple In our ablation study of adaptor designs, as shown in row (i) and (ii) of Table 8 we find that the simplest adaptor design, which consisted of standard residual connection and an encoder-decoder structure, performs the best. We have experimented with more complex designs, such adding additional adaptor the end of each transformer layer incorporating a learnable gating mechanism similar to that in (Liu et al.] 2021). but both have achieved worse performance. We also observe that having a larger bottleneck hidden size for the single adaptor improves performance.
Resampler Design: Auxiliary Knowledge Distillation In ablation study of Experts Resampler designs and different sampling strategies for encoding multi-task signals, as shown in row (iii) (v) of Table we find that keeping the number of resampler layers and latents lightweight is essential for a stable training process. We also experiment with replacing the resampler with a non-learnable random sampling approach, which results in slightly worse performance compared to using the resampler. We attempt to make the resampler more efficient by receiving full signals, including the RGB, before self-altention, but this has resulted in significantly degraded performance. Additionally, we have tried adding an additional resampler the end of the vision encoder, but this design also results in worse performance.
Frozen Backbone to Preserve Web-Scale Knowledge In our experiments on pre-training and fine- tuning whilst freezing different parts of the model, as shown in row (vi) and (vii) of Table g we find that freezing pre-trained parameters is essential for achieving strong performance and avoiding over-fitting and
22

---

Published in Transactions on Machine Learning Research (01/2024)
catastrophic forgetting of the learned web-scale knowledge: Freezing these parameters has also saved significant amount of GPU memory. Even when fine-tuning different downstream tasks, we find that freezing the vision encoder is beneficial (whilst keeping the resampler and adaptors trainable). This observation is consistent with the findings (Zhai et al. 2022), which shows that only fine-tuning the language model with a frozen vision model for vision-language contrastivv learning can achieve much stronger zero-shot. performance.
We the size of pre-training data significantly smaller than the original pre-training data used train these backbone models.
23


Reviews:
[{"review_id": "9rmUubDHET", "reviewer": "Reviewer_adRT", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n- The paper includes quite a thorough ablation study of the different components introduced. Some of this analysis is presented in Section 5, but a lot more details are available in Appendix D. The ablation results have been limited to the BASE model, so it's unclear if some of the results carry on for bigger backbones, but this is understandable, given the cost of running such ablation experiments.\n\n- The paper is well written, easy to follow, and with significant details to replicate the model and training settings. The authors also said to release the code and pre-trained models, which makes the adoption of the work and potential future follow-ups by other researchers much easier.\n\n**Weaknesses**\n- The technical novelty of the paper is quite limited. The only changes introduced to publicly available pre-trained models are the use of adapters (which are already quite popular and well known by the community), and the expert resampler (which is essentially the same as used by Perceiver & Flamingo, as the authors mention).\n\n- The statement in the abstract: \"Prismer achieves fine-tuned and few-shot learning performance which is competitive with\ncurrent state-of-the-arts, whilst requiring up to two orders of magnitude less training data\" is not correctly demonstrated. Prismer uses pre-trained backbone models: CLIP and RoBERTa. The number of training examples used for these doesn't seem to be taken into account when comparing with state-of-the-art methods that train from scratch. See also the comment below, related to training costs depicted in Table 7.\n\n- The fine-tuning results in Table 2 are not very solid. For the BASE size, when data is available BLIP is very often better or very close to the proposed method. For LARGE size, a similar thing happens with LEMON and BLIP. Overall, the benefits of Prismer in front of these other methods is not that clear.\n\n- Fewshot results in the right of Figure 4 can be misleading: Prismer BASE has a total of 940M params, while ViT-B has about 100M params. Although the param count itself is not a perfect proxy, this number correlates well in most of cases with FLOPS/image and ms/image. This needs to be addressed (see requested changes).\n\n- The comparison of training cost in Table 7 (appendix C) can be also a bit misleading, since Prismer uses already pre-trained models in its backbone, and that cost doesn't seem to be factored in, neither in the # of image-text pairs, nor in the # of PetaFLOP x days. For systems (like Prismer) using existing pre-trained models a comparison of inference runtime is a must.\n\n- When adding more experts, we are also adding 1) more parameters and 2) more time complexity. In the ablation studying the effect of adding experts (Section 5.1 and 5.2), a better baseline would be to use a backbone encoder-decoder model with a comparable number of parameters and runtime. Otherwise, it's not clear whether the benefit comes from the \"expertise\" of the new components being added or just using more compute/parameters per image.\n\n- Overall, it's not clear at all what the take-home-message is, for other researchers to build on top of the work.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper presents Prismer, a multimodal Transfromer-basedmodel  (vision-encoder + text-decoder). Both the vision and the text backbones use existing pre-trained models (ViT and RoBERTa, respectively), which are mostly frozen (i.e. not trained further) and that have been augmented with adaptors. The vision encoder also uses inputs from many vision expert models (e.g. depth estimation, object detection, ...; which are also frozen), and adds a new component, the \"Experts resampler\", that is used to post-process the outputs from the expert models.\n\nThe model is trained using a single image captioning loss on a mixture of five datasets: COCO, Visual Genome, Conceptual Captions, SBU captions and Conceptual 12M. It is later evaluated on multiple tasks: image captioning after fine-tuning, zero-shot captioning, and few-shot image classification. The reported results are especially good on Zer-shot image captioning, when compared against existing baselines.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- Please, provide inference time information (e.g. #FLOPS/image, or ms/image on a given hardware) on the Zero-shot table in Figure 4. This is fundamental to put in context (in my opinion) the strongest results from the paper. Likewise, the few-shot results must also be put in context of the inference time, for the same reason (suggestion: provide a plot for 1shot or 5shot with FLOP/image or ms/image in the x-axis, and leave the rest of n-shots for the appendix). This is critical. \n\n- Either remove the \"pre-train pairs\" column from Table 2, or add another column with \"pre-train pairs\" including the number of pre-training pairs used for the CLIP and RoBERTa pre-training. Otherwise, the comparison between methods using existing pre-trained models and approaches that train from scratch are strongly biased. Even better, as stated many times above, try to provide a comparison of inference runtime. This would strengthen the work significantly and make it more transparent.", "broader_impact_concerns": "No concerns.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "R32k4ZaxIQ", "reviewer": "Reviewer_8Dhj", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n1. The idea of tackling vision-language reasoning by ensembling models trained on different tasks is interesting. Compared with solutions that train from scratch or finetune the vision-language model, Primser is considerably cheaper but still have reasonable performance.\n2. Experiments show that Prismer is robust to noise in the experts and gets better when there are more experts.\n\nWeaknesses:\n\n1. The author has not mentioned some important related works, like Unified-IO (https://arxiv.org/pdf/2206.08916.pdf). Unified-IO also casts a bunch of vision-language tasks as sequence-to-sequence learning problems and trained a unified model that can solve all tasks. Different from Prismer, Unified-IO does not depend on the output from pretrained experts. Thus, the author should also compare with Unified-IO.\n2. The author indicated in the limitation section that Prismer is not good at adapting existing experts to new experts. This shows that performance of Prismer is tied to the choice of the experts, which impairs Prismer's flexibility.\n3. As shown in the leftmost part of Figure 5, The accuracy boost obtained by adding more experts is diminishing. Prismer with 6 experts is 0.03 better than Prismer with 4 experts. Moreover, the right part of Figure 4 shows that the performance of Prismer falls significantly behind Flamingo and GIT.\n4. There are not enough ablation study that justifies the design of the adapter. For example, will the model perform worse or better with full-finetuning? In fact, the author mentioned that Prismer is trained on significantly less amount of data than Flamingo / GIT. It is known that full fine-tuning works better than PEFT if we have a large number of data samples. The author can thus compare the performance difference between Prismer + full finetuning and Prismer + adapter.\n\nOther question: The author mentioned that Prismer used RoBERTa as the decoder. By default, RoBERTa is a birectional transformer model. Did you revise RoBERTa to be a unidirectional model with casual masking?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The Prismer model proposed by the paper fuses the outputs from a pool of pretrained experts for vision-language reasoning. The author adopted a transformer encoder-decoder architecture for expert signal fusion, and initialized the encoder with ViT and decoder with RoBERTa. The author also inserted adapters in the network and only finetunes the adapters in Prismer. Experiments show that 1) Prismer achieves state-of-the-art zero-shot image captioning results in NoCaps (as suggested by Figure 4), 2) Prismer is robust to noisy experts, i.e., adding experts that are not necessarily informative won't degrade the performance, 3) the performance of Prismer gets better when there are more experts.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "The author needs to 1) compare with Unified-IO, 2) conduct more ablation study on the design of adapters. See Strengths And Weaknesses for the whole list of weaknesses.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "mXqXpjotmf", "reviewer": "Reviewer_cr8h", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n\n-\tThe paper is well-presented and easy to follow.\n\n-\tThe proposed method offers an alternative approach for a single model capable of multiple tasks by ensembling available pre-trained models on separate tasks. It has the advantage of being less computationally expensive and smaller in size in comparison with large-scale pre-trained models.\n\nWeakness:\n\n-\tAlthough the authors illustrate that task experts contribute to some improvement, the enhancements shown in Table 2 seem relatively marginal. The primary contribution of the paper lies in leveraging pre-trained models on separate tasks; however, it appears to be limited in terms of novelty.\n\n-\tThere is a lack of in-depth analysis regarding the specific impact of each individual task on the rest.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces the Prismer model, which relies on a set of pre-trained models referred to as \"experts.\" These experts leverage diverse visual information, encompassing low-level signals like depth and high-level signals such as objects and semantic labels. The auxiliary knowledge is extracted directly from the network outputs of these individually pre-trained and frozen expert models. Through the connection of these experts using lightweight trainable components, Prismer achieves reduced computational costs and data requirements.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "No", "audience_explanation": "", "requested_changes": "-\tThe using of task experts just brings a little improvement in some specific tasks/datasets. The motivation of the proposed approach could be clearer. What happens if the selected separate tasks are not available or poorly trained?\n-\tGiven the use of cross-attention in the expert resampler, could the authors shed light on how much attention the model allocates to specific experts? Additionally, an explanation of why a particular expert proves beneficial while others do not would enhance understanding.\n-\tThe paper selects six task experts, and it would be valuable to understand the intuition behind these choices. \n-\tTo facilitate easier comprehension for readers, the authors should include the comparison of pre-training cost in the results table.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: In addition to the above, Reviewer cr8h's main reservation concerns novelty: "While the paper has some merits, it appears to be limited in terms of novelty." TMLR's [evaluation criteria](https://jmlr.org/tmlr/editorial-policies.html#:~:text=Evaluation%20criteria,the%20findings%20of%20this%20paper%3F) explicitly state that submissions should be accepted if they meet the "Claims and Evidence" and "Audience" criteria, even if the contributions or significance of the work is modest. All reviewers indicate in their official recommendation that the submission does satisfy the two criteria.
Decision: Accept (poster)