This CVPR paper is the Open Access version, provided by the Computer Vision Foundation.
Except for this watermark, it is identical to the accepted version;
the final published version of the proceedings is available on IEEE Xplore.

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model
Yatai Ji1 *Junjie Wang2 *Yuan Gong1
Lin Zhang3
Yanru Zhu1
Hongfa Wang4
Jiaxing Zhang3
Tetsuya Sakai2²
Yujiu Yang1²
1

Tsinghua University

2

Waseda University{jyt21, gong-y21, zhuyr20}@mails.tsinghua.edu.cn
wjj1020181822@toki.waseda.jp{zhanglin, zhangjiaxing}@idea.edu.cn

3

4

IDEA

Tencent TEG

yang.yujiu@sz.tsinghua.edu.cn

tetsuyasakai@acm.org
hongfawang@tencent.com

Abstract

Multimodal uncertainties
Intra-modal uncertainty

Multimodal semantic understanding often has to deal
with uncertainty, which means the obtained messages tend
to refer to multiple targets. Such uncertainty is problematicfor our interpretation, including inter- and intra-modal uncertainty. Little effort has studied the modeling of this uncertainty, particularly in pre-training on unlabeled datasets
and fine-tuning in task-specific downstream datasets. In
this paper, we project the representations of all modalities as probabilistic distributions via a Probability Distribution Encoder (PDE) by utilizing sequence-level interactions. Compared to the existing deterministic methods, such
uncertainty modeling can convey richer multimodal semantic information and more complex relationships. Furthermore, we integrate uncertainty modeling with popular pretraining frameworks and propose suitable pre-training tasks:
Distribution-based Vision-Language Contrastive learning(D-VLC), Distribution-based Masked Language Modeling(D-MLM), and Distribution-based Image-Text Matching (DITM). The fine-tuned models are applied to challenging
downstream tasks, including image-text retrieval, visualquestion answering, visual reasoning, and visual entailment,
and achieve state-of-the-art results.

1. Introduction
Precise understanding is a fundamental ability of human intelligence, whether it involves localizing objects fromsimilar semantics or finding corresponding across multiple
modalities. Our artificial models suppose to do the same,
pinpointing exact concepts from rich multimodal semantic scenarios. However, this kind of precise understanding
is challenging. Information from different modalities canpresent rich semantics from each other, but the resulting ambiguity and noise are also greater than the case with a single* Equal contribution.
² Corresponding Author.

(a) Vision uncertainty(b) Language uncertainty(a.1) A billboard(a.0) An image region / grid(b.1) A woman poses with avocado sandwich
lunch at an outdoor restaurant(b.2) Young girl having a meal in outdoor setting.
(a.2) Zebras(b.3) A woman sitting at a restaurant getting
ready to eat her food.

(a.3) Snowy Mountains

Inter-modal uncertainty(c) Vision-to-Language uncertainty(d) Language-to-Vision uncertainty(d.1) People along a narrow street and a guy riding a scooter.
(d.2) The motorcyclist is traveling down the busy street.
(d.3) Man on moped traveling through busy street.

(c.1) There are people
with umbrellas in thestreet.

(e) An example for (b) Language uncertainty (Point Rep. V.S. Distribution Rep.)
avocado sandwich

avocado sandwich
meal
food
meal

(e.1) Point Representations

food(e.2) Distribution Representations

Figure 1. Multimodal uncertainties and an example for language uncertainty (b) by modeling as point representations and distribution
representations. The images and text are from MSCOCO [30].

modality.
Multimodal representation learning methods hold thepromise of promoting the desired precise understanding
across different modalities [13]. While these methods haveshown promising results, current methods face the challengeof uncertainty [7, 51], including within a modality and between modalities. Considering image (a.0) in Fig. 1 as an
example, one vision region includes multiple objects, suchas a billboard, several zebras and others. Therefore, it isunclear which objects when mentioning this region. In thelanguage domain, the complex relationships of words leadto uncertainty, such as synonymy and hyponymy. In Fig. 1(c)&(d), the same object often has different descriptions

23262

from different modalities, such as text and images, which
manifests inter-modal uncertainty. Instead, previous methods
often neglect the uncertainty [11, 19, 46], resulting in limited
understanding ability on complicated concept hierarchies
and poor prediction diversity. Therefore, it is desirable tomodel such uncertainty.
Moreover, with multimodal datasets becoming more commonplace, there is a flourishing trend to implement pretraining models, particularly Vision-Language Pre-training(VLP), to support downstream applications [6, 18, 23, 36, 50].
Existing deterministic representations, however, often fail tounderstand uncertainty in pre-training data, as they can only
express positions in semantic space and measure the relationship between targets in certainty, such as Euclidean distance.
How can we efficiently model uncertainty in multi-modalities
when dealing with pre-training models?
Applying Gaussian distribution is one of the prominent
approaches used for modeling uncertainty in the representation space [40, 45, 51, 54]. In these methods, however, the obtained uncertainty depends on individual features rather thanconsidering the whole features together, which ignores the inner connection between features. To exploit this connection,
we implicitly model them when formulating the uncertainty
with a module called Probability Distribution Encoder (PDE).
Inspired by the self-attention mechanism [44], we further
add the interaction between text tokens and image patches
when constructing our distribution representations to capture
more information. In Figure 1 (e), we provide an example
for two different types of representations to describe the language uncertainty, where the distribution representations can
express richer semantic relationships than the conventional
point representations. The distribution variance measures theuncertainty of the corresponding text. As a byproduct, distribution representations enable diverse generations, providing
multiple reasonable predictions with random sampling.
In this paper, we integrate this uncertainty modeling in
the pre-training framework, resulting in three new tasks:
Distribution-based Vision-Language Contrastive learning(D-VLC), Distribution-based Masked Language Modeling(D-MLM), and Distribution-based Image-Text Matching (DITM) pre-training tasks. All these tasks are to deal with crossmodality alignment. More specifically, D-VLC is to handle
the coarse-grained cross-modal alignment, which measures
the whole distributions to align representations from different domains. D-MLM and D-ITM are implemented after thefine-grained interaction between different modalities, providing the token level and overall level alignment for images
and text.
Our contributions are summarized as follows:
1) We focus on the semantic uncertainty of multimodal understanding and propose a new module, called Probability
Distribution Encoder, to frame the uncertainty in multimodal
representations as Gaussian distributions.

2) We develop three uncertainty-aware pre-training tasks todeal with large-scale unlabeled datasets, including D-VLC,
D-MLM, and D-ITM tasks. To the best of our knowledge,
these are the first attempt to harness the probability distribution of representations in VLP.
3) We wrap the proposed pre-training tasks into an end-2-end
Multimodal uncertainty-Aware vision-language Pre-training
model, called MAP, for downstream tasks. Experiments show
MAP gains State-of-The-Art (SoTA) performance. Our codeis available at https://github.com/IIGROUP/MAP.

2. Related Works
2.1. Probability Distribution Representations
Current popular representation learning methods extract
features as point representations and focus on searching forthe closest position to ground truth in high-level representation space. However, there is usually more than one suitable
point representation, which shows the uncertainty in multiple tasks. To address this problem, the following researchers
introduced probability distribution representations to infer
diversely and improve robustness, avoiding model overfitting
to one single solution. In the Natural Language Processing(NLP) field, multivariate Gaussian distribution was utilized
to represent words [45] due to the powerful capability for representing the asymmetric relations among words. Since then,different distribution families were exploited for word representations [2, 28]. In Computer Vision (CV), for modelingvision uncertainty, some researchers introduce Gaussian representations into specific tasks, such as face recognition [4],person re-identification [54], 3D skeleton action representation [40] and pose estimation [42]. For solving the long-tailproblem in relation prediction, Gaussian distribution was
utilized to build objects relationship in scene graph generation [52]. Recently, constructing distributions achieved some
progress to yield diverse predictions for cross-modal retrieval
in multimodal field [7]. However, those existing methods
only consider the feature level to build the distributions for awhole image or sentence. In this work, we model not only
the whole image or sentence to the distribution representations but also each token of them, such as patches and words.
Furthermore, our approach learns the multimodal uncertainty
from sequence-level and feature-level interactions.

2.2. Vision-Language Pre-training (VLP)
Inspired by the Transformer structure [44] and pretraining tasks from BERT [8], the recent emergence ofvision-language pre-training tasks and models have been
explored to learn multimodal representations. The main process is first to pre-train the models by exploiting auxiliarytasks to understand hidden supervision information fromlarge-scale unlabeled data. Then, the pre-trained models
embed real-world objects into multimodal representations.

23263

Distribution Rep. Space

Contrastive Loss

Language
Feature Encoder

Distribution Rep. Space

Cross-modal Transformer

PDE Processing

Point Rep.

PDE

PDE Processing

Feed Forward

Text
input

Feed Forward

𝑋~𝑁(𝜇, 𝜎 2)

Cross Attention

PDE Processing

Cross Attention

D-VLC

Self Attention

VisionFeature Encoder

Self Attention

Image
input

Multivariate GD𝜇𝜎Three-view

× 𝑁𝐿

D-ITM

D-MLM

Image [CLS] Token

Text [CLS] Token

Distribution Rep.

Figure 2. Pre-training model architecture and objectives of MAP. We propose PDE to model distribution representations as multivariate
Gaussian Distributions (GD). ªNL º indicates the layer number of the cross-modal transformer. We perform two-dimensional Gaussian
distribution as an example.

With effective universal multimodal representations, they
can achieve good performance by fine-tuning on relatively
small labeled datasets of VL downstream tasks. The key challenge of VLP is to design suitable pre-training objectives.
Recently, mainstream strategies include Masked Language
Modeling (MLM) [15, 16, 20, 23, 26], Image-Text Matching(ITM) [15,16,23,26] and Vision-Language Contrastive learning (VLC) [18, 20, 26, 36]. MLM in VLP requires the model
to predict the masked language context tokens by the rest of
the language context tokens and vision context tokens. To
understand alignment information of language context andvision context, ITM requires the model to judge whether the
input of different modalities matches or not. VLC learns thesimilarity from inter-modal information and aligns point representations of different modalities. However, those methods
only are designed in the point representation space without
considering multimodal uncertainty. Therefore, we propose
the D-VLC, D-MLM and D-ITM to pre-train our modelin the distribution representation space. The details will be
explained in Sec. 3.2.

certainty, we further frame the input features as multivariate
Gaussian distributions. Specifically, PDE predicts a mean
vector (µ) and a variance vector (σ 2 ) for each input feature.
The mean vector represents the center position of distributions in probabilistic space, and the variance vector expresses
the scope of distributions in each dimension.
As shown in Fig. 3, we propose a probability distribution
encoder (PDE) while considering that modeling the mean
and variance vectors takes feature-level and sequence-levelinteractions. Specifically, Feed Forward layer is used forfeature-level interactions and Multi-Head (MH) operationis responsible for sequence-level interactions. By applying
the MH operation, the input hidden states H ∈ RT ×D are
split into k heads, where T is sequence length and D is
hidden size. In each head, we split the features and send
them to two paths (µ, σ 2 ). In each path, the input hidden
states H (i) ∈ RT ×D/2k are projected to Q(i) , K (i) , V (i) in
i-th head. As an example, the operation in the µ path is:[Qµ(i) , Kµ(i) , Vµ(i) ] = Hµ(i) Wqkv ,⊤ pHeadµ(i) = Act Qµ(i) Kµ(i) / dk Vµ(i) ,MHµ = concati∈[k] Headµ(i) WO ,3. Approaches
In this section, we introduce our proposed PDE and thearchitecture of MAP (Sec. 3.1), and the overall structure is
described in Fig. 2. The details of our proposed distributionbased pre-training tasks are presented in Sec. 3.2. In addition,
we further discuss the ethical considerations in Appendix C.

3.1. Model Overview
3.1.1

Probability Distribution Encoder (PDE).

The input features of PDE are from the point representation
space of different modalities. To model the multimodal un-(1)

where dk is set to D/(2k). The weight Wqkv ∈ Rdk ×3dk
is to project the inputs in the subspace of each head. The
weight WO ∈ Rkdk ×D projects the concatenation of k head
results to the output space. The ªActº includes an activation function and a normalization function for considering
sequence-level interaction. The σ 2 path is similar to the µpath. Since the input point representation correlates with themean vector, an add operation is employed to learn the mean
vector. The motivations of design choices are in Sec. 4.3.2.
After PDE, each vision or language token is represented as a

23264

𝑁 ℎ𝑒𝑎𝑑×and CA is as follows:
SAvision : Ii′ = MHA(Ii−1 , Ii−1 , Ii−1 ),Q+

Feed Forward

V

Feed Forward

ConcatK

Interaction

𝝁
Path

𝜇

Q
Feed Forward

V

Concat𝝈𝟐 PathInteractionK

𝜎2

SAlanguage : Ti′ = MHA(Ti−1 , Ti−1 , Ti−1 ),CAvision : Ii = MHA(Ii′ , Ti′ , Ti′ ),(2)CAlanguage : Ti = MHA(Ti′ , Ii′ , Ii′ ).

For the overall structure design of MAP, we apply PDEs
after feature extractors and cross-modal transformer, respectively. PDE after the feature extractor learns unimodal distribution representations to conduct the D-VLC pre-trainingtask. PDE at the end of MAP is responsible for D-MLM,
D-ITM and downstream tasks.

3.2. Distribution-based Pre-Training Tasks
Figure 3. The architecture of Probability Distribution Encoder(PDE) block.

gaussian distribution in high-dimension probabilistic space.

3.1.2

Feature Extraction.

To extract features, we utilize an image encoder and a language encoder. In detail, we employ CLIP-ViT [36] as theimage encoder and RoBERTa-Base [31] as the language encoder. An image is encoded as an patch feature sequence{v[CLS] , v1 , . . . , vN } , where v[CLS] is the overall feature.
Moreover, the input text is embedded into a sequence of
tokens {w[CLS] , w1 , . . . , wM }.

In order to learn the multimodal uncertainty in common
sense, we pre-train our model with distribution-based pretraining tasks on large-scale datasets.
3.2.1

Coarse-grained Pre-training.

We propose Distribution-based Vision-Language Contrastive
Learning, called D-VLC, to realize coarse-grained semantic alignment of overall unimodal distributional representations before fusion. We compute the 2-Wasserstein distance [21, 22, 33] to measure the distance between multivariate Gaussian distributions. For two Gaussian distributions
N (µ1 , Σ1 ) and N (µ2 , Σ2 ), their 2-Wasserstein distance isdefined as:
1/2

1/2

D2W = ||µ1 − µ2 ||22 + Tr(Σ1 + Σ2 − 2(Σ1 Σ2 Σ1 )1/2 ). (3)3.1.3

Cross-modal Transformer.

Recently, there are two main types of the multimodel transformer to fuse the different modalities: single-stream [6, 39,56] and dual-stream [26, 32, 43] models.
In our method, the image patch sequences are much
longer than text sequences, making the weights of vision
features too large to compute the attention scores together.
To address this issue, we choose the dual-stream modulewith two transformer branches, where self-attention scores
are calculated separately.
As shown in Fig. 2, the main structure has NL layersof cross-modal encoders. Each encoder mainly consists of
two Self-Attention (SA) blocks and two Cross-Attention(CA) blocks. In the SA block of each modality, query, key
and value vectors are all linearly projected from vision or
language features. In the vision-to-language cross-attention
block of i-th layer, query vectors represent language feature Ti′ after the self-attention block, and key/value vectors
denote vision feature Ii′ . By employing the Multi-Head Attention (MHA) operation, the CA block enables language
features to learn visual information across modalities. The
language-to-vision CA block is similar to the vision-tolanguage one. The workflow of i-th layer encoder with SA

In our modeled distributions, Σ1 and Σ2 are both diagonal
1/2
1/2
matrices, which indicates Σ1 Σ2 Σ1 = Σ1 Σ2 . The above
formula can be rewritten as:
1/2

D2W = ||µ1 − µ2 ||22 + Tr((Σ11/2− Σ2 ) 2 )= ||µ1 − µ2 ||22 + ||σ1 − σ2 ||22 ,(4)

where σ refers to a standard deviation vector. The overall
unimodal features denote the distribution representations of[CLS] from the PDEs following single-modal feature extractors. The similarity between an image and text is given by:s(I, T ) = a · D2W (v[CLS] , w[CLS] ) + b ,(5)where a is a negative scale factor since similarity is inversely
proportional to the distance, and b is a shift value. For N
image-text pairs in a batch, there are N positive matchedsamples and N (N − 1) negative samples. We use InfoNCE
loss as follows:exp(s(Ii , Ti )/τ )LI2T,
N CE (i) = − log PN
n=1 exp(s(Ii , Tn )/τ )exp(s(Ti , Ii )/τ )LTN2I,
CE (i) = − log PN
n=1 exp(s(Ti , In )/τ )(6)

where τ is a learned temperature parameter. The above aresummed as D-VLC loss LD-V LC .

23265

3.2.2Fine-grained Pre-training.

After the cross-modal transformer with fine-grained interaction on each token of different modalities, our proposed
Distribution-based Masked Language Modeling (D-MLM)
and Distribution-based Image Text Matching (D-ITM) can
assist the model in learning fine-grained cross-modal alignment.
D-MLM requires the model to predict the masked words
by understanding the text with an image. Specifically, theinput text tokens are replaced by [MASK] with a probability of
15% (Details in Appendix A.3). For conducting classificationin the word list to predict the original words, we sample
the point vectors from distribution representations. D-MLM
minimizes a Cross-Entropy (CE) loss for µ point and othersample point vectors:
LD-M LM =KX

1(CE(ϕ(µ), y) +CE(ϕ(z (i) ), y)), (7)
K +1
i=1

where K is the sample number from gaussian distributions
and y serves as a masked word label. µ is a mean vector andz (i) refers to stochastic sample point vectors; then, they are
fed to MLM classifier ϕ. During the inference process, thefinal output is the mean pooling of all samples’ prediction
results:
P =KX
1(ϕ(µ) +ϕ(z (i) )).
K +1
i=1

(8)

D-ITM provides a binary classification that predicts
whether a pair of image-text is matched or not. Specifically,
we sample the point vectors from wCLS distributions of visionand language features and concatenate them as the fusion
features to generate the prediction.
1(CE(ϕ(concat[vµ , wµ ]), y)
K +1
K
X
+
CE(ϕ(concat[v (i) , w(i) ]), y)),LD-IT M =h(N (µ, Σ)) =1
log(det(2πeΣ)),2(11)where Σ is the covariance matrix, which is a diagonal matrix
in our method. Therefore, the diagonal vector of Σ is σ 2 andthe Eq. (11) can be transformed to:dh(N (µ, σ 2 )) =1X
log(2πe · σi2 )
2 i=1

d
X
d
= (log(2π) + 1) +log σi ,
2
i=1

(12)

where d is the feature dimension.
Note that the sampling operation for N (µ, σ 2 ) causes the
problem of preventing gradients from propagating back. By
applying the reparameterization trick [24], we first sample arandom noise ϵ from standard normal distributions, insteadof directly sampling from N (µ, σ 2 ):z = µ + σϵ,ϵ ∼ N (0, I).

(13)

After Eq. (13), the output z obeys the predicted distributions
from the PDE. Therefore, we can separate the calculations of
the mean and standard deviation from the sampling operationand they are trainable.
3.2.3

Training Objectives.

During pre-training phase, the model will propagate forward
three times at one step with conducting D-MLM, D-ITM
and D-VLC tasks separately. Therefore, the full pre-training
objective is given by:(9)

Lpre = LD-M LM + LD-IT M + LD-V LC + αLreg ,(14)

where α is its weight.

i=1

where vµ , tµ are mean vectors of vision and language [CLS]
distributions. v (i) , w(i) are sample points and ϕ is the DITM classifier. The image-text pairs in the datasets serve as
positive examples, and negative examples are constructed byrandomly replacing images or text descriptions.
However, random sampling increases training difficulty.
When the model is trained only with the aforementioned
losses, it will lead to variance collapse. Since all sample
point vectors will converge to the optimal position, the distribution representations eventually degenerate into point
representations, resulting in losing the ability to learn multimodal uncertainty. Therefore, we append a regularization
loss to prevent the uncertainty level of distributions is lower
than a certain threshold:Lreg = max(0, γ − h(N (µ, σ 2 ))),

where γ is a set threshold, which affects the uncertainty levelof distributions. h(N (µ, σ 2 )) is the entropy of a multivariate
Gaussian distribution, which should be defined as:(10)

4. Experiments
4.1. Experimental settings
By following a popular setting [9], we set all hiddenfeature sizes as 768, and the head number as 12 in MHA.
Unless otherwise specified, the layer number (NL ) of thecross-modal transformer is set to 6. As for data processing,
we resize and crop each image into the size of 384×384. The
size P of the image patch is 16. And the maximum lengthof input text dealt is set to 50. In PDE, the head number k is
set to 6 and the default ªActº function in Eq. (1) is Softmax.
For pre-training, we pre-train our model with D-MLM,
D-ITM and D-VLC. The pre-training datasets include
MSCOCO [30], Visual Genome (VG) [25], SBU [34] andConceptual Captions (CC-3M) [38]. Specifically, we resize

23266

Model

IR@1

IR@5

MSCOCO (5K test set)IR@10 TR@1 TR@5

Flickr30K (1K test set)IR@10 TR@1 TR@5

TR@10

IR@1

IR@5

TR@10

94.3

97.2

85.6

97.5

98.9

95.9

99.8

100.0

Group 2: Pre-training datasets include > 10M images
ALBEF (14M) [26]

60.7

84.3

90.5

77.6

Group 1: Pre-training datasets include < 10M images
UNITER-Large [6]VILLA-Large [10]UNIMO-Large [27]VinVL-Large [56]

52.9
58.8

79.9
83.5

88.0
90.3

65.7
75.4

88.6
92.9

93.8
96.2

75.6
76.3
78.0
-

94.1
94.2
94.2
-

96.8
96.8
97.1
-

87.3
87.9
89.4
-

98.0
97.5
98.9
-

99.2
98.8
99.8
-

ViLT [23]UNITER-Base [6]
ALBEF (4M) [26]
TCL [53]
METER [9]MAP (ours)

42.7
50.3
56.8
59.0
57.1
60.9

72.9
78.5
81.5
83.2
82.7
86.2

83.1
87.2
89.2
89.9
90.1
93.1

61.5
64.4
73.1
75.6
76.2
79.3

86.3
87.4
91.4
92.8
93.2
94.8

92.7
93.1
96.0
96.7
96.8
97.6

64.4
72.5
82.8
84.0
82.2
83.8

88.7
92.4
96.7
96.7
96.3
97.2

93.8
96.8
98.4
98.5
98.4
98.7

83.5
85.9
94.3
94.9
94.3
94.9

96.7
97.1
99.4
99.5
99.6
99.5

98.6
98.8
99.8
99.8
99.9
99.8

Table 1. An overall comparison with SoTA models on fine-tuned image-text retrieval tasks. The best scores are in bold and the second bestscores are underlined.

and crop each image into the size of 288 × 288. Please findmore pre-training and fine-tuning details in Appendix B.1.
In all experiments, we employ the randomized Tukey
HSD p-values and effect sizes based on one-way
ANOVA [37] to support the statistical significance of all
results (Please refer to Appendix B.7 for more details).

4.2. Results of VL Downstream Tasks
In this section, we apply our pre-trained MAP on the following 4 VL downstream tasks with 5 widely-used datasets.
Image retrieval task (MSCOCO [30] and Flickr30K [35])aims to understand the multimodal uncertainty which resultsfrom the multiplicity of concepts in images and text. This issimilar to the objective of our uncertainty modeling in nature. Meanwhile, visual question answering (VQA2.0 [12]),visual reasoning (NLVR2 [41]) and visual entailment (SNLIVE [49]) implicitly perform ambiguous semantics in unimodal and cross-modal items. Therefore, we further evaluate
our MAP on the aforementioned tasks to varying the effectiveness and generalization ability of uncertainty modeling.
For fair experimental environments, we group previous models with different sizes of pre-training datasets. Please findmore details of the datasets, model descriptions and additional experiments in Appendix B.
4.2.1

Evaluation on Image-Text Retrieval

As shown in Table 1, our MAP achieves the best performanceon MSCOCO and gains either the best or the second-bestscores on Flickr30K. Specially, while ALBEF has speciallydesigned objectives for retrieval, the MAP also outperforms
ALBEF (14M pre-training images) in all metrics on the
MSCOCO retrieval task. The results show the effectivenessand advantages of uncertainty modeling. For the Flickr30K

VQA2.0test-dev

Model

NLVR2
dev
test-p

SNLI-VE
val
test

Group 2: Pre-training datasets include >10M images (Base size)
ALBEF (14M) [26]
SimVLM-Base [48]

75.84
77.87

81.72
82.55

81.77
83.14

84.20
80.80

84.15
80.91

Group 1: Pre-training datasets include <10M images (Base size)
ViLT [23]UNITER-Base [6]OSCAR-Base [29]UNIMO-Base [27]
ALBEF (4M) [26]VinVL-Base [56]VLMo-Base [47]
METER [9]MAP (ours)

71.26
72.70
73.16
73.79
74.54
75.95
76.64
77.68
78.03

75.70
77.18
78.07
80.24
82.05
82.77
82.33
83.30

76.13
77.85
78.36
80.50
83.08
83.34
83.05
83.48

78.59
80.00
80.14
80.86
81.40

78.28
79.10
80.30
81.19
81.39

Table 2. An overall comparison with SoTA models on visual question answering, visual reasoning, visual entailment tasks. The bestscores are in bold and the second best scores are underlined.

dataset, our MAP achieves the best performance or only
about 0.1 point behind the best score. PCME also utilizes
probabilistic distribution representations to conduct retrievaltask, and we show the comparison in Appendix B.6.
4.2.2

Evaluation on VQA2.0, NVLR2, and SNLI-VE

As shown in Table 2, our MAP outperforms the previous
SoTA models in Group 1. Compared to VLMo-Base, the
MAP improves 0.53 points on NLVR2 dev. Our model
brings the +0.35 points improvement on VQA2.0 test-dev
and +0.54 points performance gains on SNLI-VE val over
METER. Notably, MAP outperforms SimVLM-Base (1.8B
pre-training images) in all tasks, which further demonstrates

23267

VQA2.0
test-dev

SNLI-VE
val
test

NLVR2
dev
test-p

75.91
76.67

76.28
76.86

50.86
51.12

51.07
51.07

79.42
80.05

79.84
80.31

77.72
78.96

79.31
79.64

Structure

VQA2.0 (test-dev)

MLP only

72.01

Random initialization
MAP w/o PDE
MAP

72.09
73.35

PDE

Pretained on MSCOCO
MAP w/o PDE
MAP

74.57
75.01

Table 3. The effectiveness of probability distribution representations on VL downstream tasks. For ªMAP w/o PDEº, we train a
new model without PDE to conduct the experiments. Pre-trained
methods for MAP: D-MLM, D-ITM. Pre-trained methods for MAP
w/o PDE: MLM, ITM.

the effectiveness of uncertainty modeling.

4.3. Ablation Studies
4.3.1

How do the probability distribution representations affect VL downstream task?

As shown in Table 3, applying PDE helps the model toachieve a better performance, which matters significantly
in VL downstream tasks. In both random and pre-trained
weights initialization cases, distribution representations gaina better capability of VL understanding than the point representations, which is because the distribution representations
can express richer semantics by learning multimodal uncertainty.
4.3.2

How does the structure of PDE behave?

We remove the sequence-level interaction in PDE and call itªMLP onlyº (MultiLayer Perceptron), which is the common
method in the previous works [7, 51, 54]. Table 4 shows that
PDE (Softmax) outperforms ªMLP onlyº in VQA2.0, which
benefits from the sequence-level information. Moreover, we
design several candidate activation functions: ReLU, ReLU2 ,Sigmoid, and Softmax. Notably, ªMLP onlyº outperformsªReLUº and ªReLU2 º, which demonstrates that it is important to consider how to design the sequence-level interaction.
The function Sigmoid projects the input values between 0 to
1, which smoothly assigns weights between different tokens.
The function Softmax outperforms the others in VQA2.0,
which implies that Softmax is suitable to express the correlation between tokens. Therefore, we set Softmax as thedefault activation function in sequence-level interaction.
4.3.3

What is the performance of different pre-training
objectives?

Table 5 presents that different choices of pre-training tasks
affect the VL downstream tasks performance. According

ReLU+Normal
ReLU2 +Normal
Sigmoid+Normal
Softmax

69.70
70.53
73.34
73.35

Table 4. Effect of different structures of PDE. We explore thedifferent designs of ªActº in Equation 1. Normal denotes the normalization operation.

Training strategies

VQA2.0test-dev

SNLI-VE
test-p

NLVR2
test

Random Initialization
D-MLM, D-ITM
D-MLM, D-VLC
D-ITM, D-VLC
D-MLM, D-ITM, D-VLC

73.35
75.01
75.06
71.02
75.16

76.86
80.31
80.12
78.54
80.39

51.07
79.64
77.90
73.64
79.47

Table 5. The effect of distribution-based pre-training tasks. We
pre-train the model on the MSCOCO dataset.
Layer Number

Random Initializing

Pre-training

2
4
6
8

72.71
73.32
73.35
73.31

73.78
74.73
75.16
75.26

Table 6. The effect of different layer numbers in the cross-modaltransformer on VQA2.0.

to the chart, results without D-MLM pre-training are theworst in all pre-training strategies, which means D-MLM
plays the most important role in pre-training. Both D-VLC
and D-ITM assist the model in learning semantic similarity
between vision and language modality. In VQA2.0, D-VLC
makes a larger improvement than D-ITM, whereas D-ITM
is more effective than D-VLC in SNLI-VE and NLVR2.
4.3.4

Does the number of layers of cross-modal transformer matter?

As shown in Table 6, we explore the effect of layer number
on VQA2.0 by random initializing or pre-training with DMLM, D-ITM, D-VLC on the MSCOCO dataset. By random
initializing, the model with six layers achieves the best performance, which encounters a bottleneck. After pre-training,the model with eight layers makes little progress from six
layers, which implies that pre-training helps the model break
the aforementioned bottleneck of parameters. The reason
perhaps is that pre-training with large-scale data alleviates
the problem of over-fitting by more parameters. Moreover,as the layer number decreases, the effect of pre-training will

23268

8

Case 1
Question: What is this food?
MAP w/o PDE: Cake
MAP:
DessertCake
Muffin

9 | A little boy withearphones on listening
to something.
10 | A group of people
sitting at desk using
computers.
11 | Children sitting atcomputer stations on along table.
4

5 | a little boy wearing
headphones andlooking at a computer
monitorCase 2
Question: What is in the man's ear?
MAP w/o PDE: Headphones
MAP:
Headphones
HeadsetEarbuds

6 | He is listening
intently to the computerat school
7 | A young boy stares
up at the computer
monitor.
12

13 | A kitchen area withtoilet and various cleaning
appliances.
14 | A commercial dishwashing station with atoilet in it.
15 | A toilet and mop
bucket in a kitchen.

0

1 | A young man holding anumbrella next to a herd of cattle

Case 3
Question: Where are the people flying
kites?
MAP w/o PDE: Field
MAP:
Field
Park
Grass

2 | a young boy barefoot holding
an umbrella touching the horn ofa cow
3 | A young boy with anumbrella who is touching thehorn of a cow

Figure 4. Visualization of the distribution representations from pretrained MAP2 . Each 2D Gaussian distribution is represented as
an ellipse with 95% confidence. The labels of images and related
captions are in the legend.

Figure 5. Predictions sampled from the distribution representations3 .

reduce due to the model’s limited learning capacity.

4.4. Uncertainty Modeling Analysis
Visualization. To perform visualization analysis for distribution representations from pre-trained MAP, we conduct
2D toy experiments for the images and related descriptions.
Fig. 4 shows the behaviors of the distribution representations,which present that distributions with similar semantics are
clustered together. The shapes of images and related descriptions are similar, and the ellipses are closed, showing that theimages and text cover similar meanings. For example, since
image ª4º is a part of image ª8º, ellipse ª8º almost includes
all regions of ellipse ª4º. The intersection of ellipses (images ª0º, ª4º, ª8º and their corresponding captions) might
indicate ªa young boyº in images and text. Similar behaviors
of our MAP can be found in more visualizations in Appendix B.8. Intuitively, as shown in visualization results,
uncertainty modeling facilitates the model to express richsemantic information and complex relationships.
Cases for diverse predictions. Semantic uncertainty is ubiquitous in multimodal tasks. For multimodal understanding
tasks such as VQA, an advantage of uncertainty modeling isthat multiple predictions can be sampled from distribution
representations, which provides diversity. Consider case 3
in Fig. 5, MAP can learn multiple plausible answers (field,
park and grass) from the distribution representations, which
is close to our real world. In contrast, the point representations from MAP without PDE always generate one answer
ignoring other possible expressions. Moreover, distribution
representations can also help other multimodal tasks, such
1 The images and related captions come from MSCOCO dataset [30].

as image captioning, to generate several suitable captions,which benefit from diverse correspondences caused by uncertainty modeling.

5. Conclusions
In this work, we focus on the multimodal uncertainty inreal-world objects by modeling this onto probability distributions. By considering sequence-level and feature-levelinteractions, we proposed a Probability Distribution Encoder (PDE) to gain distribution representations for different modalities. Our experiments showed that distribution representations are beneficial for the VL downstreamtasks. In addition, uncertainty modeling facilitates diverse
predictions. To learn multimodal uncertainty in large-scaledata, we designed three new pre-training tasks (D-MLM, DITM and D-VLC). Furthermore, we propose an end-to-end
Multimodal uncertainty-Aware vision-language Pre-training
model (MAP) to obtain generic distribution representations.
We demonstrate the effectiveness of the proposed MAP onseveral VL downstream tasks empirically. In the future, wewill explore more distribution subspaces and experiments onlarger datasets.

6. Acknowledgements
This work was partly supported by the National Natural Science Foundation of China (Grant No. 61991450)
and the Shenzhen Science and Technology Program(JSGG20220831093004008; ZDSYS20200811142605016).
2 The images and questions come from VQA2.0 validation set [12].

23269References[1] Haifa Alwahaby, Mutlu Cukurova, Zacharoula Papamitsiou,
and Michail Giannakos. The evidence of impact and ethicalconsiderations of multimodal learning analytics: A systematic literature review. The Multimodal Learning Analytics
Handbook, pages 289±325, 2022. 14[2] Ben Athiwaratkun and Andrew Gordon Wilson. Multimodal
word distributions. In Proc. of ACL, 2017. 2[3] Shruti Bhargava and David A. Forsyth. Exposing and correcting the gender bias in image captioning datasets and models.
CoRR, abs/1912.00578, 2019. 14[4] Jie Chang, Zhonghao Lan, Changmao Cheng, and Yichen
Wei. Data uncertainty learning in face recognition. In Proc.
of CVPR, 2020. 2[5] Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu
Soricut. Conceptual 12m: Pushing web-scale image-text pretraining to recognize long-tail visual concepts. In CVPR,
2021. 11[6] Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy,
Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter:
Universal image-text representation learning. In Proc. of
ECCV, 2020. 2, 4, 6, 12, 14[7] Sanghyuk Chun, Seong Joon Oh, Rafael Sampaio de Rezende,
Yannis Kalantidis, and Diane Larlus. Probabilistic embeddings for cross-modal retrieval. In Proc. of CVPR, 2021. 1, 2,
7, 13[8] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina
Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL,
2019. 2, 11, 12[9] Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang
Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu
Yuan, Nanyun Peng, Zicheng Liu, and Michael Zeng. An
empirical study of training end-to-end vision-and-languagetransformers. In Proc. of CVPR, 2022. 5, 6, 11, 12, 14[10] Zhe Gan, Yen-Chun Chen, Linjie Li, Chen Zhu, Yu Cheng,
and Jingjing Liu. Large-scale adversarial training for visionand-language representation learning. In NeurIPS, 2020. 6,
12[11] FrancËois Gardères, Maryam Ziaeefard, Baptiste Abeloos, andFreddy Lecue. ConceptBert: Concept-aware representationfor visual question answering. In Proc. of EMNLP Findings,
2020. 2[12] Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra,and Devi Parikh. Making the V in VQA matter: Elevating therole of image understanding in Visual Question Answering.
In Proc. of CVPR, 2017. 6, 8, 13[13] Wenzhong Guo, Jianwen Wang, and Shiping Wang. Deep
multimodal representation learning: A survey. IEEE Access,2019. 1[14] Eyad Hakami and Davinia HernÂandez Leo. How are learning
analytics considering the societal values of fairness, accountability, transparency and human well-being?: A literatureÂreview. MartÂınez-MonÂes A, Alvarez
A, Caeiro-RodrÂıguez M,
Dimitriadis Y, editors. LASI-SPAIN 2020: Learning Analytics Summer Institute Spain 2020: Learning Analytics. Time

for Adoption?; 2020 Jun 15-16; Valladolid, Spain. Aachen:
CEUR; 2020. p. 121-41, 2020. 14[15] Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez Opazo,and Stephen Gould. VLN BERT: A recurrent vision-andlanguage BERT for navigation. In Proc. of CVPR, 2021. 3[16] Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu,
Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-toend pre-training for vision-language representation learning.
In Proc. of CVPR, 2021. 3[17] Drew A. Hudson and Christopher D. Manning. GQA: A newdataset for real-world visual reasoning and compositionalquestion answering. In CVPR, 2019. 11[18] Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh,
Hieu Pham, Quoc V. Le, Yun-Hsuan Sung, Zhen Li, and Tom
Duerig. Scaling up visual and vision-language representation
learning with noisy text supervision. In Proc. of ICML, 2021.
2, 3, 11[19] Chenchen Jing, Yuwei Wu, Xiaoxun Zhang, Yunde Jia, andQi Wu. Overcoming language priors in vqa via decomposed
linguistic representations. In Proc. of AAAI, 2020. 2[20] Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. MDETR - modulated
detection for end-to-end multi-modal understanding. In Proc.
of ICCV, 2021. 3[21] Leonid V Kantorovich. Mathematical methods of organizing
and planning production. Management science, 1960. 4[22] Leonid V Kantorovich. On the translocation of masses. Journal of mathematical sciences, 2006. 4[23] Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-andlanguage transformer without convolution or region supervision. In Proc. of ICML, 2021. 2, 3, 6, 11, 12[24] Diederik P. Kingma and Max Welling. Auto-encoding variational bayes. In Proc. of ICLR, 2014. 5[25] Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson,
Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A. Shamma, Michael S. Bernstein, and
Li Fei-Fei. Visual genome: Connecting language and visionusing crowdsourced dense image annotations. Int. J. Comput.
Vis., 2017. 5, 11[26] Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq
Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with
momentum distillation. Proc. of NeurIPS, 2021. 3, 4, 6, 12[27] Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu,
Jiachen Liu, Hua Wu, and Haifeng Wang. UNIMO: towards
unified-modal understanding and generation via cross-modal
contrastive learning. In Proc. of ACL, 2021. 6, 12[28] Xiang Li, Luke Vilnis, Dongxu Zhang, Michael Boratko, andAndrew McCallum. Smoothing the geometry of probabilistic
box embeddings. In Proc. of ICLR, 2019. 2[29] Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei
Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu
Wei, Yejin Choi, and Jianfeng Gao. Oscar: Object-semantics
aligned pre-training for vision-language tasks. In Proc. of
ECCV, 2020. 6, 12[30] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays,
Pietro Perona, Deva Ramanan, Piotr DollÂar, and C Lawrence

23270

Zitnick. Microsoft coco: Common objects in context. In Proc.
of ECCV, 2014. 1, 5, 6, 8, 11, 13, 14[31] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar
Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized
BERT pretraining approach. CoRR, 2019. 4[32] Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert:
Pretraining task-agnostic visiolinguistic representations forvision-and-language tasks. In Proc. of NeurIPS, 2019. 4, 14[33] Anton Mallasto and Aasa Feragen. Learning from uncertain
curves: The 2-wasserstein metric for gaussian processes. Proc.
of NeurIPS, 2017. 4[34] Vicente Ordonez, Girish Kulkarni, and Tamara L. Berg.
Im2text: Describing images using 1 million captioned photographs. In NIPS, 2011. 5, 11[35] Bryan A. Plummer, Liwei Wang, Chris M. Cervantes,
Juan C. Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase correspondences for richer image-to-sentence models. In ICCV,2015. 6, 11, 13[36] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya
Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry,
Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen
Krueger, and Ilya Sutskever. Learning transferable visualmodels from natural language supervision. In Proc. of ICML,
2021. 2, 3, 4[37] Tetsuya Sakai. Laboratory experiments in information retrieval: Sample Sizes, Effect Sizes, and Statistical Power.
Springer, 2018. 6, 13[38] Piyush Sharma, Nan Ding, Sebastian Goodman, and Radu
Soricut. Conceptual captions: A cleaned, hypernymed, image
alt-text dataset for automatic image captioning. In ACL (1),2018. 5, 11[39] Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei,
and Jifeng Dai. VL-BERT: pre-training of generic visuallinguistic representations. In Proc. of ICLR, 2020. 4[40] Yukun Su, Guosheng Lin, Ruizhou Sun, Yun Hao, andQingyao Wu. Modeling the uncertainty for self-supervised
3d skeleton action representation learning. In Proc. of ACM
MM, 2021. 2[41] Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun
Bai, and Yoav Artzi. A corpus for reasoning about natural
language grounded in photographs. In Proc. of ACL, 2019. 6,
13[42] Jennifer J. Sun, Jiaping Zhao, Liang-Chieh Chen, FlorianSchroff, Hartwig Adam, and Ting Liu. View-invariant probabilistic embedding for human pose. In Proc. of ECCV, 2020.
2[43] Hao Tan and Mohit Bansal. LXMERT: learning crossmodality encoder representations from transformers. In Proc.
of EMNLP, 2019. 4[44] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In Proc. of NeurIPS,
2017. 2, 11[45] Luke Vilnis and Andrew McCallum. Word representations
via gaussian embedding. In Proc. of ICLR, 2015. 2[46] Junjie Wang, Yatai Ji, Jiaqi Sun, Yujiu Yang, and Tetsuya
Sakai. Mirtt: Learning multimodal interaction representations
from trilinear transformers for visual question answering. In
Proc. of EMNLP Findings, 2021. 2[47] Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei.
Vlmo: Unified vision-language pre-training with mixture-ofmodality-experts. CoRR, 2021. 6, 12[48] Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia
Tsvetkov, and Yuan Cao. Simvlm: Simple visual language
model pretraining with weak supervision. CoRR, 2021. 6, 12[49] Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment task for visually-grounded language learning.
CoRR, 2018. 6, 13[50] Haiyang Xu, Ming Yan, Chenliang Li, Bin Bi, Songfang
Huang, Wenming Xiao, and Fei Huang. E2E-VLP: end-toend vision-language pre-training enhanced by visual learning.
In Proc. of ACL, 2021. 2[51] Gengcong Yang, Jingyi Zhang, Yong Zhang, Baoyuan Wu,
and Yujiu Yang. Probabilistic modeling of semantic ambiguity
for scene graph generation. In Proc. of CVPR, 2021. 1, 2, 7[52] Gengcong Yang, Jingyi Zhang, Yong Zhang, Baoyuan Wu,
and Yujiu Yang. Probabilistic modeling of semantic ambiguity
for scene graph generation. In Proc. of CVPR, 2021. 2[53] Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda,
Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou
Huang. Vision-language pre-training with triple contrastive
learning. In CVPR, pages 15650±15659. IEEE, 2022. 6, 12[54] Tianyuan Yu, Da Li, Yongxin Yang, Timothy M. Hospedales,
and Tao Xiang. Robust person re-identification by modellingfeature uncertainty. In Proc. of ICCV, 2019. 2, 7[55] Zhou Yu, Jun Yu, Yuhao Cui, Dacheng Tao, and Qi Tian. Deep
modular co-attention networks for visual question answering.
In Proc. of CVPR, 2019. 14[56] Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei
Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl:
Revisiting visual representations in vision-language models.
In Proc. of CVPR, 2021. 4, 6, 1223271