VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Authors: Shraman Pramanick
Li Jing
Sayan Nag
Jiachen Zhu
Hardik J Shah
Yann LeCun
Rama Chellappa
Venue: Submitted to ICLR 2023 (2023)
Keywords: self-supervision
vision-language pre-training
transformer
patch-word alignment

Abstract: Vision-language pre-training (VLP) has recently proven highly effective for various uni- and multi-modal downstream applications. However, most existing end-to-end VLP methods use high-resolution image-text-box data to perform well on fine-grained region-level tasks, such as object detection, segmentation, and referring expression comprehension. Unfortunately, such high-resolution images with accurate bounding box annotations are expensive to collect and use for supervision at scale. In this work, we propose VoLTA (Vision-Language Transformer with weakly-supervised local-feature Alignment), a new VLP paradigm that only utilizes image-caption data but achieves fine-grained region-level image understanding, eliminating the use of expensive box annotations. VoLTA adopts graph optimal transport-based weakly-supervised alignment on local image patches and text tokens to germinate an explicit, self-normalized, and interpretable low-level matching criterion. In addition, VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre-training and removes fusion-specific transformer layers, further reducing memory requirements. Extensive experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fine-grained applications without compromising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations.

Under review conference paper ICLR 2023
VOLTA: VISION-LANGUAGE TRANSFORMER WITH WEAKLY-SUPERVISED LOCAL-FEATURE ALIGNMENT
Anonymous authors Paper under double-blind review
Captiom: Eanding quitar call thane.
Caption:
bottle
Captioon siting next the umtrolla stcking out FOCKs.
Caption: bowl of dessert sitinng next phone.
Figure I: We introduce VoLTA, Vision- Language Transformer with wuakly-supervised local-feature Alignment, VLP paradigm trained with graph optimal transport (GOT) based image-text matching. VoLTA learns fine- grained local visual representation only using global image-caption pairs, eliminating the of expensive grounding annotations. This figure shows how different words in captions attend relevant image regions, produced the GOT module of VoLTA pre-trained COCO.
ABSTRACT
Vision-language pre-training (VLP) has recently proven highly effective for various uni- and multi-modal downstream applications. However, existing end-to-end VLP methods use high-resolution image-text-box data to perform well on fine- grained region-level tasks, such object detection, segmentation, and referring expression comprehension. Unfortunately, such high-resolution images with curate bounding box annotations are expensive to collect and use for supervision at scale. In this work, we propose VoLTA (Vision-Lane Language Transformer with weakly-supervised local-feature Alignment), new VLP paradigm that only utilizes image-caption data but achieves fine-graid grained region-level image understanding, eliminating the use of expensive box annotations. VoLTA adopts graph optimal transport-based weakly-supervised alignment local image patches and text kens germinate an explicit, self-normalized, and interpretable low-level matching criterion. In addition, VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre-training and removes fusion-specific transformer layers, further reducing memory requirements. Extensive experiments on wide range of vision- and Dision-languse downstream tasks demonstrate the effectiveness of VoI.A fine-grained applications without compromising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations.
INTRODUCTION
Inspired by the escalating unification of transformer-based modeling in vision (Dosovitskiy et al., 202 Liu al.) 2021: Chen al. aa and language (Devlin al., 2019 Liu al. 2019) domains, coupled with readily available large-scale image-caption pair data, vision vision-language pre- gre-training (VLP) (Lu et al. 2019 Li et al., 2020aa Kim et al., 2021 Kamath et al. 2021 Zhang et al. 2021) has recently been receiving ever-growing attention. VLP has not only
proven the de-facto for several VL tasks, but it has also been beneficial for traditional vision-only tasks, such as image classification and object detection. Such wide-range applications of VLP can broadly be categorized into two groups: (i) tasks requiring image-level understanding, e.g., image classification, image &

---

Under review as conference paper ICLR 2023
text retrieval (Plummer et al. 2015), visual question answering (Antol et al. 2015), and (it) tasks requiring region-level understanding, e.g., object detection, instance segmentation, and referring expression comprehension (Kazemzadeh et al. 2014]] Yu et al. 2016). Most existing VI.P methods support either application, leaving the question of a generalizable and unified VL framework.
Traditional VLP methods with image-level understanding (Li et al. 202Taa] Wang et al., 202 Dou
al. 2022b) utilize large-scale image-caption pair datasets and are commonly trained with image-text contrastive objectives computed on global features. Hence, it is not trivial extend such methods region-level applications. On the other hand, VLP methods with region-level understanding (Kamath ct al. 2021 Li ct al. 2022c] Zhang et al. 2022) use mmage-text-box grounding data and are designed predict bounding boxes during pre training. Consequently, they do not support image-level tasks. Furthermore, accurate bounding box annotations require high-resolution input images, which are often expensive to collect, annotate and use for pre-training at scale. Recently, FIBER (Dou et al.) 2022a) addressed the problem of such unified VLP and proposed a two-stage pre-training algorithm requiring fewer box annotations than previous region-level pre-training methods. Moving a step forward, we aim to eliminate the use of costly box annotations and ask the challenging but natural question: Can attain region-level understanding from global image-caption annotations and unify image- and region-level tasks in single VL frameoork?
Subsequently, we focus on achieving region-level fine-grained understanding by Seakkl-suuprvisee alignment of image patches and text tokens. Previous VLP methods (Chen et al.) 2020d] Kim et al., 202T) in this direction use Wasserstein distance (WD) (Peyré et al. 2019), a.k.a Earth Mover's distance (EMD)-base
optimal transport (OT) algorithms for such aligument problem. However, we argue that WD is not optimum for intricate images with multiple similar entities. Thus, we propose to jointly utilize distance (WD) in a setup known as graph optimal transport (Chen et al. 2020a), Moreover, instead of using commonly deployed contrastive objective, we propose to utilize redundancy reduction from Barlow Twins (Zbontar al., 202T), which is less data-intensive and does require hard-negative mining. We also follow Dou et al. (2022a) to incorporate deep multi-modal fusion into the uni-modal backbones, removing the need for costly VoLTA, Vision-Language Transformer with weakly-supervisee local-feature Alignment, unified VLP paradigm only utilizes capge-c annotations but achieves fine-grained region-level image understanding, eliminating the use of expensive box annotations. Figure] I image-text alignment. VoLTA can attend text tokens to the corresponding visual patches without relying on low-level supervision.
distance (GWD) (Peyré al. 2016) and Wasserstein
transformer layers. To this end, we introduce
visualizes the feature-
In summary, our contributions are three-fold. (i) We propose to use graph optimal transport for veakly-supervised feature-level patch-token alignment. (fi We introduce VoL.TA, a unified VI.P paradigm for image-level and region-level applications. but pre- pre-trained only using image-caption pairs. VoLTA is memory, compute, and time-efficient and can easily be scaled up with readily available large-scale image-caption data harvested from the web. (fii) We performed a wide range of vision- and vision- langugge coarse- and fine-grained downstream experiments to demonstrate the effectiveness of VoLTA compared to strong baselines pre-trained with significantly more caption and box annotations.
2 RELATED WORKS
Uni-modal Self-supervised Pre-training: In recent years, the machine learning community has observed a boom of self-supervised pre-training. In the language domain, representations learned by BERT (Devlin al. 2019), RoBERTa (Liu et al. 2019) have become the default setting for any downstream tasks. Generative models such as GPT RRafford et al. 2019: Brown et al., 2020) have also achieved impressive www-shot/zeo-sshot performances on novel applications. SimCSE (ao et al. 2021 uses contrastive learning to help learn useful sentence representations.
the vision domain, series of contrastivefjoint-ember
methods (He al. 2020; Chen et al. et al. 202 |
Zhontar et al. 2021 Bardes et al.) 2022 AAssan et al. (022) have outperformed supervised counterparts. Recently, generative models such as BEiT (Bao et al., (02T) and MAE (He et al. 2022) have also achieved impressive performances with much more scalable potential.
2020c 202 20206) Grill et al.] [2200] Chen & He|
Vision-language pre-training: Vision-language pre-training (VLP) mainly relies on image-text pair datasets to leam joint visual- lannuag- representations. One line of work is to train separate vision and

---

Under review conference paper ICLR 2023
language encoders and only fuse their representation in the representation space. CLIP (Radford et al. 2021). UniCL (Yang et al.] 2022a) and ALIGN (Jia et al. (021) use image-text contrastive loss to learn aligned representations, SLIP (Mu et al. 2021) combines self- sell-supervised visual representation learning and contrastive multi multi-modal learning. M3AE (Geng et al. 2022). FLAVA (Singh et al. 2022)
leamming combines masked image modeling and masked language modeling. Another line of work is to use
cross attention to fuse vision and language information in the early stage (Kamath al. 202 Dou
al. 20226; Lu et al., [2019; Li et al. 2020b; Kiclaa ct al. 2019 Kim et al. 202T]/Zhang et al. [021
Li et al. [022bb Wang et al. 2022c). These works focus on learning semantic- aligned vision-
language representations. In addition, UniTAB (Yang et al. (022c). OFA (Wang et al. (022b). GLIIP
Li et al. (2022c). and FIBER (Dou et al. 2022a) use expensive grounding image-text-Be annotations to learn fine- tine-grained aligned representations. Our work uses representation space aligoment and cross-attention fusion, but we do not use any box annotation to learn robust feature-level alignments.
Unsupervised Representation Alignment: Unsupervised multi-modal alignment typically relies specific metrics. Wasserstein distance (WD) (Peyré et al. 2019). a.k.a Earth Mover's distance (EMD)- based optimal transport (OT) algorithms have widely been adopted to various domain alignment tasks, including sequence-to-sepuee sequence learning (Chen et al. 2019). few-shot learning (Zhang al.
2020), knowledge distillation (Balaji et al. 2019), unsupervised domain adaptation (Balaji et al.
2019). generative networks (Han et al. [015] Genevay et al., 2018 MMouch et al. 2018 2019),
and multi-modal learning (Yuan et al. [ 20200 Chen et 22.2d Kim et al. [2021] Li et al. 2022d
Pramanick et. al., 2022). Previous VLP methods (Chen et al. 2020d Kim et al. (021). which
use OT-based patch-word alignment, only utilize WD. However, we argue that jointly modeling
Gromov-Wase Wasserstein distance (GWD) (Peyré et al. (2016) and Wasserstein distance (WD) results in a
superior multi-modal alignment for intricate images. To the best our knowledge, this is the first work to apply WD and GWD-hased optimal transport for feature-level alignment in VI.P.
3 METHOD
In this section, we present our proposed framework, VoLTA, which contains three broad modules (i) intra- and inter- modaalto redundancy reduction, (ii) reekly-supprrisee cross-mes modal alignment (CMA) of local features, and (iii) cross-modal attention fusion (CMAF). Next, we introduce the fine-tuning strategies for various uni- and multi-modal downstream tasks as supported by VoLTA. An overview of different modules of VoLTA is depicted in Figure
3.1 PROPOSED SYSTEM VOLTA
3.1.1 INTRA- AND INTER-MODALITY REDUNDANCY REDUCTION
We use Barlow Twins (BT) (bontarr al. 2021), non-contrastive covariance regularization the foundational objective of VoLTA. The recent success of contrastive Vision-language pre-training (VLP) (Radford el al., 202T: Li et al. 202Th Dia et al. 202 I] Kim et al. 202T] Yang et al.] 2202aa]
Dou et al. 2222a by has already shown that, compared to single modality, image-caption pairs offer a significantly higher-level of abstractive and semantic concepts about the training samples. However, common contrastive VLP objectives, like InfoNCE (Oord et al. 2018). are data-hungry, as they require large batch sizes and well-mined hard negatives. On the other hand, the BT objective operates on the dimensions of the embeddings across the two views of training samples. Hence, it has proven robust to batch size and can be trained using lower memory resources. In this work, we extend the BT objective for mulli-modaa setup.
The original BT algorithm, which operates on joint embeddings of distorted samples, was proposed only for image modality. Specifically, for each image of a batch X. two distorted views are obtained using a distribution of data augmentation T with disparate probabilities. These distorted images then fed into a shared image encoder which contains feature extraction network (e.g.. ResNet (He et al. (0166) cascaded with trainable linear projection layers, producing batch of parallel embeddings d and 2. The BT loss computed using the encoded embeddings can be denoted as:
Σg Fb.
BB
LBT [ (1-Ca) C.) + 3E (Co)², Cū 344
b.j
(1)
E. (=A
E. (=2))
Where X is a positive weighting factor; C is the cross- corss-oorelltoon matrix computed between 2 and 2 along the batch dimension; b stands for sample indices in batch; i refers to the dimension

---

Under review conference paper ICLR 2023
Bar low Twins + Graph Optimal Transport
Mesked Language Modeling
Image Text Matching
25
M Fland
ITM Hend
Solvle
Selt-le
Serve
Sahan
Seriias
Saivle
Cet Mes
en
levers
Lever
Lanera
unver
Leners
Levrn
Mesuctal
Agoup of surg SPASE
snivenin wnttrr
Mosked Copten
Captice
cleying briirhal
Figure 2: Computation of four different objectives, LST, LoOT, LMLM, and LITM by the proposed framework, VoLTA Vision-Language Transformer with weakly-supervisee local-feature Aligmment). Inspired by (Dou et al. 2022a). VoLTA inserts cross-modal attention fusion (CMAF) inside uni-modal backbones with a gating mcchanism. During VoI.AA pre-training, every forward iteration consists of three steps (i) CMAF is switched off, VoLTA acts as dual encoder, LBT and Laor are computed. (ii) CMAF is switched on, VoLTA acts as fusion encoder, image-masked caption pair is fed into the model to compute CMLM- (fi)) CMAF is kept on, randomly sampled image-captin pair is fed into the model to compute CrTM. Such fusion strategy results in a lightweight and flexible model compared to using musion-specitic transformer layers.
indices of A and I The first term in Eqquation I ii the invariance term which attempts to equate the diagonal elements of the cross-correlation matrix to I, whereas the second term the redundancy reduction term which pushes the off-diagonal elements of the C² matrix to O.
In this work, we propose to use BT for image-caption pairs. Specifically, we use stochastic data augmentations for both images and text=] and directly apply the BT objective for all the 2 2 pairs, resulting in additional supervision. Note that, this simple, straightforward and instinctive extension enables to apply redundancy reduction in-hetween and across modalities, which intuitively results superior visual representation. Moreover, in this bi-modal setting, we can pre-train text encoder in parallell with the image encoder, and thus, can generalize our system to a wider range of uni- and multi-modal downstream applications.
Intra-modal Objective:
Intra-modal objective refers to applying the BT loss in in-between the pairs of image and text embedddings. Given an image-caption pair, we first have two augmented views (7, I') for each image, and two augmented views (T, T') for each text. Then, we resort to Equation] [ individually for the image and text pairs.
cbr & (1
C + Cl-
+ (KE S+L
Inter-modal Objective: inter-modal objective refers to applying the BT loss across the image and text embeddings. Since the image and text encoder can output features with different shapes, we design the projector layers with same output dimension. Hence, in addition the original BT loss between (I, Zbontar et al. (202I). we get three more loss terms (I,T"). (I,T'). (I',T), leading to 3x diverse and high-quality additional supervision. The inter-modal BT losses can also directly be computed following Equation]
C 4 Σ
C1) + + jAi
(1-
(ch) Wke (IT,
(3)
The resulting mi-ma BT loss is LTT = En - 4 kE (I,,T,,, IT,
Image and text augmentations details can be found in Appendix D.I

---

Under review conference paper ICLR 2023
3.1.2 ALIGNMENT OF LOCAL FEATURES
Though the inter-modal redundancy reduction provides high-quality semantic supervision, it is com- puted on the global image- and text features and, thus, only simulates implicit and non-interpretable multi-modal alignment. However, fine-grained region-level downstream applications like detection, segmentation, and reference expression comprehension require local visual feature descriptors with specific spatial information. To achieve this. most existing cop-performing VLP methods, including UniTAB (Yang et al. 2022c), OFA (Wang et al.) (022b), GLIP (Li et al. 2022c), and FIBER (Dou) ct al. 2022ap. use high-resolution image-text-box data for fine-grained pre-traiming However, bound- ing box annotations are expensive to collect and use for supervision. Hence, we seek an alternate weekly-supervised solution for local feature-level alignment using global image-caption annotations.
Recently, Wasserstein distance (WD) (Peyré et al. 2019), a.k.a Earth Mover's distance (EMD)- optimal transport (OT) algorithms have been used for -eperv--spper patch- word alignment VLP (Chen ct al. 2020d Kim et al. 2021). Such OT-based learning methods are optimized for distribution matching by minimizing the cost of tramsport plan. We pose the patch-word alignment as more structured graph-matching problem and use the graph optimal transport (GOT) algorithm, which utilizes Gromov- Wasseesstttnnnn distance (GWD) (Peyré et al. (016) in conjunction with WD to ensure the preservation of topological information during cross- moos- alignment. More specifically, we obtain the patch- and token-level features from the last layers of corresponding visual and textual transformer encoders, and use these encoded local-feature vectors to construct modality specific dynamic graphs G. Ex)) for image patches and Gy Ey) for text tokens. Each node in these graphs i e (V., V2 represented by corresponding feature vectors, and intermediate edges e E &E,,, F, by thresholded cosine similarity.
Importance of GOT in patch-word alignment: As men- tioned previously, GOT adopts types of OT distances -WD for node matching and GWD for edge matching. In contrast, previous Aasion-languse
pre- vee-rrainin algorithms
[Chen al. 2020d Kim et al. 202 I) using OT for patch- word alignment only considered WD. However, we argue that intricate images with multiple similar objects with different shapes and colors require both WD and GWD for Figure
accurate, fine-grained matching. For example, 3 there are multiple "men" present in the image. WD can only match nodes in the graph, and will treat all "men" entities as identical and will ignore neighbouring relations like "in blue shirt" and "holding the scissors". However, by using proper edge matching with GWD, we can pre- serve the graph's 's topological structure, We can correctly identify which "man" in the image the sentence is refer- ring to. Hence, we couple WD and GWD mutually beneficially and a joint transport plan for accurate parch-word matching.
the blue shirt is holding the
scissors.
Figure 3: Example of an intricate image containing multiple similar entities, and the visual attention map corresponding to the marked word in the caption, produced by the GOT module of VoLTA.
Once G2 and Gy are computed, we follow Chen et al. (2000a) to compute WD and GWD.
Wasserstein Distance (WD) calculates the pairwise distances between two sets of cross-domain node embeddings. Consider two discrete distributions, & P(X)) and ei E P(Y). where
Ei= u,ōg, and si- m Uyd, and & being the Delta-Dirac function centered on x. Since @ and &p are both probability distributions, sum of weight vectors is 1, Σ, ug = Σ, Vj- The WD distance between é and db is defined as:
Dx(0, min ΣΣTH
Ta c(x..1))
(4)
Tell(u.e)
where [[u, (T IT T v). c(2;, 1;) is cosine distance metric, and T is the transport plan, interpreting the amount of mass shifted from Φ to wij.
Wassersttnn Distance (GWD) assists in edge matching and preserves graph topology by calculating distances between pairs of nodes in each domain, as well as measuring how these
distances compare to the counter domain. In the same discrete graph matching setting, GWD between

---

Under review conference paper ICLR 2023
& and d can be mathematically represented as:
Daw (d. 1b)
min Σ Tel((.,)) (PP.
Vy. 4/ v/)
(5)
where intra-graph, structural similarity between two node pairs 2f) and (H)+3/5) is represented as L(x;, V5. 2/ vf) = |ler (#a, (2 B()l oz being cosine similarity between a node pair in any
graph Gs Transport plan T is periodically updated to align the edges in different graphs belonging disparate modalities.
We further follow Chen et al. (020a) to combine WD and GWD transport plans, leading to a unified GOT objective given as:
(oo v) (ó. v)) + go) (0.
(6)
where regulates the importance of two loss terms.
3.1.3 CROSS-MODAL ATTENTION FUSION (CMAF)
BT and GOT losses computed in dual encoder setting, which does not contain cross-modal interactions and are not suitable complex multi-modal feature representation. Most existing methods, including UNITER (Chen et al. 2020d), ViLT (Kim et al. 2021), METER (Dou et al. 20226) and GLIP (Li et al. 2022c) design cross modal fusion by stacking additional transformer layers on top of uni-modal encoders, introducing large number of added parameters during pre- training. We follow a more efficient solution proposed by FIBER (Dou et al., (022a), which inserts cross-modal fusion into the uni-modal backbones with gating mechanism. Specifically, the top M transformer layers in the vision and language backbone, cross-attention signals, weighted by a gating scalar a, are added to self-attention:
a Self-At((2))
x x + + a Cross-Att(;, y)
x x + FFN(x)
where is trainable parameter initialized O. Following existing literature (Li al. 2021a Wang al. 2021a] laj Dou el al. 2022b a). we use Masked Language Modeling (MLM) and Image-Tee Text Matching (TTM) to pre-train train the cross-attention parameters. For MLM, we randomly mask 15% text tokens, and the loss aims reconstruct the masked tokens. We feed the network with randomly sampled image-caption pairs for ITM, and the loss predicts whether they are matched, The gating mechanism is a good choice for CMAF because (i) cross- coss-attention parameters can casily be switched off by setting the gating scalar 0 when computing the BT and GOT losses. Thus, cross-attention parameters without affecting the original computational flow of uni-modal backbones. (ii) gating mechanism is more lightweight and memory-efficien than adding fusion-specific layers (GLIP and METER use 4x more fusion parameters than FIBER).
learn the
Overall, VoLTA training pipeline can be summarized in the following three steps:
BT GOT: CMAF switched off (ax = 0), VoLTA acts as dual encoder, LgT and LGOT are computed.
MLM & ITM: CMAF switched on (a * 0). VoLTA now acts as fusion encoder, CMLM and LITM losses are computed.
pack-propagation: the four losses are added, giving Ciotal = CeT + weor Lcor + CMIM + CrrM. and back- poockpprppggted into the model end-to-end. An ablation on woor is given in Appendix
33
The overall VoLTA pipeline for computation of different training objectives is shown in Figure 2 The pseudo code for VoLTA can be found in Appendix A
3.2 FINETUNING FOR DOWNSTREAM TASKS
We adopt VoLTA to a wide range of vision- and lasion-language downstream tasks. We switch off the inserted cross-attentoon modules for the vision-only tasks and the image encoder. For the

---

Under review as conference paper ICLR 2023
Table 1: Uni-modal downstream: lincar image classification. We benchmark learned representations on image classification task by training linear classifiers on fixed features. We report top-1 accuracy on ImageNet-Il Ik validation set, classification mAP on VOCO7, and per-class (PC) and overall (O) Fl scores on COCO, Numbers with are re-implemented by Yuan et al (02T)) and the numbers with | are re-implemented by us. methods trained with significantly larger dataset are colored gray. Best results are in bold.
Limer probing on ImageNee Validation Set
Linear probing on VOCO7 and COCO
Method
Pre-train
Arch.
Supersision
Top-1116)
Method
Pre-train
Arch.
NVM
VOCS
XLEP
SUPTPCOOT
COCO
Sap
IN-TK
RN30
Label
16.5
Sip.
IN-T
RS20
67.5
50.8
55.200
Sup.
DN-100
RN50
Label
53.3
SioCLR
DN-KK
RN30
85.5
MoCoo
COCO
RN30
NA
44.5*
MoCoo
IN-1K
RN50
Mo
NA
Moto-v2
IN-1K
KN30
96.1
Viniex
XNC0
Captiou
SWAY
IN-IK
KN3B
58.9
ICMLM
COCO
KN3I
Captioa
51.9
BYUL
IN-1K
KN30
96.6
MCT
CO00
KN30
Caption
59.9
BT
IN-1K
KN30
96.2
91,91
1/3.0'
MCT
0000
RN50
Capson Toe
55.3
VICRay
IN-IK
RN30
86.6
91.1
01S77
VOLITA/Wi CMAAF)
0000
KN3U
Capiio
55.3
NaLITACOO CSIAE)
COO0
RN30
59.5
91.3
71,47443
VaLTTAcwi CMAF)
0000
Swin-1 1
Caption
56.3
NaLTAC CMAF)
0000
Swi-T T
53.2
43.5
73.4/1577
VoLTA//wio CMAF)
00C0
Swat-B B
Captioa
62.5
VoLTAcwio CMAF)
COCO
Swa- 8
59.5
93.9
74,1766.1
VoLIA
0000
Swat 8
Capiion
62.5
VoLIA
0000
Swa 8
58.7
94.0
74,5766
Table Uni-modal downstream: object detection and instance segmentation with fine-tuning. We
benchmark Jearned representations on VOCO7 12 object detection task using faster R-CNN (Ren el. 2015).
and on COCO2017 object detection and instance segmentation using mask R-CNN He et al. C4 backbone variant [Wu et al. 2019). Best results are in bold.
2017). both with
Method
Pre-trat
AF...
APG
APS
Instance SOX
APP
Sip.
IN:IK
382
1m12
DN-IK
57.4
13.5
193
36.5
1K
32.6
38.4
33.8
55.2
SinSium
59.0
34.3
56.0
36.5
MuCo
COCO
31.1
153
35.3
3530
55.6
31.3"
MoCoo x2
COCo
RNG0
52.1
39.8
59.6
35.8
56.9
38.88
VirTex
COC0
61.3
40.9
61.7
36.9
55.4
39.7
MCT
COCO
RNG0
56.1
62.4
82.1
41.1
61.8
44.9
36.9
58.2
40.0
KETTA
COCO
RND0
56.6
02.7
84.4
41.9
61.8
44.5
36.5
59.5
40.8
Sup
IK
Swm
50.5
09.3
54.9
13.7
56.6
47.1
MEaD
IK
3022
03.3
35.
133
76.1
103
MOLTA
0000
Swin
50.9
69.6
55.5
43.8
66.9
47.5
Sap
IN
51.9
7014
45.0
66.4
46-7
VALTA
0000
52.1
113
36.6
45.2
18.3
4000
dision-language tasks, following Dou et al. (2022a), we utilize the learned cross-attention parameters required. For example, VQA and visual reasoning employ all cross-altention modules, whereas captioning requires only image-to-text cross-attention. Again, during IRTR, we switch off cross- attentions and use VoLTA in dual encoder setting. We keep all cross-attention parameters during multi-modal object detection and referring expression comprehension and train an object detection head from scratch using the language-aware image features.
4 EXPERIMENTS
Pre-training & downstream datasets: First, we pre-train VoLTA the image-caption pairs of COC02017 (Lin et al. 2014) split which consists 123k images (118k and 5k for training and validation, respectively) with 5 captions for each image. Following/Chen et al. (2020d) and Huang al. (2021), we scale-up pre- pre-training corpus by appending the VG dataset (Krishna al.) 2017]
with COCO2017, 77, together consisting 231k images. We divide our downstream tasks into three categories (i) uni-modal tasks includes image classification on ImageNet (Deng et al. 2009). VOC07 (Everingham al. 2010). COCO, object detection on VOCO7 12, COCO and instance segmentation on COCO. (ix) multi-moda coarse-grained tasks includes image-level VI. tasks visual question answering on VQAv2 (Antol et al. (2015), visual reasoning on NLVR2 (Suhr et al. 2019) image- and text retrieval on Flicker30k (Plummer et al. 2015) and captioning COCO. (6i) multi-modal fine-grained tasks includes region-level VL tasks referring expression comprehension (REC) on RefCOCO, RefCOCO+, RefCOCOg (Kazemzadeh et al. 2014.] Yu et al. 2016), and nngguac--conditione object detection on COCO and LVIS (Gupta al. 2019). We exclude any overlap between our pre-training and downstream validation/test splits. Detailed statistics all downstream datasets are given in Appendix [
Network architectures: Following FIBER (Dou et al., 2022a), we adopt Swin-Base (Liu et al. 202T) and RoBERTa-Base Liu al. (2019) as our vision and text encoders, which are initialized with weights from uni-modal pre-training. We collect patch- and token features from last transformer layers, and feed them into local projector network to compute GOT loss. Furthermore, we apply AvgPool 1 on patch and feken-fea,ures, and fed them into global projector network compute BT

---

Under review conference paper ICLR 2023
Table 3: Multi-modal coarse-grained downstreams: visual question answering, visual reasoning, retrieval and captioning. We only compare with methods pre-trained on comparable amount of dataset. For captioning, 4 metries are reported- B&4: BLEU@4, M: METEOR, C: CIDEr, S: SPICE. Best results are in bold. VoLTA-B B denotes Swin-B backbone.
Method
SPre-train
Images
dec
YOA2 sdd
des TosE.P
NEVR
FSR IRTR IXSI
TRG1
Methed
SPre-train Images
Birl
COCO Captioning
M
S
Meddlle pre-mmminn in CDCC
230 mndor (GT108))
Modele frur-amood undkoar annnigggnnn
SCAN
108k
48.6
67.4
VL-T3
1sik
54,5
28.7
116.5
21.9
SOG
1081
49.3
71.
VL-BART
180k
35.1
28.7
116.5
21.5
PIAN
108k
50.4
70.0
VOLIN B
25Lk
38.2
30.7
126.5
22.5
Mastaat
123k
54.1
518
VOLTA GOLD-B B
281k
18.9
30.5
128.5
23.0
VeoiBBRR
125k
70.5
71.0
67.
67.0
Ridee Ane--onee with C7DA optioncation
LXMERT
211k
72.4
72.5
74.9
74.5
VoLTA
129k
39.4
50.3
132.5
23.5
SOHO
2311
73.2
73.4
76.3
77.3
72.5
86.5
VaLTA-B
251k
50.7
50.5
133.6
23.7
VoLTA-B
125k
72.8
72.9
69.1
72.9
66.2
80.0
GOLDA-
128k
39.8
36.5
139.5
23.6
VOLIA
72.7
VoLTA GOLD
30,9
loss. Both local and global projector networks has 3 linear layers with dimension 2048-2048-1022
with batch normalization and ReI.U after first two layers. An ablation on projector dimension is given in Appendix [E.4 During downstream tasks, use the image and text features after the AvgPool layer. For CMAF, we insert the cross- noss-attentionn into the top blocks of the vision and text encoders. Morcover, for direct comparison with existing uni-modal baselines, we re-train VoLTA with ResNet50 (He et al. 2016) and Swin-Tiny image encoders.
Implementation details: Detailed implementation setup for pre-training and downstream experi- ments are given in Appendix and [D3] respectively.
4.1 RESULTS ON VISION-ONLY TASKS
We first experiment on three uni-modal tasks - classification, object detection, and instance segmen- tation. For direct comparison with existing ResNet50 and Swin-T baselines, we re-train identical encoders with VoLTA pipeline. Furthermore, since the uni-modal tasks do not utilize cross-attention parameters, we perform an ablation by dropping the CMAF module from VoLTA.
Table T presents the linear probing results of uni-label classification on ImageNet, and multi-label classification VOCO7 and COCO. For all uni-modal tasks, we report results with COCO pre-training for a fair comparison with existing baselines. For ImageNet, we adopt all COCO baselines from Yuan et al. (2021). Even without the CMAF module, VoLTA achieves better performance than all baselines across three datasets with ResNet50 backbone. The Swin backbones and cross-attention module further improve the performance. For VOCO7, we report the results for both SVM and MLP-basee linear classifiers. VoLTA with ResNet50 backbone achieves state-of-the-art results on VOCO7 SVM evaluation, beating the nearest baseline, SwAV, by 0.7 mAP score. These results indicate the ability of VoLTA to learn effective image-level visual features.
Image Classification:
Object Detection & Instance Segmentation: Next, we perform two uni-modal region-level tasks object detection VOCO7 + 12 and COCO2017, and instance segmentation COC02017. As shown in Table 2 VoLTA yields the sfate-of-the-rrr art performance in both tasks across majority metrics. Worth noting, VoLTA, pre-trained with 123k image-caption pairs achieves better performance than baselines pre-trained with 1.3M images of ImageNet, proving the efficacy of VLP with fine-grained patch-token alignment over vision-only pre-training.
4.2 RESULTS ON COARSE-GRAINED "ISION-LANGUR TASKS
Next, we perform image-level muuli-modaa downstream tasks visual question answering (VQA), visual reasoning, retrieval, and captioning.
VQA& & Visual Reasoning: As reported in Table VoLTA achieves the best performance on VQA and visual reasoning across the baselines pre- pre-trained with a comparable amount of data. Moreover, on VQA, VoLTA beats LXMERT, which is trained with 2x more data. These results demonstrate the efficacy of our method even when utilizing mid-scale pre-training corpus.
Most existing VLP methods use fusion encoder for image and text retrieval and feed every image-text pair into the model. Though such fine-tuning often results in higher performance, it introduces quadratic time-cost and is not scalable. Following Dou et al. (2022a), we adopt a more efficient strategy. We drop the cross-attention parameters for this task and compute the dot product of image and text features extracted separately in the dual-encoder setting. As shown in Table even with such an approach, VoLTA produces superior performance among the baselines trained with a similar amount of data, beating all three baselines by a significant margin.
Retrieval:

---

Under review conference paper ICLR 2023
Table 4: Multi-modal fine-grained downstream: referring expression comprchension. Methods pre-trained on mn-Txt-BBx data or significandly larger amount of Im- data are colored gray. Best comparable results are in bold. VoLTA B denotes Swin-B B backbone.
Method
Data
ReCOCO
ReCOOCO+
ReCOCog
Im-1xt
Im-Txt-Box
val
tesTA
testB
val
testA
test
val
lest
MAnNNt
76.4
B0.4
69.3
64.9
70.3
56.0
66.7
67.0
VLBERT
3M
21.6
77.7
61.0
VILBERT
72.3
62.6
Ennee-VL-Large
75.9
82.4
66.9
Rosita
1M
84888
88.0
78.3
76.1
82.0
67.4
78.2
78.3
UNITER-L
-M
81.4
87.0
74.2
75.9
81.5
66.7
74.9
75.8
VILLA-1
76.2
Models pre rreiiee on 7 Dett datei
MDLIK-B
1.160
87.5
9014
82.77
81.1
85.5
73.0
83.4
83.3
UnITAB
1.3M
86.3
88.8
80.5
78.7
83.2
69.5
80.0
30.0
X-VL
80.2
71.0
OFA-L
10M
3M
90.1
92.9
85.3
84.5
90.1
77.8
84.5
85.2
FIRRR-R
4M
0.5M
90.7
92.6
87.3
85.7
90.11 1
79.4
87.1
87.3
VOLTA
125k
83.9
86.3
79.4
73.5
78.8
65.5
15.2
75.9
VoLTA-B
88.6
81.8
77.0
67.8
78.3
Captioning: We perform captioning on the COCO dataset to evaluate if VoLTA can adopt a generation task. We integrate GOLD (Pang & He (02T) into VoLTA during fine-tuning it produces significant improvements. As shown in Table our approach maintains superior captioning performance across all baselines pre-trained with comparable data. Using CIDEr optimization further improves performance.
is worth mentioning that besides achieving superior result than all baselines using comparable amount of data on multi-modal coarse graeeeg tasks, VoLTA also outperforms multiple methods pre-trained using magnitude more data. These results, shown in Table F.T, indicate the effectiveness and generalizability of VoLTA across these tasks.
4.3 RESULTS ON FINE-GRAINED VISION LANGUAGE TASKS
Next, we perform region-level mulli-modal down- stream tasks- referring expression comprehen- sion (REC) and language-guides object detec- tion.
Table 5: Multi-modal fine-grained downstream: language-condittonee object detection on COCO and LVVIS All available haselines pre-trained on Im-Txt-Box data, and are colored gray. VoLTA-B de- notes Swin-B B backhone.
REC: This task aims to localize target objects in an image described by a referring expres- sion phrased in natural language and, thus, per- fectly evaluates the fine-grained feature represen- tation capability of VoLTA. As depicted in Ta- ble[4] VoLTA beats larger-sized UNITER-L L and VILLA-I. models on the challenging testB split of both RefCOCO and RefCOCO+. Morcover, VoLTA performs comparably with MDETR and UniTAB, even without being trained on grounding data. These results indicate our model's s efficacy in learning fine-grained local visual features.
COCO Val 2017
LVIS MiniVal
Method
AP
APr
APC
API
AP
Models pre-trained Im-Txx-Box andior with larger size
Mask R-CNN
26.3
34.0
33.9
33.3
MDETR
20.9
24.9
24.3
24.2
GLIP-B
57.0
31.3
48.3
56.9
51.0
GLIP-II
60.8
FIBER-B
58.4
50.0
56.9
58.1
56.9
VoL.A-BB
51.6
34.4
43.1
43.8
42.7
Object Detection: We evaluate VoLTA on two challenging language-cone conduuii-ood object detection benchmarks COCO and LVIS. Note that, all existing baselines for this tasks are pre-trained on fine-grained image-texi-bon box data, whereas VoLTA only utilizes image-caption pairs. As shown in T VoLTA performs comparatively with these strong baselines. Note that VoLTA beats Mask R-CNN, MDETR, and GI.IP-B LVIS APr, which denotes average precision rare objects. Thus, we conclude that VoLTA achieves impressive localization ability and robustness, even without utilizing any grounding annotations.
5 CONCLUSION
We present VoLTA, a unified VLP paradigm that utilizes image-caption data but achieves fine- fine-grained region-level image understanding, eliminating the use of expensive box annotations. VoLTA adopts graph optimal transport-based weakly supervised patch-token alignment and produces an explicit, nolf-normallzedd and interpretable low-level matching criterion. Extensive experiments demonstrate the effectiveness of VoLTA on wide range of coarse- and line-grained tasks. In the future, we plan pre-train VoL.A larger-scale datasets.

---

Under review conference paper ICLR 2023
REPRODUCIBILITY STATEMENT
The pre- pre-training code of VoLTA can be found in the supplementary material. We also provide a detailed implementation setup for pre-training and downstream experiments in Appendix D.2 and D33 respectively. After publication, will provide pre-trained checkpoints and open-source the code on a public repository.
REFERENCES
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European conference computer vision, pp. 388-398. Springer. 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-24233 2015.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael G. Rabbat, and Nicolas Ballas. Masked siamese networks for label- efficient learning. ArXiv, abs/2204.07 141, 2022.
Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation. In Proceedings of the IEEE/CVE International Conference on Computer Vision, pp. 6500-6508, 2019.
Satanjeev Banerjee and Alon Lavie. Meteor:
An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summurization, pp. 65-72, 2005.
Hangho Bao, Li Dong, Songhao Piao, and Furu Wei. Beit: Bert pre-training of image transformers.
In International Conference on Learning Representations, 2021. 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun, Vicreg: Variance- aawaaa---iivarinnc--ovaaaaaa regularization for self-supervised learning. In International Conference on Learning Representations, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 13:1877-1901, 2020.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' egou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in superviee vision transformers. 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9630-9640, 2021.
Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda, CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification. In International Conference Computer Vision (ICCV), 2021a.
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. Improving sequence-to-sequems learning via optimal transport. In International Conference on Learning Representations, 2019. https:
//openzeview.net www.:rrrnnnnddittass--
Liqun Chen, Zhe Gan, Yu Cheng. Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning, pp. 1542-1553. PMLR. 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. simple framework for contrastive learning of visual representations. In International conference on machine learning, pp- 1597-1607. PMLR, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning, 2021 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 15745-15753, 2021.
10

---

Under review conference paper ICLR 2023
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 04297, 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. 2021 IEEF/CVF International Conference on Computer Vision (ICCV), pp. 9620- 9629, 2021b.
Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. Uniter: Universal image-text representation learning. In European conference on compuler vision, pp. 104-120. Springer, 2020d.
Jaemin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying lision-and-len lan-laag tasks via text generation. In International Conference on Machine Learning, pp. 1931-1942. PMLR, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. 2009 IEEE conference on computer vision and pattern recognition, pp. 248-2582555 255. leee, 2009.
Jacob Devlin, Ming Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers). pp. 4171-4186, 2019.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer. Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition scale. In International Conference on Learning Representations, 2021. httphtttt////tenttt http:///peeeeeeieeeeeee net/ http//ooumndddddeoo ENTTY
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, al. Coarse-to-fine vision-language pre-training with fusion in the backhone. Advances in neural information processing systems, 2022a.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang. Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng. al. An empirical study of training end-to-end vision-and and- languag transformers. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18 18166-18176, 2022b.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journaa of computer vision, 88(2): 303 338, 2010.
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894-6910, 2021.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun. On duality between contrastive and non-contrastive self supervisdd learning, arXiv preprint arXiv:2206.02574,
2022.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn di- vergences. In International Conference Artificial Intelligence and Statistics, pp. 1608-1617. PMLR, 2018.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and P. Abbeel. Multimodal masked autoencoders learn transferable representations. ArXiv, abs/2205.14204, 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Picter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch. Yangqing Jia, and Kaiming He. Accurate, large minibatch sgd: Training imagenet hour. arXiv preprint arviv: 776777 2017.

---

Under review conference paper ICLR 2023
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised learning. Advances in neural information processing processsig systems, 33:21271-21286 2020.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pp. 5356-5364, 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances neural information processing systems, 28, 2015.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEF conference on computer vision and pattern recognition, pp. 770-778, 2016.
Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 9729-9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 16000- 16009, 2022.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020.
Zhicheng Huang. Zhaoyang Zeng, Yupan Huang. Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for lasoua- representation learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 12976-12985, 2021.
Chao Jia, Yinfei Yang, Ye Xia, Yi-Ting Chen, Zarana Parekh, Hieu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Li, and Tom Duerig. Scaling up visual and Langun-anguage representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916, PMLR, 2021.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mddtr-modulated detection for end-to-end multi-modal understanding. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1780-1790, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 conference empirical methods natural language processing (EMNLP), pp. 787-798. 2014.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitransformers for classifying images and text. ArXiv, abs//199..nnm 1119999999000 2019.
Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision-aandllanuuag transformer without convo- lution region supervision. In International Conference Machine Learning, pp. 5583-5594. PMLRR, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. International journal of computer vision, 123(1):32-73, -3, 2017.
Juncheng Li. Xin Hc, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xic, Yucting Zhuang, Qi Tian, and Siliang Tang. Fine-grainee semantically aligned vision- language pre-trainin training. In Advances in neural information processing systems, 2022a.

---

Under review conference paper ICLR 2023
Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694-9705 2021a.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip:
Bootstrapping language image pre- training for unified vision-language understanding and generation. In International Conference on Machine Learning, 2022b.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does bert with vision look at? In Proceddings of the 58th Annual Meeting of the Association for Computational Linguistics, pp. $265-5275, 2020a.
Liunian Harold Li. Pengchuan Zhang, Haotian Zhang. Jianwei Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang. Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre- pre-traiming. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
10965-10975, 2022c.
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 16420-16429) 2022d.
Xiujun Li, Xi Yin. Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang. Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. European Conference on Computer Vision. pp. 121-137. Springer. 2020b.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan, Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference on Learning Representations, 2021b.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755, Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint 19Xiv:1907.11192 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10012-10022, 2021.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task agnostii visiolinguistic representations for vision-and-language tasks. Advances in neural information processing systems, 32, 2019.
Zhiyuan Ma. Jianjun Li, Guohui Li. and Kaiyan Huang. Cmal: A novel cross-modal associative learning framework for lasion-language pre-training. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4515 4524, 2022.
Youssef Mroueh, Chun-Liang Li, Tom Sercu, Anant Raj, and Yu Cheng. Sobolev gan. In International Conference on Learning Representations, 2018.
Youssef Mrouch, Tom Sercu, and Anant Raj. Sobolev descent. In The 22nd International Conference on Artificial Intelligence and Statistics, pp. 2976-2985. PMLR, 2019.
Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. Slip: Self- supervieeeinn meets prrguage-image pre-training. ArXiv, abs/21 12.500 2021.
Aaron den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding, arXiv preprint arXiv:1807.0374 037444 2018.
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In ICLR, 2021.

---

Under review conference paper ICLR 2023
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wasserseinn averaging of kernel and distance matrices. International Conference on Machine Learning, pp. 2664-26662222 2672. PMLR. 2016.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 1(5-6))355-607, 2019.
Bryan A Plummer, Liwei Wang. Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting eegion-to-phras correspondences for richer amage-to-sentence models. In Proceedings of the IEEE international conference on computer vision, pp.
2015.
Shraman Pramanick, Aniket Roy, and Vishal M Patel. Multimodal learning using optimal transport for sareasm and humor detection. In Proceedings the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 9930-3940, 2022.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International Conference on Machine Learning, pp. 8748-8763. PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojciech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 2022.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. VI-bert: Pre-training of generic vinual-iinguitti representations. In International Conference Learning Representationss 2019.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi A corpus for reasoning about natural language grounded in photographs. In Proceedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transform- In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-UJCNLP), pp. 5100-5111, 2019.
Ramakrishna Vedantam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4566-4575. 2015.
Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for language and video-language tasks. In Advances neural information processing systems, 2022a.
Peng Wang, Yang, Rui Men, Junyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through a simple sequence-to-sewe -o-sequnce learning framework. In International Conference on Machine Learning, pp. 18-4440. PMIRR 2022h.

---

Under review conference paper ICLR 2023
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vasion-language pre-training arXiv preprini
with -oxxure-o-mmodaltt---xxpop
2021a.
Wenhui Wang, Hangbo Bao, Dong, Johan Bjorek, Zhiliang Peng, Qiang Liu, Kriti Aggarwal. Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a foreign language: Beit pretraining for all vision and vision-language tasks, ArXiv, abs/2208. 10442, 2022c.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia Tsvetkov, and Yuan Cao. Simvim: Simple
visual language model pretraining with weak supervision. In International Conference on Learning Representations, 2021b.
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance on text classification tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLPP) pp.
2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Yen Lo. and Ross Girshick. Detectron2. 2019.
https://gi hhb. htwphu/.oooaaacaiiaiie www.cccooooeeeeeeeeeeeeeeeeeeeee
Hongwei Xue, Yupan Huang, Bei Liu, Houwen Peng, Jianlong Fu, Houqiang Li, and Jiebo Luo. pre-training.
Probing inter-modality: Visual parsing with self-attention for vision-and- Advances in Neural Information Processing Systems, 34:4514-4528, 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-Beeeeee leb--label space. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 19163-19173 2022a.
Jinyu Yang. Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng. Trishul Chilimbi, and Junzhou Huang. Vision-language pre- pre-training with triple contrastive learning. In Proceedings of the IEFF/CVF Conference Computer Vision and Pattern Recognition, pp. 15671-15680, 2022b.
Zhengyuan Yang. Zhe Gan, Jianfeng Wang, Xiaowei Hu, Faisal Ahmed, Zicheng Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In ECCV, 2022c.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang. Zhenguo Li, Xin Jiang, and Chunjing Xu. Filip: Fine- tine-erained interactive International Conference on Learning Representations, 2022.
pre-treee pre-training, In
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-shared contrastive language-image pre-training. In European Conference Computer Vision, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint 17XXv:1708.03888 2017.
Licheng Yu, Patrick Poirson. Shan Yang, Alexander C Berg, and Tamara L Berg. Modeling context referring expressions. In European Conference on Computer Vision, pp. 69-85. Springer, 2016.
Siyang Yuan, Ke Bai, Liqun Chen, Yizhe Zhang, Chenyang Tao, Chunyuan Li, Guoyin Wang, Ricardo Henao, and Lawrence Carin. Advancing weakly supervised cross-domain alignment with optimal transport. In BMVC, 2020.
Xin Yuan, The Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Faieta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In International Conference on Machine Learning, pp. 12310- 12320. PMLR, 2021.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grainer vision language pre-training Aligning texts with visual concepts. In International Conference on Machine Learning, 2022.
15

---

Under review conference paper ICLR 2023
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Few-shot image classifica- tion with differentiable earth mover's distance and structured classifiers. In Proceedings of the IEEF/CVF conference computer vision and paltern recognition, pp. 12203-12213, 2020.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unifying localization and vision-language understanding. Advunces in Neural Information Processing Systems, 2022.
Pengchuan Zhang. Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang. Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021.

---

Under review conference paper ICLR 2023
A
PESUDO CODE OF VOLTA
The training psuedo code for VoLTA as follows:
Algorithm/ Py PyTorch-style pscudocode for VoLTA.
FLT: Tmage Encoder, f_T: Text Encoder
task_ _aak_nnmmes: string containing task names
4 I: Image input, T: Text input, N: Batch size, D: Projector dim
4 BT: Barlow Twins loss function
WD, GWD: Wasserstein and Gromov-Wasserseiin loss functions
1TM: and
loss functions, respectively
garma: coefficient of GMD loss in GOT w_GOT: weight of GOT loss
def GOT x_2, f_1, f_2): It compute igg 2_ 2_B [_1(x_1) 2x H
# normalize representation along batch dimension
2 norn = A A 2 A.man (dim-0)) / 2 a.dd
norr B m.man (dim=0)) 2 (dd
# cosine distance matrix
C cos itt_aaltttmtaiiiie..... 2_R)
Wasserstein distance
loss_w (c, (_),, (_A,, 2_B.size(1)
# Gromov- distance
loss gu
GW D (2_ t.ransppse (2,1), 2 c_B.rranspoee (2)
return gamma torch. (less gw) (1 gamma) torch.mean (loss_ w)
def VoLTA (I, :):
total loss wtrchhtteensee (0.)
for x in loader: # load a batch with N samples
+ two augmented versions of I,
I1,
augment_image
augment_texx (T)
if "BTGoT" in task_names: names:
# вт less
intra loss (1111, 12, f (T f inter loss = 2T Th, E_I, f-T) + BI, T2, f.I. f_1)
BT BT_loss = intra loss loster_osss
total_loss
Bo5s
# GOT loss
GOT loss GOT (11, T1, f f_T) + GOT (12, 12, f_1, f_m)
total loss w GOT loss
cross-attention is enabled if "MLM" in task_names:
# MLM loss MiM_loss MLM( II, mask_Tl, I_I, E_T)
total_loss McM_
if "1TM" in task_names: names:
ITM loss [IM_1oss =ITMITI, 11, false image_l, E_I, f_T) total total_loss += ITM_coss
total_lost

---

Under review conference paper ICLR 2023
Table B.1: Overview of VLP models. OD: objective detector. Xformer: transformer. Emb. embedding.
MIMM/MIM masked language/image modeling. ITM: image-text matching. WRA: word-region alginment.
ITC: image-text contrastive learning. Grnd: Grounding. Cap: Captioning. TP: Token Prediction. CA: Contrastive Alignment, NNS: Nearest Neighhour Supervision, MVS: Multiview Supervision, SL. Sim-siam Loss, MHA: Multi head attn., LM: Language Modeling, UniVLC: Unified Vision Language Contrastive, VLM: Vision Language Matching, BBP: Bounding Box Prediction, MP: Matching Prediction, FSA: Fine-grained Semantic A Aligument, TSA: Token-level Semantic A Aligament, AMC: Associative Mapping Classification, CMA:
Cross- Alignment, IMC: Intra-Modal Contrastive, LMI: Local Mutual Information Maximization, 1-T:
Image-Text, Text, I-T-B: Image-Text-Box.
Model
Venue
Vision Encuder
Text Enc.
Multimodality Fusion
H
Pre-train
I-T-B
Pre-training Objectives
WILRERT
NeuriPS
OD+Xlormer
Xformer
Co-atin.
MIMMITTMEMMM
LXMERT
EMNLP'14
MLM-ITM-MIM--QR
VisuaBBERT
ACI/20
MLM-IITM
VL-BERT
ICLR'20
MLM--MM
UNITER
ECCV"20
MLM-ITM--MM--WRR
OSCAR
ECCV'20
OD
Binh.
Mergad-artn.
MLM-IIM
VinVL
CVPR'21
MLM-ITM
VL-TS
ICML/21
SOTO
CVPR21
Embh
MIM--INF-XIIX
SimVLM
ICLR'22
CNN
Merged atin.
PrefisLM
MDETR
ICCV221
Xlommer
ODFTP-CA
VILT
ICME221
Finh.
Embh
Merged-artn.
MIMM-ITM
Visual Parsing
NourlPS 21
MLM-ITM-MIM
ALBEF
NeurlPS'21
Xlomer
MLM-ITM-ITC
METER
CVPR'22
Xtormer
MLM-ITM
CLIP
ICML221
CNNXiimmer
ITC
DeCLIP
ICLR
Nome
ITC-MLM+SL-MVSAN ITC
ALIGN
ICML/21
"CNN
GLIP
CVPR'22
OD&CEAWRA
GLIP22
NeurlPS 22
OD+Xtormer
OD-CE-WRA-MLM
BLIP
ICML/22
Cross modality MIIA
ITC-IIM-LM
OmniVI.
NeurlPy
X-VLM
ICME
Xformer
PRP-ITC-MP-MI MLM
Xformer
CMAL
ACM MM'22
AMC+ MEM-LL-MMM-MM-IICE
LOUPE
NeurlPS 22
None
IICHESA+TSA
FILIP
ICLR
ITC
UniCL
CVPR'22
CNNDXfomer CNN
ITC
UnITAR
RCCV"22
LM
Merged
TCL
MS-CLIP
CVPR'22
ECCV'22
Xlormer
Shined "ShinddTiitenbbor
ITC MLM-IIMAIC
FIBER
NeurlPy
Xlomer
Xformer
Merged Co
VoLTA
CNN/Xformet
Xlommer
Merged Co-atin
BL-GOT
B OVERVIEW OF VISION-LANGUAGE PRE-TRAINING MODELS
Vision-Language Pretrained (VLP) models have proven extremely beneficial for multi-modal tasks the recent years. Earlier works were predominantly focused on using pre-trained object detectors to extract patch (region) level information from corresponding images (Lu et al. 2019 Li al. 2020a Tan Bansall 2019 Chen et al. 2020d) [Su et al. 2019).. In some these models, such as ViLBERT
(Lu et al. (019) and LXMERT (Tan & Bansal (2019). multi-modality fusion has been achieved via co-attention which utilizes third transformer containing fused information independently obtained
from respective vision and language encoders (transformer). On the contrary, VisualBERT {Li
ct al. 2020a). VL-BERT (Su et al.] (2019) and UNITER (Chen et al.] (0204) employ a merged attention strategy to fuse both image patches and text features together into unified transformer through corresponding image and text embedders.
addition to image patches and texts a unified transformer, OSCAR (Li et al., 20206) uses object tags as inputs. VinVI. (Zhang et al. (20TT) follows similar strategy to that of OSCAR, the only difference being their novel 3-way contrastive loss which optimizes the training objectives used for VQA and text-image matching. VL-T5 (Cho al., 20 exploits bounding-box coordinate information, image IDs, and region IDs along with ROI features for visual embedding. Encoded visual and textual features are fed into bi-directional multi-modal encoder and an auto-regressive text decoder framework for pre-training.
In all the aforementioned methods, pre-trained object detectors are usually frozen during the training. and extracting region-level features from images can be tedious. To address these shortcomings. end-to-end pre-training methods have been developed. PixeIBERT (Huang et al. 2020) uses convolu- tional neural networks (CNNs) based visual encoder and sentence encoder to obtain image and text representations, respectively. These representations are then fed to a subsequent transformer via a cross-modality alignment. SOHO (Huang al. 202T) uses grid features discretization via learned
18

---

Under review as conference paper ICLR 2023
vision dictionary which are then fed into cross-modal module. SimVLM (Wang et al. (021b) uses CNN and text token embedding for image and text feature representation extraction along with a unified encoder-decoder transformer trained on PrefiXLM objective. Finally, MDETR (Kamath el. )021) uses CNN and RoBERTa (along with corresponding projection layers) for image and text feature extraction, which are then concatenated before passing through a unified transformer trained on 1.3M Image-Text-Box (I-T-B) annotated data.
In recent years, the rise of Vision Transformers (ViT) (Dosovitskiy et al., 2021) has motivated the research community to have an all-transformer framework by incorporating ViTs (instead
CNN backbones) in VLP models. Image patch features and text token embeddings are fed directly into ViT model for pre-training in VILT (Kim et al. (202I). Visual Parsing (Xue et al.) [ ALBEF (Li et al, aa) and METER (Dou et al. (022b) use ViTs as vision encoders for image feature generation. For multimodal fusion, ALBEF and METER use co-attention in their pre-training frameworks,
Another class of VLP models in the form of CLIP (Radford al. 2021,, DeCLIP (Li et al. 202
and ALIGN (Jia et al. 2021) have been introduced lately. Although, known for their impressive zero-shot recognition ability and excellent transferability to downstream tasks, these models typically rely on huge amount of image-text pairs for pre-training. Contrastive loss forms the core component of the pre-training objectives in these VLP models. In such models (c.g., CLIP (Radford et al. (02T).
DeCLIP (Li et al., (202Ib)) separate encoders have been used for each modality. On the contrary. Modality-Shared Contrastive Language-Iman Imagg Pre-training (MS-CLIP) (You et al. (022) leveraged knowledge distribution across multiple modalities (image and text) through parameter sharing. In their unified framework, the parameters which are being shared between two modalities include the attention and feedforward modules, and the LayerNorm layers.
GLIP (Li al. 2022c) and GLIPv2 Zhang et al. 2022) use a localization loss along with word- region alignment loss for pre-training corresponding encoders using image-text-box annotations. BLIP (Li et al. 2022b) employs image and text encoders connected through cross-modality multi- head attention which are pre-trained on image-text pairs using contrastive and language modeling objectives. OL (Wang et al. 2022a) utilizes a unified image(and video) encoder and a text encoder pre-trained on image-text, magec-lbel,, video-text and video-label pairs using unified vision- language contrastive, vision- lainu-l matching and language modeling losses. Furthermore, a visual-grounded alignment decoder is also present for facilitating better learning and alignment between various modalities. X-VLM ZZng et al.) (022) employs vision transformer to extract features from the subset of patches representing mnages/rggionssoojectssss These patch features are then paired with associated text features for contrastive learning, matching, and MLM. Additionally, image and text pairings are also done for bounding-boo prediction which is used to locate visual concepts in the image. CMAL (Ma et al. 2022) proposes interactions between features (obtained from respective image and text encoders) via cross-modal associative mappings which help in fine-grained semantic alignment between the learned representations. LOUPE (Li et al. 2022a) implements token-level and semantics-level Shapley interaction modeling with global image-text contrastive loss (in dual- encoder setting) for explicit learning of fine- fine-grained semantic alignment between visual regions and textual phrases without using expensive bounding-hor annotations. FILIP (Yao et al. [2022) removes the need for cross-modality attention fusion by modeling the fine-grained semantic alignment between visual and textual tokens via a novel cross-modal late interaction mechanism in the contrastive loss.
TCL YYang et al. 2022b) uses global cross-modal alignment, intra-modal alignment and local mutual information maximization losses along with Masked Language Modeling and Image- Text Matching to learn robust image-text representations during pretraining. UniCL (Yang et al. (2022a) utilizes a unified learning method with two-way contrastive loss (image-to-text and text-to-image) in the image-text-label space which can leamn representations from either of the image-label and image-text data or both. TniTAB (Yang et al. (022c) employs a transformer based encoder-decode framework which can jointly output open-ended text and box, encouraging alignment between words and boxes.
FIBER (Dou al. 2022al fuses vision and language encoder backbones through merged co co-attention which are then pre-traine trained on 4M data with two stage pre-training (coarse- and fine-graing grained). Image- text pairs are used in the coarse-grained pre-training stage which is then followed by fine-grained pre- praiiigi stage with image-text-box annotations. However, these bounding box annotations come with extra overheads. Therefore, in our model, VoLTA, we propose an alternate solution for optimal- transport hased local feature-level alignment using global image-caption annotations which performs well not only coarse- grained tasks (such as VQA and Image Captioning). but also on fine- -naiird
19

---

Under review conference paper ICLR 2023
Table C.1: Dataset statisties for uni- uni-modal and multi-modal downstream tasks.
Modality
Dataset
Imuge
#lmages
IN-IK
IN-IK
13M
Accuracy
Image Classfn.
COCO
VOCU7+12
VOCU7-12 12
16K
mAP
Uni-modal
"CRO0"
"CUCO""
T23R
Object Det.
VOCU7+12
VOC07-12
16K
AP
Irstrinee Seg.
COCO
"COCO"
T23R
VQA NLVR IR-TR
VQA
COCO
204K
LIM
VQA-Soore
Multi-medal
NLVR
Crawled
214K
107K
Accuracy
coanse coanse-grained
Flick30KK
Flick33KK
32K
160K
Recall@1
Captioning
COCO
COCO
123K
615K
B0LMC.S
RefCOCO
20K
142K
Ref. Exp. Comp.
RefCOCO
20K
142K
Accuracy
Multi-modal fine-grained
RefCOCOg
26K
95K
CCCC
COCO
615K
Mul. Obj. Det.
LVIS Mini
COCO
123K
615K
tasks (such as Referring Expression Comprehension and Object Detection). Table B. encapsulates an overview of the details of all these aforementioned methods.
C DOWNSTREAM DATASETS
Our downstream tasks can be categorized into three groups: uni-modes and multi-modal fine-grained.
coarse-grained,
Uni-modal: For uni-modal tasks, we fine-tune (and validate) our pre-trained model on ImageNet- Ik (Deng et al. 2009) for Image Classification, VOC07+12 (Everingham et al. 2010) for image classification and object detection, and COCO (Lin et al. 2014) for image classification, object detection and instance segmentation.
Multi-modal Coarse-grained: Here, we fine-tune (and validate) our pre-trained model on VQAv2 (Antol et 2015) for visual question answering, NL.VR2 (Suhr al. 2019) for visual reasoning, Flickr30k (Plummer et al. 2015) for image and text retrieval, and COCO (Lin et al. 2014 for image captioning.
Multi-modal Fine-grained: For these tasks, fine-tune (and validate) pre-trained model on RefCoco, ReiCOCO+, and RefCOCOg (Kazemzadeh et al. (2014) Yu et al., (2016) for referring expression comprehension, and COCO (Lin el al. 2014) and LVIS Mini (Gupta et al. (019) for language-conditioned object detection.
It is be noted that several multi-modal downstream tasks are built based the COCO dataset where the validation and test splits of these tasks are scattered across the raw COCO splits. Therefore, when pre-training model, we carefully selected the portion of the COCO dataset which does not overlap with the validation/test splits of these multi-modal downstream tasks.
D IMPLEMENTATION DETAILS& &
VALUES
D.1 DATA AUGMENTATION
We use ResNet50/Swin, T/Swin-B et. 2016 Liu et al. 2021) as image encoder and RoBERTa (Liu et al. 2019) as text encoder. Encoders are followed by corresponding projector network which where d represents the embedding
a 3-layer MLP with the configuration [d-2048-204s)
dimension of the encoder's output.
Image Augmentations: Two sets of random transformations sampled from an augmenta- tion pool are applied on each input image to generate two disparate distorted views. The augmentation policy is composed of LandomResizedCrr ColorJitter, RandomGrayscale where RandomRe s zedCrop is applied with a probability of 1.0, whilst the remaining ones are applied randomly with varying probabilities following Zhontar al. (202T) as outlined in Table D.I
RandonHori zontalFlip, GaussianBlur, and Solarizat. ion augmentations
20

---

Under review conference paper ICLR 2023
Table D.I: Image and text augmentation details.
Data Type
View
Augmentation
Probability
RandomResi zed
1.0
RandomHor Flip
0.5
Co orJittar
0.8
RandonGrayscal
0.2
Saussianns Br
1.0
Solarization
0.0
Image
2
RansenResiieddFop
[.0
Randenttori.zsntsi....
0.5
ColorJitter
0.8
RandemGrayscale
0.2
2
GauasianBLus
0.1
Solarizatien
0.2
Synonym Replacement
Random Insertion
0.1
Random Swap
0.1
Random Deletion
Text
2
Synonými Replacement
0.[
2
Random Insertion
0.2
2
Random Swap
0.1
2
Random Deletion
0.2
Text Augmentations: Two sets of random transformations are applied on input text using EDA (Wei & Zou 2019) including synonym replacement, random insertion, random swap, and random deletion with different probabilities outlined in Table D.I
D.2 PRE-TRAINING SETUP
Table D.2 shows the details of hyper-parameters used during training.
Our proposed VLP model, VoLTA comprises vision encoder and language encoder with merged
co-attention for cross-modality fusion. While conducting experiments, we have considered two types of vision encoder backhones- ResNet-50 (He el al. (2016) and Swin Transformer (Liu et al.) (02T). is to noted that for fair comparisons with related works (Dou et al. 2022b a). input image resolution for ResNet-50 encoder backbone is kept as 224 x 224, whereas for Swin-B, is 384 x 384. The output embedding dimension of the image encoder in both cases is 1024. Similarly, be consistent with Dou et al. (2022b a). we have selected RoBERTa as the language encoder with vocabulary size of 50265, tokenizer as robeeta-base" maximum input text length of 30, and an output embedding dimension of 768 (please refer Table D.2 for more details).
Vision and language encoders are individually followed by projector heads, each consisting of linear layers each with 2048 output units (except for the last one which has 1024 output units) accompanied with a Batch Normalization layer and ReLU activation (for exact configuration please refer to Table D.2 The final projected output denotes the feature representation of the input (image and/or text) which are used for downstream tasks. order to learn these representations, the embeddings (i.c., output from respective encoders) are fod to the loss function of VoLTA.
The loss function of VoLTA includes four different loss components, namely, multi multi-modal Barlow Twins for intra- and inter modality redundancy reduction, GOT for alignment of local features. MLM and ITM together for encouraging cross-mon modal attention fusion. For MLM, we randomly mask 15% (MLM probability in Table (D.2 of the input tokens and the model is trained reconstruct the original tokens. For ITM, the model predicts whether given image-text pair is matched.
follow the same protocol described in Zbontar et al. (202I) where we use the LARS (You et al.) (2017) optimizer to train our model for 20 epochs with a batch size of 256. A base LR of 0.1 is used for the weights and 0.0048 for the biases and batch normalization parameters which are then multiplied by a factor of We employ a learning rate warm-up (linear) upto a period 2 epochs followed by a cosine decay schedule to reduce the LR by a factor of 1000. A weight decay parameter of 1c-6 is used excluding the biases and batch normalization parameters. We conducted
For optimization,
Following BERT, we decompose this 15% into 10% random words, 10% unchanged, and 80% with a special token [MASK].

---

Under review conference paper ICLR 2023
grid search for the GOT loss hyperparameter (wGoT) and we empirically found the best value to be 100.
Table D.2: Pre-training hyper-parameter details for VoLTA.
Hyper-paaameeers
Notation
Value
Model
Img. proj. layer conbg.
[d. 20-88 2018
1024
Img. embed.. dim
1024
Img. resu. (RN-50 & Swin-T)
224 x 224
Img. reso. (Swin-B)
384 x 384
Txl. proj. layer conlig.
[d. 2048 2048 1024
Txt. embed. dim
768
Tokenizer
'roherta--usee
Vocab size
50265
ML. proh.
0.15
Max. length of text
30
# Heads of Xlurmer
12
# Layers of Xfonner
12
# Fusimm bluck
6
Dropout rale
0.1
Task names
BIGGT,, MLM. ITM'
Training
Batch size
256
Epochs
20
Lambda_ BT
0,005
WD and GWD loss weight
0.1
GOT loss weight
WOOT
100.0
Optimizer
LARS
Base I.R for weights
Basc LR for biases
0.0048
Momentum
0.9
LR scheduler
Cosine LR decay (with lincar warm-up)
Warm-up steps
2 x Epochs
Weight decay
1c-6
Find I.R Factor
0.001
Cosine LR amplitudo factor
0.5
Pre-training cost: Our Swin-B backbone takes 6 hours per epoch to train on 64 V100 GPUs, with per GPU batch-size 4.
D.3 DOWNSTREAM SETUP
D.3.1 UNI-MODAL DOWNSTREAM TASKS
Linear Evaluation: For ImageNet, the linear classifier has been trained for 100 epochs with batch size of 256, LR of 0.3 and cosine LR schedule. Cross-entropy loss is minimized with SGDM optimizer (momentum
0.9) and weight decay of le-6. For both COCO and VOC, the linear classifier has been trained for 100 epochs with AdamW W optimizer with batch size of 256, LR of 5e-2 and weight decay of 1e-6.
For training the detection model, the detectron2 library (Wu et al. 2019) has been used. The backbone networks for Faster R-CNN (Ren et all 2013) and Mask R-CNN (He et al. 2017 has been initialized using our pre-trained model.
Object Detection:
For VOCO7+12 12, used the trainval set comprising of 16K images for training Faster R-CNN (Ren et al. 2015) C-4 backbone for 24K iterations using batch size across 8 GPUs using SyncBatthhNorm. The initial learning rate for the model is 0.15 which is reduced by a factor of 10 after 18K and 22K iterations. Linear warmup (Goyal et al. 2017) is used with slope of 0.333 for 1000 iterations.
For COCO, Mask R-CNN (He et al. 2017) with a C-4 backbone on the COCO 2017 train split is used for training and the results are reported on the val split. A learning rate of 0.03 is used and the other parameters are kept the same in the 1x schedule in detectron2 (Wu al. 2019).
22

---

Under review conference paper ICLR 2023
Table E.1: Ablation study on different losses the training objective VoLTA for multi-modal coarse-
grained downstream tasks, Each model is pre-trained on 231k samples from COCO2017 and VG.
VoLTA
VQAv2
NLVR
F30k IRTR
LGOT
CBT
AMLM
LITM
dev
std
dev
test-P
IR&1
TRA1
Lw
Law
71.5
72.2
72.9
70.5
81.5
/
72.3
74.2
75.5
71.0
82.6
/
73.3
75.2
77.0
72.4
82.8
74.6
74.6
76.7
78.1
72.7
83.6
D.3.2 COARSE- GRAINED MULTI-MODAE DOWNSTREAM TASKS
Vision-Language Classification (VQAv2 and NLVR*): Vision- Vision-Language Classification task en-
compasses VQAv2 and N whose hyper-parameter setup has been taken from METER (Dou ct al. (022b) and FIBER (Dou et al.) 2022ah. Model finetuning is done with peak learning rates of 2e-5 for the backbones, lc-4 for the cross- moosl parameters, and 1e-3 for the head layer for 10 epochs with a batch size of 512. The image resolutions are set to 576 for VQAv2 and 384 for NLVR2 and the models are evaluated with the VQA-Scores for VQAv2 and accuracy for NLVR (Table C.I
Image-Text Retrieval (IRTR): We follow Dou et al. (2022a) for IR-TR setup for the Flickr30k dataset where the cross-attention layers in the backbones are removed during IR-TR fine-tuning and evaluation. The peak learning rates are set to 2c-5 for the backbones and le-4 for the head layer, a batch size of 1024 considered and each image resolution set to 576, We evaluate on the Recall @ metric for both text and image retrieval tasks as outlined in the Table C.
Image Captioning: For image captioning, only the image-to-ter text attentions are kept for the cross- modality attention fusion and the model is converted into standard seq2seq model (Dou al., 2022a),. We used causal mask in the decoding side and the outputs are predicted auto-regresyively (Duu et al. 2022a).. Models are trained with the cross-entropy loss for 5 epochs with the peak learning rates of 5e-5 for the backbones, and 2.5e-4 for the of the parameters followed by two-stage finetuning. the first stage, finetuning with GOLD (Pang & He| (02T)) is done for epochs with a peak learning rate of le-5 for the backbones. since it is efficient and has been proven to be effective when the model input can correspond to different outputs. Second stage finetuning involves CIDEr optimization where the learning rate is further reduced to le-6 and the model is trained for epochs. A batch size of 512 is considered in both these cases and a beam size of 5 is used during inference. Evaluation metrics include BLEU (Papineni et al. 2002), J, METEOR (Banerjee & Lavie 2000), CIDEr (Vedantam et al. 2013), and SPICE (Anderson et al. 2016) scores as shown in Table
D.3.3 FINE-GRAINED MUDTI-MODAL DOWNSTREAM TASKS
We follow Dou et 2022a) for training and evaluation on different datasets (RefCOCO, RefCOCO+, and RefCOCO) where the models are finetuned with a batch size of 16 for 20 epochs. A warmup of 2000 steps with peak LR of le-5 for both the OD head well as the rest of the model's parameters are used. LR drops twice, once 67% and the other at 89% of the total number of steps. Horizontal flip augmentation has been turned off during REC training because it was observed in Dou et al. (2022a) that horizontal tlip adversely affected the performance, particularly on the RefCOCO dataset. Accuracy is used the evaluation metric in this case (Table C.I).
Referring Expression Comprehension (REC):
We follow the training and evaluation setup of Dou et al. (022a) for text- conditioned (multi-modal) object detection. For both COCO and LVIS datasets, model has been finetuned for 24 epochs with batch size 32, a LR of le-5, and two learning rate drops, once at 67% and the other at 89% of the total number of steps. AP scores are used in this case for model evaluation (Table C.1.
Object Detection:
E ABLATION
We have conducted ablation studies, particularly, on pre-training objectives, GOT loss weight and the dimension of projectors which are summarized below:
23

---

Under review conference paper ICLR 2023
Table E.2: Ablation study on different losses the training objective of VoLTA referring expression comprehension tasks. Each model is pre-trained on 231k samples from COCO2017 and VG.
VoLTA
ReiCoCo
RefCOCO +
RefCOCOg
LGOT
LBT
DMIM
LITM
testA
testB
testA
testB
val
test
Lu
Ligu
/
81.7
84.1
77.8
71.2
76.6
62.2
71.7
71.7
/
82.7
85.2
78.1
72.0
77.7
62.5
72.8
72.7
/
/
/
83.9
86.6
80.5
73.9
79.5
64.1
74.6
74.3
/
86.1
88.6
81.8
77.0
82.7
67.8
78.3
78.3
Table E.3: Ablation study on Intra- and Inter-modal Barlow Twins objective for multi-label image
classification on VOCO7 and COCO. We report classification mAP VOCO7, and per-class (PC) and overall (O) F1 scores COCO, Each model is pre-trained 123k train-val samples from COCO2017.
C BT
ST
ET
L'T
VOCUT
MLP
(PC((C
COCO
90.8
12.2171.7
91.8
74.0/73.7
94.0
74.5/76.4
E. ABLATION ON PRE-TRAINING ORJECTIVE
We analyze the effectiveness of different pre-training objectives through an ablation on coarse- and fine-grained downstream tasks (Tables [._ [nd First, we pre-train VoI.A only with the multi- modal BT loss. In this setup. VoLTA only acts dual encoder; thus, the cross-attention parameters are not pre-trained. Next, we add MLM and ITM loss which helps model to learn cross-man moos- information via attention fusion. Next, we add the GOT pre-training objective. Note that GOT adopts two types of OT distances WD for node matching and GWD for edge matching. As shown in table E.2 the GWD helps to improve the performance of reference expression comprehension across RefCOCO, ReSCOCO+, and ReiCOCOg datasets. Specifically on RefCOCOg, adding Lgww yields a significant 4.0% boost in the challenging test set. Since this dataset contains intricate images with multiple similar objects with different shapes and colors, GWD is crucial in distinguishing between them. Overall, this set of experiments demonstrates that all of the objectives are necessary for our model to obtain good performance different coarse- and fine-grained multi-modal tasks.
E.2 ABLATION ON INTRA- AND INTER-MODAL BARLOW TWINS LOss
We verify the effectiveness of the multi modal Barlow Twins (BT) objective by ablating the intra- and moter-modal terms. The first row of Table B is identical to the original image-only BT objective, Next, we introduce the text branch and add the same BT objective between the two views of the caption, Afterward, we add the inter-modal BT objectives. As shown Table E.3 each loss term improved the image classification performance, demonstrating the importance of both intra- and inter-modal objectives.
E.3
ABLATION ON GOT LOSS WEIGHT
our loss formulation, we introduce a GOT loss weight wGOT which regulates the alignment local features through GOT loss. By conducting grid-search on uni-modal downstream classification tasks, we assessed the impact of woor shown in the Table E.4 and experimentally found its best value to be 100 in our case. to be noted that very high value of wgor considerably degrades the performance on downstream tasks.
E.4 ABLATION ON PROJECTOR DIMENSION
The design of projector head plays a pivotal role the downstream performance of the model (Garrido ct al. 2022). To investigate the impact of hidden and feature (projector output) dimensions, we have tested 4 different configurations on uni-modal downstream classification tasks. It can be noticed from
Table E.5 that an increase in the number of parameters in the projector head does not necessarily lead
24

---

Under review conference paper ICLR 2023
Table E.4: Ablation study on the value of wgoT,, the weight of GOT loss in Liotal in the objective of VoLTA for multi-label image classification on VOCO7 and COCO. We report classification mAP VOCO7, and per-elass (PC) and overall (O) Fl scores on COCO. Each model is pre-trained on 123k train-val samples from COCO2017.
VOCO7
COCO
WGOT
MLP
MLP (PC/O)
50
93,4
74.1/76.0
100)
94.0
74.5/76.4
200
93.2
73.1/75.5
500)
93.1
72.8/75.3
Table E.S: Ablation study the dimension of local and global projector networks of VoLTA for multi-label image classification VOCO7 and COCO. We report classification mAP VOCO7. and per-class (PC) and overall (O) Fl scores COCO. Each model is pre-trained 123k train val samples from COCO2017.
VOCO7
COCO
Projector Config.
MLP
MLP
8192-8192-128
91.3
71.3773.0
3192-8192-256
91.9
72.2/73.3
2048-2048-513
93.4
73,9/76.1
2048-20 2048 200--20088--444
94.0
74.5/76.4
an increase in performance. For example, projector configuration of 8192-81922 8 times more parameters than 2048-204S-1024 although, the latter performs better in downstream tasks (Table|E.5) indicating that the output (feature) dimension projector plays a crucial role final performance of the model.
119-2599 has roughly
RESULTS ON COARSE-GRAINES VISION LANGUAGE TASKS: COMPARISON WITH METHODS USING MORE PRE-TRAINING DATA
Table presents comparison of VoLTA on multimodal coarse-grained tasks with state-of-the-ae methods pre-trained using magnitude more data. On VQA, VoI.TA beat ViLBERT, UNITER-B, VILLA-B, UNIMO-B and ViLT-B, each pre-trained on 3 4M datasets. Note that VoLTA is trained only on COCO and VG, whereas other methods use combination of COCO, VG, CC, and SBU datasets. Such strong performance proves the generalizability of VoLTA. On captioning, VoLTA beats Unified VLP, OSCAR, UFO-B, VITCAP, VinVL-B, METER-CLIN and XGPT. However, for IRTR and NLVR, VoLTA can not yield better performance over these baselines. assume that the large domain difference between pre-training and downstream datasets is the reason behind the limited performance IRTR and NLVR.
G QUALITATIVE RESULTS
Visual Question Answering and Visual Reasoning: Visual question answering (VQA) a widely recognized multi-modal task which infers an answer in response to a text-based question about an im- age. In Figure G.1 we demonstrated several example inage-question pairs along with corresponding answers predicted by VoLTA on VQAv2 validation set. The primary aim the visual reasoning task is to ascertain the veracity of a natural language statement against an associated image pair. Figure G.2 displays examples of responses (True/False) predicted by VoLTA NLVR? validation set.
Lannguage-ondditooned Object Detection: Object detection forms indispensable constituent of several multi- modal understanding systems. However, conventional object detection pipeline, being employed as black-box tool, predicts all possible objects in the image. On the other hand, for better apprehension of combinations of these objects in free-form texts, language-conditioned object detection task is considered (Kamath et al. [202T] Dou et al.) (022a). We use pre-trained VoLTA for fine-tuning and evaluation on COCO and LVIS datasets for text- conditioned object detection task. As illustrated in the Figure G.3. VoLTA predicts bounding boxes relevant the text prompts (captions)
25

---

Under review conference paper ICLR 2023
Table F.1: Multi-modal coarse-grained downstreams: visual question answering, visual reasoning, retrieval captioning. Methods pre-training with significantly larger datasct are colored gray, For captioning. 4 metrics are reported B@4: BLEU@4, M: METEOR, C: CIDEr, S: SPICE. Best comparable results are bold.
VoL-TA- denotes Swin-R B backhone.
Method
#Pre-train
Images
dev
VQA NLVK
sld
dev
test-P
IR81
IRTR
TRa1
Method
#Pre-train
Images
BT1
M
Caplioning
Munels pre-truiard
COCO (133))
Models
SCAN
67.4
VL-T5
15UK
34.5
116.3
SCG
108k
49.3
7188
VL-BART
180%
35.1
28.7
116.5
21.5
PTAN
108k
30.4
70.0
Unitied VLP
3M
36.5
28.4
117.7
21.3
OSCAR
VisuaBBKKT
123k
70.8
71.0
67.4
67.0
URO-R
4M
36.0
28.9
122.8
22.2
LXMERT
231k
72.4
72.5
74.9
74.5
VITCAP
4M
36.3
29.3
125.2
22.6
SOHO
231k
73.4
86.5
METER-CLIPP
4M
38.8
128.2
Minletr pre-tnaiend COCCO
3RV
(TM)
CINDOO (OMR)
VioVE-8
38.2
VILBERI
3M
70..
70.9
55.2
XGPT
3. 1.1M
37.2
28.6
120.1
21.8
UNITER-B B
4M
72.7
72.9
77.2
77.9
72.5
85.9
FIBER B
IM
39.1
30.4
128.4
23.1
VILLA-D
UNIMO-B
4M
73.8
74.0
BOLD--GOLDEE
231K
38.9
30.5
128.5
23,4
VILTER
4M
71.3
75.7
76.1
64.4
83.5
Model Suned wiGh C CID) qplonizaiion
ALBEF-B
VLMo-B
4M
76.5
76.9
$2.8
79.3
92.3
ViaVL-B
1M
40.9
30.9
140.4
25.1
METER Swin
4M
76.4
76.4
$2.2
833.5
79.0
92.4
FIDER B
4M
42.8
31.0
142.8
24.3
FIBER-B B
78.4
84.6
85.5
81.4
92.9
FIBER GOLD
4M
43.4
144.4
24.6
VoLTA-B
2311
74.6
74.6
76.7
78.
83.6
NoLTA-GOLD-E
2918
40.2
30,9
137.5
23.7
and labels them with the corresponding spans from the text. For example, the top-middle image has 4 objects, however our model predicts boxes only for person and cup based on text prompt.
Referring Expression Comprehension (REC): The objective of REC is to align the entire referring expression (text) with the corresponding box by disambiguating among the several occurrences of an object belonging the same category and therefore, one box per expression is be predicted. For example, bottom-left image in Figure depietss VoLTAS box prediction for the corresponding referring expression: the slice of cuke on the left.
26

---

Under review conference paper ICLR 2023
Examples from VQAv2 validation set
What moving the proto? train
Q: te the mug half ful? A: yos
O: What about do? nit ball
What is the desk made out of? wood
G: Are people on me batcony? A: yes
Q: mis natural environment? A:
holding
What
Figure G. Examples on Visual Question Answering from VQAv2 validation dataset. display variety examples (e.g., number of items, color objects, type objects, events actions) with respective answers predicted by VoLTA
Exampies from NLVR* validation
OW
achrer cootie
There boba dunbowit.
Dos
the right.
Fiie
Falte
The right Unage COCONNA uuiuity colerhb parrett
Time
A Aprrson la heising cup the right Iuage
Tiae
Figure G.2: Examples Visual Reasoning from NLVR2 validation dataset. statement (text prompt), 2 images are shown alongside and VoLTA predicts whether given statement True (green box) False (red box).
27

---

Under review conference paper ICLR 2023
tast-condiloned cotection from coco velidation set
parson eup
handbag.
Figure G.3: Examples of Object Detection from COCO validation dataset with various text prompts. Our model predicts boxes relevant the text (caption) and labels them with the corresponding spans.
Examples from Bafco.c validation oet
contatter Exarples From Refcoco + volidation set
eaptent
red kpdop
group
podee top
Figure G.4: Examples of Referring Expression Comprehension from RefCOCO (top), ReSCOCO+ (middle)
and ReSCOCOg (bottom) validation datasets. The expressions in RefCOCOp typically have florid and longer constructions compared RefCOCO and RefCOCO+ The model has the entire and disambiguate amongst different objects in the image.
28

---

Under review conference paper ICLR 2023
Caption: polar holding
Caption: shown staring
Caption: motorcycle parsed
Caption: bananas lyying
Captien: parson
snowy
Caption: Two male elephants standing tole.
Caption: Three lugaage bags en the ground near one another
Caption: Two pretty birds are shtting on branch.
Caption: piece of pie with piace Cut DUT.
Caption: Two zebras standing a field yellow grass.
Caption: group of colorful umbrellas flyng in #ky sbove a ctt.
Caption: Severa cows grazing
tel.
Caption: coughnut pate
Caption: siver hydrant
Caption: box teed an umbrella flceding In the ar.
Caption: An apple and long sendwrich out In sit on cutting board.
caption: A man Butting King an too of chocolate cookies.
caption: A hand hoddng a mirocure ricced doughnut with nonpanatss.
Caption: man standing next baby elephant on top rieer
Caption: A chocolate doughnut and phone sit on this table
Figure G.5: This figure shows how different words in captions attend relevant image regions, produced by the GOT module of VoLTA pre-trained on COCO. Extension of Figure [ All image-caption pairs are taken from the COC02017 train split.


Reviews:
[{"review_id": "lBsjMslpT1u", "reviewer": "Reviewer_mig1", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors propose VoLTA, a unified Vision Language Pre-training paradigm for image-level and region-level applications, in which only image-caption pairs are used. They use graph optimal transport for weakly-supervised feature-level and region-level alignment. They conduct a wide range of vision- and vision-language downstream experiments to demonstrate the effectiveness of VoLTA.", "summary_of_the_review": "This paper proposes a new VLP framework using GOT, but the evaluation is in-sufficient and the effectiveness of the proposed module is not clear.", "strength_and_weaknesses": "Strength:\n1.\tMany experiments on downstream tasks are conducted.\n2.\tPaper writing and structure are easy to understand.\n3.\tThe idea two views of image and text in GOT part is interesting.\n\nWeakness:\n1.\tThe figures are not clear. For example, in figure 2, it’s confused for the relation of 3 sub-figures. Some modules are not labeled in figure, such as CMAF, L_BT, VoLTA.\n2.\tThe experiments results are not significant.\n3.\tThree steps for training are shown in VoLTA, a) switching off CMAF, b) switching on CMAF, c) keep CMAF and random sampling for training. The ablation study on these parts should be conducted.\n4.\tThe key point of this paper is GOT, but no ablation on this part. The authors are encouraged to verify which parts works.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "BM5_kGn0wVl", "reviewer": "Reviewer_RqXJ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new Vision-language pre-training (VLP) paradigm that only utilizes image-caption data but achieves fine-grained region-level image understanding, eliminating the use of expensive box annotations. Existing end-to-end  VLP methods use high-resolution image-text-box data to perform well on finegrained region-level tasks, such as object detection, segmentation, and referring expression comprehension. However, it comes at high annotation cost. As a solution to this problem, the authors proposed VoLTA which uses adopts graph optimal transport-based weakly-supervised alignment on local image patches and text tokens to germinate an explicit, self-normalized, and interpretable low-level matching criterion. As a result, VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre-training and removes fusion-specific transformer layers, further reducing memory requirements.  Extensive experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fine-grained applications without compromising the coarse-grained downstream performance. ", "summary_of_the_review": "Although i think the whole idea is interesting and worthy of discussion in the research community, i am a bit concerned at the use-case of this pre-training as compared to let say off-the-shelf GLIPv2. However, I agree with the authors that such pre-training is indeed necessary in cases where annotations are not available and can provide some useful insights in the near-future. Regarding the draft, the reviewer is slightly confused in how the fusion module is trained and what is the imapct of the textual augmentations in Barlow Twins. Some minor issues like grammatical errors and missing ablations also need some attention. Therefore, at this stage, I will vote for marginal accept and re-evaluate based on authors and other reviewer response.", "strength_and_weaknesses": "Strength -\n\n(+) The concepts and higher level overview of the problem is somewhat well defined. \n\n(+) The idea of using Barlow-Twins for both the Image and Text modality and thereby performing inter-modal and intra-modal redundancy reduction is well appreciated and interesting. \n\n(+) The idea of using Graph Optimal Transport instead of Optimal Transport for patch-word alignment seems intuitive and novel for this vision-language domain. \n\n(+) Extensive experiments have been performed on multiple standard benchmarks often surpassing supervised approaches.\n\nWeakness - \n\n(-) The reviewer is very curious about the textual augmentations. The amount of augmentations done on the textual caption looks like a strong data augmentation and in such cases Barlow-Twins may not perform well. Can the authors shed some light on this issue ? This might affect the overall model performance while doing the intra-modal alignment. \n\n(-) The cross-modal attention fusion block part appeared a bit unclear to this reviewer. How is this CMAF module trained ? When alpha=0, only BT and GOT losses are computed , however when non zero then MLM and ITM is computed, does that mean , this is trained in a 2-stage fashion ? i.e first the BT and GOT loss is computed to get a good initilaization point and then CMAF is activated to do the fusion ? This needs some clarification. \n\n(-) This reviewer is also curious to know how is the inference on dense downstream task like object detection and instance segmentation performed ? After VolTa pre-training, is it fine-tuned on the target dataset or the decoders are attached to the CMAF only for finetuning ? This needs some clarification. \n\n(-) What is the purpose of such pre-training in general ? Since the method performs very well on uni-modal downstream tasks but lags behind in multi-modal downstream tasks, it is hard to understand the motivation for such pre-training. The reviewer agrees that annotations are costly, and such weak supervision provides an alternate solution, but only surpassing uni-modal downstream tasks using multi-modal pre-training looks to this reviewer a slightly weaker claim. \n\n(-) Why is the gating based attention is better than traditional cross-attention transformer. Such ablations should be done to justify the design choice which is missing in current form. \n\n(-) The paper has minor grammatical errors which should be fixed for better readibility", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "-3iiEXx9cMp", "reviewer": "Reviewer_PoWP", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a new vision-language pre-training (VLP) method for local feature alignment. The key idea is to use a graph optimal transport mechanism to align image patches and text tokens. This paper also extends BT to VLP by introducing an intra-modal objective and an inter-modal objective. Finally, a gated cross-attention is designed to eliminate the fusion encoder that is widely used in previous work.\n", "summary_of_the_review": "Although this paper extends existing methods to the VLP field, the motivation is not clear. Furthermore, the advantages of the proposed modules are not investigated or empirically proved. The technical novelty/contribution and performance improvement is quite marginal.\n", "strength_and_weaknesses": "Strength:\n1. Using graph optimal transport (GOT) in multimodal problems is interesting, although the advantage of GOT vs other alignment methods is not investigated (see more details in weakness).\n2. The paper is well-organized and easy to follow.\n3. Thanks for providing the code which can help readers to reproduce the results.\n\nWeakness:\n1. This paper extends BT to VLP problems given the fact that BT does not require large batch size. What if contrastive-based methods use the same batch size as BT, will it degrade the performance significantly? If not, then we can simply use small batch sizes.\n2. How was GOT applied to image features and text features? This is not clearly discussed in the Method section.\n3. On page 5: WD can only match nodes in the graph, and will treat all “men” entities as identical and will ignore neighbouring relations like “in blue shirt” and “holding the scissors”. Is there any evidence to support this statement?\n4. Intra-modal objective and inter-modal objective share the similar idea of TCL [1], except that TCL is built upon contrastive losses. Furthermore, TCL is a very important work in VLP.\n5. In Equation 2 and 3, what is $k’$ in $C_{i,i}^{k’}$?\n6. What does “weakly-supervised” represent in this paper?\n7. On page 7: \t“For MLM, we randomly mask 15% text tokens”. Have you replaced the masked tokes with other special tokens?\n8. What’s the advantage of using GOT compared with existing local alignment methods such as OT?\n9. The performance improvement is marginal by using the same backbones.  \n\n[1] Vision-Language Pre-Training with Triple Contrastive Learning, CVPR 2022\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject