VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Authors: Shraman Pramanick
Li Jing
Sayan Nag
Jiachen Zhu
Hardik J Shah
Yann LeCun
Rama Chellappa
Venue: Accepted by TMLR ()

Abstract: Vision-language pre-training (VLP) has recently proven highly effective for various uni- and multi-modal downstream applications. However, most existing end-to-end VLP methods use high-resolution image-text-box data to perform well on fine-grained region-level tasks, such as object detection, segmentation, and referring expression comprehension. Unfortunately, such high-resolution images with accurate bounding box annotations are expensive to collect and use for supervision at scale. In this work, we propose VoLTA (Vision Language Transformer with weakly-supervised local-feature Alignment), a new VLP paradigm that only utilizes image-caption data but achieves fine-grained region-level image understanding, eliminating the need for expensive box annotations. VoLTA adopts graph optimal transport-based weakly-supervised alignment on local image patches and text tokens to germinate an explicit, self-normalized, and interpretable low-level matching criterion. In addition, VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre training and removes fusion-specific transformer layers, further reducing memory requirements. Extensive experiments on a wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fine-grained applications without compromising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations.

Published in Transactions Machine Learning Research (09/2023)
VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment
Shraman Pramanick Li Jing"? Sayan Nag
Jiachen Zhu Hardik Shah?
Yann LeCun Rama Chellappa'
'Johns Hopkins University "Meta Univerritty of Toronto "New York University
Reviewed on O
Abstract
Vision-language pre-training (VLP) has recently proven highly effective for various uni- and multi-modal downstream applications. However, most existing end-to-end VLP methods use reghbresolution amago-text-box data to perform well on fno-grained region-level tasks, such as object detection, segmentation, and referring expression comprehension. Unfortamately, such rigl-resolttion images with accurate bounding box annotations are expensive to col- lect and use for supervision al scale. In this work, we propose VoLTA Transformer with suakiy-supervised local-feature Alignment), a new VLP paradigm that only utilizes image-caption data but achieves fine-grained region-level image understand- ing, eliminating the need for expensive box annotations. VoLTA adopts graph optimal transport-based weakly-superrvsee alignment on local image patches and text tokens to germinate an explicit, nelf-normalize, and interpretable low-level matching criterion. In addition, VoLTA pushes multi-modal fusion deep into the uni-modal backbones during pre- training and removes fusion-specific transformer layers, further reducing memory require- ments. Extensive experiments on wide range of vision- and vision-language downstream tasks demonstrate the effectiveness of VoLTA on fino-grained applications without com- promising the coarse-grained downstream performance, often outperforming methods using significantly more caption and box annotations. Code and pre-trained model are available at https:/ htttss:/githubb.ooo githtb..ttmmmm/. SShaaaanraamannn VaVna.ick/VoLT. VoLTA.
1 Introduction
Inspired by the escalating unification of transformer- based modeling in vision (Dosovitskly et al., 2021; Liu et al., 2021; Chen et al., 2021a) and language (Devlin et al., 2019; Liu et al., 2019) domains, coupled with readily available large-scale image-caption pair data, vision-language pre-training (VLP) (Lu et al., 2019; Li et al., 2020a; Kim et al., 2021; Kamath et al,, 2021; Zhang et al., 2021) has recently been receiving increasing attention. VLP has not only been proven the de-facto for several vision-language lasks, but it has also been beneficial for traditional vision-only tasks, such as image classification and object detection. Such wide-range applications of VLP can broadly be categorized into two groups: (i) tasks requiring image-level understand- ing, c.g., image classification, image & text retrieval (Plummer et al., 2015), image captioning (Zhou et al., 2020), visual question answering (Antol et al., 2015), and (fi) tasks requiring region-level understanding, e.g., object detection, instance segmentation, and referring expression comprehension (Kazemzadeh et al., 2014; Yu et al., 2016). Most existing VLP methods address only one group of application, leaving the question of generalizable and unified VL framework under-explored.
Traditional VLP methods with image-level understanding (Li et al., 2021a; Wang et al., 2021b; Dou et al., 2022b) utilize large-scale Emage-caption pair datasets and are commonly trained with image-text contrastive
Equal technical contribution. Part of this work was done during an internship Meta.

---

Published in Transactions Machine Learning Research (09/2023)
Existing systems perform separate pre-training coarse- and fine-grained downstream
Image-Text
VQA
Image Captioning
VOL TA performs coarse- and fine-grained
Coarse-grained
downstream with a unified pre-training setup
What he
Giraftc mino grazing flelc
Image Text
baseball player ewings hit ball
Coarse-grained Downstream
Unified VoLTA
Image-Iux--lest
Object Detection
Refornnng Expression Comprchension
A beseball player Wwings a ball
Fine-grained
VLP
beseball player swings
Coffee cups
cup on
Fine-grained Downstream
Figure! Different Categories of VLP frameworks. Existing VLP systems perform separate pre- training for image-level (Li et al., 2021a; Wang et al., 2021b; Dou et al., 2022b) and region-level (Kamath et al., 2021; Li et al., 2022c; Zhang et al., 2022) understanding and do not generalize well to coarse- and fine- grained downstream tasks. In contrast, our proposed method, VoLTA, unifics both downstream with a single pre-training setup and attains fine-grained region-level understanding without using expensive bounding box annotations during pro-training.
objectives computed on global Teatures. Hence, it is not trivial to extend such methods to region-level applications. On the other hand, VLP methods with region-level understanding (Kamath et al., 2021; Li al., 2022c; Zhang al., 2022) use image-tezt-baz grounding data and are designed predict bounding boxes during pro-training. Consequently, they do not support image-level tasks. Furthermore, accurate bounding box annotations require high-resolution input images, which are often expensive to collect, annotate and use for pre-training at scale. Recently, FIBER (Dou et al., 2022a) addressed the problem of such unified VLP and proposed a two-stage pre-training a algorithm requiring fewer box annotations than previous region- level pro-training methods. Moving a step forward, as shown in Figure 1, we aim to eliminate the use of costly box annotations and ask the challenging but natural question: Can ottain region-level understanding from global image-caption annolations and unify image- and region-level tasks in a single VL framework?
Subsequently, we focus on achieving region-level fino-grained understanding by weakly-supervieet alignment of image patches and text tokens. Previous VLP methods (Chen et al., 2020d; Kim et al., 2021) in this direction use Wasserstein distance (WD) (Peyré al., 2019), a.k.a Earth Mover's distance (EMD)- optimal transport (OT) algorithms for such alignment problems. However, we argue that WD is not optimum for images with multiple similar entities. Thus, we propose to jointly utilize Gromov- Wassov-tsssrnniin distance (GWD) (Peyré et al., 2016) and Wasserstein distance (WD) in setup known as graph optimal transport (Chen et al., 2020a). Morcover, instead of using a commonly deployed contrastive objective, we propose to use redundancy reduction from Barlow Twins (Zbontar et al., 2021), which less data-intensive and does not require hard-negative mining. We also follow Dou et al. (2022a) and incorporate deep multi-modal fusion into the uni-modiee modal backbones, removing the need for costly fusion-specific transformer layers. These steps when integrated yield VoLTA, Vision-Language Transformer with weakly-supervised local-feature Alignment, a unified VLP paradigm only utilizes émage-coption annotations but achieves fine-grained region-level image understanding, climinating the need for expensive box annotations. Figure 4 visualizes the foaturo-level image-tex text alignment generated by VoLTA, which can attend text tokens to the corresponding visual patches without relying on low-level supervision.
In summarry our contributions are three-fold. (i) We propose to use graph optimal transport for weakly- supervised feature-level patch-token alignment in VLP. (it) We introduce VoLTA, unified VLP paradigm for image-level and region-level applications, but pre-trained only using image-caption pairs. VoLTA memory, compute, and time-efficient and can easily be scaled up with readily available large-scale image-caption data harvested from the web. (iii) We present the results of a wide range of vision- and vision-language coarse- and fino-grained downstream experiments to demonstrate the offectiveness of VoLTA compared to strong baselines pre-trained with significantly more caption and box annotations.

---

Published in Transactions on Machine Learning Research (09/2023)
2 Related Works
Uni-modal Self-supervised Pre-training: In recent years, the machine learning community has observed a boom in self-supervised pre-training. In the language domain, representations learned by BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019) have become the default setting for many downstream tasks, Gencrative models such as GPT (Radford et al., 2019; Brown et al., 2020) have also achieved impressive few-shot/zeree /zero-shot performances on novel applications. SimCSE (Gao et al., 2021) uses contrastive learning to help learn uscful sentence representations.
In the vision domain, several snuuraatiie/jint--mmeedee 2020b; Grill et. al., 2020; Chen & He, 2021; Caron et al., 2021; Zbontar et al., 2021; Bardes et al., 2022; Shah et al., 2022; Assran et al., 2022) have outperformed supervised counterparts. Recently, generative models such BEiT (Bao al., 2021) and MAE (He al., 2022) have also achieved impressive performances with much more scalable potential.
methods (He et al., 2020; Chen et al., 2020c; 2021b;
Vision-Language Pre-training (VLP): Vision-language pre-training mainly relies on image-tert pair datasets to learn joint visual-language representations. One line of work is to train separate vision and language encoders and only fuse in the representation space. CLIIP (Radford et al., 2021). UniCI. (Yang et al., 2022a), and ALIGN (Jia et al., 2021) use the image-text contrastive loss to learn aligned representations. SLIP (Mu et al., 2021) combines self-supervised visual representation learning and contrastive mulli-modal learning. M3AE (Geng et al., 2022), FLAVA (Singh al., 2022) combines masked image modcling and masked language modeling. Another line of work uses cross attention to fuse vision and langnage information in the carly stage (Kamath et al., 2021; Dou et al., 2022b; Lu et al., 2019; Li ct al., 2020b; Kiela et al., 2019; Kim al., 2021; Zhang et al., 2021; Li et al., 2022b; Wang al., 2022c; Pramanick et al., 2023; Park & Han, 2023; Li el al., 2023a; Jang et al., 2023; Wang et al., 2023a). These works focus on learning semantic- level aligned Dision-languae 2022b), GLIP (Li et al., 2022c), and FIBER (Dou al., 2022a) use expensive grounding tmage-lest-bor annotations to learn the fine-grained aligned representations. Our work uses representation space a alignment and cross-attcntion fusion, but we do not use any box annotation learn robust fcaturo-level aliguments.
representations. In addition, UniTAB (Yang et al., 2022c), OFA (Wang et al.,
Unsupervised Representation Alignment: Unsupervised multi-modal alignment typically relies on spe- cific metrics. Wasserstein distance (Peyré et al., 2019), a.k.a EMD-based optimal transport (OT) algorithms have been widely adopled to various domain alignment lasks, including sequence-to-ssqqueee learning (Chen et al., 2019), few-shot learning (Zhang et al., 2020), knowlodge distillation (Balaji et al., 2019), unsupervised domain adaptation (Balaji et al., 2019), generative networks (Han al., 2015; Genevay et al., 2018; Mroueh et al., 2018; 2019), and multi-modal learning (Yuan et al., 2020; Chen et al., 2020d; Kim et al,, 2021; Li et al., 2022d; Pramanick et al., 2022). Previous VLP methods (Chen et al., 2020d; Kim et al., 2021), which use OT-baaed porch-word alignment, only utilize the Wasserstein distance. However, we argue that jointly modeling GWD (Peyré et al., 2016) and WD results in a superior multi-modaa alignment for intricate images. To the best of our knowledge, this the first work to apply WD and GWD-bbasd optimal transport for feature-level alignment in VLP.
Proposed System - VoLTA
In this section, we present our proposed approach, VoLTA, which contains three broad modules (i) intra- and inter-modality redundancy reduction, (ii) weekly-supervised cross-modal aligument of local features, and (iit cross-modal attention fusion (CMAF). Next, we introduce the fine-tuning strategies for various uni- and multi-modal downstream tasks supported by VoLTA. An overview of the different modules of VoLTA is presented in Figure 2.
3.1 Intra- & Inter-modality Redundancy Reduction
We use Barlow Twins (BT) (Zbontar et al., 2021), non-contrastive covariance regularization as the foun- dational objective of VoLTA. The recent success of contrastive vision-language pre-training (Radford et al., 2021; Li et al., 2021 by Jia et al., 2021; Kim al., 2021; 1 Yang al., 2022a; Dou et al., 2022a;b) has already

---

Published in Transactions on Machine Learning Research (09/2023)
Barlow Twins
Graph Optimal Transport
Masked Language Modeling
Image Text Matching 4
Eoor
Proj
MLM Head
JTM Head
Patm
Pest Pntt
Layeee
(4 intured Layers
H+l
Hel
(M--00) Untused Layars
(igot Unfured Layaer
(4-44) Inteed Layvrr
H+l
(1 Lift and Layers
Mi infused
Uni
Layers
Laywer
Notwcholl is tene of souns 2000
buarh young bays are baskertal.
Aguus young cRiGE
poler bears snimming in wwalar
Masbed Cuption
Wronn Caption
Augmented Imagss
Augmented Textta
grop toys pirrig
Figure 2: Computation of four different objectives, LBT, LGOT, LMLM, and 4YTM by the proposed VoLTA framework. Inspired by Dou et al. (2022a), VoLTA inserts mross-modal attention fusion (CMAF) inside uni-modal backbones with a gating mechanism. During VoLTA pro-training, every forward itcration consists of three steps (i) CMAF is switched off, VoLTA acts as dual encoder, LaT and Loor are computed. (ii) CMAF switched on, VoLTA acts as fusion encoder, and image-masked caption pair fed into the model to compute LMLM. (ist) CMAF is kept on, randomly sampled image-caption pair is fed into the model to compute LITM- Such a fusion strategy results in a lightweight and flexible model compared to using fusson--pecifiic transformer layers.
shown that, compared single modality, image-caption pairs offer a significantly higher-lovel of abstractive and semantic concepts about the training samples. However, common contrastive VLP objectives, like In- foNCE (Oord et al., 2018), are data-hungry, as they require large batch sizes and well-mined hard negatives. On the other hand, the BT objective operates on the dimensions of the embeddings across the two views of training samples. Hence, it is more robust. to batch size and can be trained using lower memory resources. In this work, we extend the BT objective for a multi-modal setup.
The original BT algorithm, which operates joint embeddings of distorted samples, was proposed only for image modality. Specifically, for cach image of a batch X, two distorted views are obtained using a distribution of data augmentation T with disparate probabilities. These distorted images are then fed into a shared image encoder containing a feature extraction network (e.g., ResNet (He el al., 2016)) cascaded with trainable linear projection layers, producing a batch of parallel embeddings 2d and 2 The BT loss computed using the encoded embeddings can be denoted
B EB B.
LBT A
C1-Ca) + ΣΣΣ (Ci)) 2 where, Cij jit
(1)
VE, (3
Es (32))²
À is positive weighting factor; C' is the cross-correlation matrix computed between 2d and 2B along the batch dimension; b stands for sample indices in a batch; i,j refers to the dimension indices of 2 and 2 The first term Equation 1 is the invariance term which attempls to equate the diagonal elements of the cross-correlation C' matrix to 1, whereas the second term is the redundoncy reduction term which pushes the olf-diagonal elements of C' matrix to O.
In this work, we use BT for inage-caption pairs. Specifically, use stochastic data augmentations for both images and text', and directly apply the BT objective for all the 2x2 pairs, resulting in additional supervision. Note this simple, straightforward, and instinctive extension enables us apply rodundancy reduction between and across modalities, which intuitively results in superior visual representation. Moreover, in this
Augmentattion details are provided in Appendix D.1.

---

Published in Transactions on Machine Learning Research (09/2023)
bi-modal setting, we can pre-train text encoder parallel with the image encoder and, thus, can generalize our system to broader range of uni- and multi-modal downstream applications.
Intra-modal Objective: Intra-modal objective refers to applying the BT loss in-between pairs of image and text embeddings. Given an smage-caption pair, first have two augmented views (I, (1) for each image, and two augmented views (T.T) for each text. Then, we resort to Equation 1 individually for the image and text pairs.
cbr - C (Ct) j$H
+A
(II (I', (T')
Inter-modal Objective: Inter-modal objective refers applying the BT loss across image and lext embed- dings. Since the image and text encoders can output features with different shapes, we design the projector layers with same output dimension. Hence, addition to the original BT loss between (I, (I,1") Zbontar et al. (2021), get three more loss terms (T,T', (I,T'), (I', T), leading to 3x diverse and high-quality additional supervision. The inter-modal B3 losses can be directly computed following Equation 1.
cb) (1-0
+ + EE (c))
Pykk ITT)
The resulting bi-modal BT loss is LBT = Ex Cer &
3.2 Alignment of Local Features
Image Patch Features
Though the inter-modal redundancy reduction pro- vides high-quality semantic supervision, is com- puted on the global image- and text features and, thus, only simulates implicit and non-interpretable multi-modal aligument. However, fine-graine groigga region-level downstream applications like detection, segmentation, and reference expression comprehen- sion require local visual feature descriptors with specific spatial information. To achieve this, most existing top-performing VLP methods, including UnITAB (Yang et al., 2022c), OFA (Wang et al, 2022b), GLIP (Li et al., 2022c), and FIBER (Dou et al., 2022a), use high-resolution image-text taage-text-bor box data for fine-grained pre-training, However, bound- ing box annotations are expensive to collect and use for supervision. Hence, we seek an alternate weekly- supervised solution for local feature-level alignment using global image-caption annotations.
G.(V..
Wassersten Distanca (WD)
PPP
Layora
Layora
CONC AUE
Saltsrm
Seltvint
AM
(i
PRN
Layers
Edge Matching by Gronoy-leessenteen Distanze (COWT)
9v1V 4a)
There or oarge on the laptop
There
arangg
m
Text Taken Features
Figure 3: Illustration of the graph optimal trans- port (GOT) algorithm used for patch-word alignment in the proposed VoLTA framework. We obtain patch- and token-level features from the last layers of corresponding visual and textual trans- former encoders and use these encodod local-foature vectors to construct modelity dynamic graphs - (V,. for image patches and (,,,,, for text tokens. Next, we perform patch- align- ment by utilizing the and Wasserstein distance for edge and node match- ing, which preserves the topological graph structure. See Section 3.2 for details. Darker edges denote larger weights.
Recently, WD (Peyré et al., 2019), a.k.a EMI)-based OT algorithms have been used for weakly-superviee patch-word alignment in VLP (Chen et al., 2020d; Kim et al., 2021). Such OT-based learning meth- ods are optimized for distribution matching by min- imizing the cost of transport plan. We pose the patch-word alignment as a more structured graph- matching problem and use the graph optimal trans- port (GOT) algorithm, which utilizes GWD (Peyré et al., 2016) in conjunction with WD to ensure the preservation of topological information during cross- modal alignment. More specifically, we obtain the patch- and token-level features from the last layers of corresponding visual and textual transformer encoders, and use these encoded local-fcature vectors to con- struct -ppeeit dynamic graphs (VV,,, Ex) for image patches and Gy y, Eg) for text tokens. Fach
distance

---

Published in Transactions Machine Learning Research (09/2023)
node in these graphs E (V., Vy) is represented by corresponding feature voctors, and intermediate odges {IR,, Byy by thresholded cosine similarity.
Importance of GOT Patch-Word Alignment: As mentioned previously, GOT adopts two types OT distances WD for node matching and GWD for edge matching. In contrast, previous vision-langnage pre-training algorithms using OT for patch-word alignment. only considered WD (Chen et al., 2020d; Kim et al., 2021). However, roquire both WD and GWD for accurale, fine-grained matching. For example, in Figure 3, there are multiple "orange" present in the image. WD can only match nodes in the graph, and will treat all "orange" entitics as identical and will ignore neighboring relations like "on the laptop'. However, by using proper edge matching with GWD, we can preserve the graph's topological structure. We can correctly identify which "orange" in the image the sentence is referring to. Hence, we couple WD and GWD) mutuaaly beneficially and use a joint. transport plan for accurate patch-word matching.
argue that intricate images with multiple objects with similar shapes and colors
Once GR and Gy are computed, we follow Chen et al. (2020a) to compute WD and GWD.
Wasserstein Distance calculates the pairwise distances between two sets of cross-domain node embeddings. Consider two discrete distributions, & E P(X) and & E P(Y), where & = Σ u,d2, and e = Σ By and 6g being the Delta- Dirac function centered on 2. Since and & are both probability distributions, sum of weight vectors is 1, E, u = -, Vz. The WD distance between @ and v is defined as:
Dm(0,
min ΣΣ Ta c(
(4)
TelI(u.))
where I[(,, v) = TT E R4 R4X T1, u. 1,, = v), c(s,, is cosine distance metric, and T is the transport. plan, interpreting the amount of mass shifted from @ to 8/y.
Distance assists in edge matching and preserves graph topology by calculating distances between pairs nodes in each domain and measuring how these distances compare to the counter domain. In the same discrete graph matching setting, GWD between and vo can be mathematically represented as:
(d,
Σ ((t, Tel(..)) "..
min
(5)
where intra-graph structural similarily between two node pairs (æs xf) and (yy, U5) is represented as L(x, Vys 24,4//) 1/4) = [|c| (1,, af) - c2(n). o being cosine similarity between a node pair in any graph G.. Transport plan T't is periodically updated to align the edges in different graphs belonging to disparate modal- ities.
We further follow Chen et al. (2020a) to combine WD and GWD) transport plans, leading to a unified GOT objective given as:
Lcor(e, =
= (Dr +) + (1 7) v))
(6)
where y regulates the importance of two loss terms.
3.3 Tross-Modal Attention Fusion (CMAF)
BT and GOT losses are computed in dual encoder setting, which does not contain cross-modal interac- tions and is not suitable for complex multi-modal feature representation. Most existing methods, including UNITER (Chen et al., 2020d), ViLT (Kim et al., 2021), METER (Dou et al., 2022b). and GLIP (Li et al., 2022c) design cross-modal fusion by stacking additional transformer layers on top of uni-modal encoders, introducing large number of added parameters during pre-training. We follow a more efficient solution proposed by FIBER (Dou et al., 2022a), which insorts cross- modal fusion into the uni-modal backbones with gating mechanism. Specifically, at the top M transformer layers in the vision and language backbone,

---

Published in Transactions Machine Learning Research (09/2023)
cross-altention signals, weighted by a gating scalar a, are addod to sclf-attention:
f Self-Att(2)
x = + 2 + a Cross-Att(2, y)
(7)
FFN(x)
where a is a trainable parameter initialized to O. Following existing literature (Li et al., 2021a; Wang et al,, 2021a; Dou et al., 2022b;a), we use masked language modeling (MLM) and image-text matching (TTM) to pre-train the cross-attention parameters. For MLM, randomly mask 15% text tokens, and the loss aims to reconstruct the masked tokens. We feed the network with randomly sampled image-caption pairs for ITM, and the loss predicts whether they are matched. The gating mechanism is good choice for CMAF because (1) cross-attention parameters can easily be switched off by setting the gating scalar to 0 when computing the BT and GOT losses. Thus, we can learn the cross-altention parameters without affecting the original computational flow of uni-modal backbones. (it) gating mechanism more lightweight and memory-eficient than adding fusion-specifie layers (GLIP and METER use 4x more fusion parameters than VoLTA).
Overall, VoLTA training pipeline can be summarizod in the following three steps:
BT & GOT: CMAF is switched off (ax = 0), VoLTA acts as dual encoder, LET and LGOT are computed.
MLM & ITM: CMAF is switched on (a * 0), VoLTA now acts fusion encoder, LMLM and LrTm
losses are computed.
Back-propagation:
the four losses are added, giving Liotal LBT + woor Loot + LMLM + LITM, and propppppee into the model end-to-end. An ablation on different pre-training objectives of VoLTA and values of wgor given in Section 4.7.
The overall VoLTA pipeline for computation of different training objectives shown in Figure The pseudo-code for VoLTA is presented in Appendix A.
3.4 Finetuning For Downstream Tasks
We adopt VoLTA various vision- and vision-language downstream tasks. We switch off the inserted cross-attention modules for the vision-only tasks and use the image encoder. We utilize the learned cross- attention parameters as required for the vision-language tasks, following Dou et al. (2022a). For example, VQA and visual reasoning employ all pros-attenntin modules, whereas captioning requires only image-to- lext cross-atlention. Again, during IRTR, we switch off all cross-atlentions and use VoLTA in dual encoder setting. We keep all cross-attens attsstattnnoon parameters during multi-modal object detection and referring expression comprchension and train an object detection head from scratch using the langunge-aware image fcatures.
4 Experiments, Results, and Analysis
4.1 Pre-training & Downstream datasets
Following Chen et al. (2020d) and Huang et al. (2021), we perform pre-training by appending the VG datasct (Krishma et al., 2017) with COCO2017 (Lin et al., 2014), together consisting of 231k images. We divide our downstream tasks into three categorics (i) Uni-modal tasks such as image classification on ImageNet (Deng et al., 2009), VOCO7 (Everingharn et al., 2010), COCO; object detection on VOCO7+12 COCO, and instance segmentation on COCO. (W) Multi-modal fine-grained tasks such as region-level VI. tasks referring expression comprehension (REC) on RefCOCO, RefCOCO+, RefCOCOg (Kazemzadeh et al,, 2014; Yu et al., 2016), and language-conditiioee object detection on COCO and LVIS (Gupla et al., 2019). (Mi) Multi-modal coarse-grand coarse-grained tasks such as image-level VL tasks visual question answering on VQAv2 (Antol et al., 2015), visual reasoning NLVR2 (Suhr et al., 2019), image- and text retrieval on Flicker30k (Phumner et al., 2015) and captioning on COCO. We exclude any overlap between our pro-training and downstream validation test splits. Detailed statistics of all downstream datasets are given in Appendix C.

---

Published in Transactions on Machine Learning Research (09/2023)
Linear probing ImngeeNnt Validation Set
Lineur probing VOCUT and COGO
Method
Pre-train
Arch.
Supervision
Pro-troiniug
Top-1
Method
Pre-train
Arcb.
SVM
MLP
MLP (PC/O)
COCO
IN-IK
Labhel
76.6
Sup.
IN-IK
RN50
87.5
90.8
55.2/0008
Sup.
IN-100
RN30
Label
53.3
BYOL
IN-1K
RN50
86.0
MaC
COCO
RN30
NA
44.5
RT
DX-IK
RN50
86.2
91.9F
56.1
MoCovv2
BN30
VICRog
IN-1K
RN50
H6.6
5100/57.9
CAST
COCD
RN50
Caption
48.7
CAST T
CXCO
RN50
74.0
51.0/67.9
VirTec
COCO
RN30
Capiion
52.8
VirTex
0OCO
RN50
86.7
JOMLM
BN30
Caption
ICMLM
0OCO
BN50
87.5
MCT
COCD
RN30
Caption
64.91
Lorltx
00CO
RN50
88.4
MCT
COCD
RN30
Captioes+Teg
65.3
LocTee
C0C0=Openln
RN60
02.6
VoLTA(W/o
RN30
Caption
VoLTA(W/o MLM, ITM)
RN50
89.6
94.3
71.4/74.
VaLTAboyo MLM, ITM)
COCD
Swin-T
Caption
66.3
VoL/ MLM, (TM)
CXCO
Swin-T
88.2
93..5
73.4/78.7
VoL.A MLM, ITM)
COCO
Swin-R
Capiion
62.5
(w// MLM, ITM)
000O
Swin- B
88.5
08.9
74.1/76.
VoLTA
COCO
Swto-B
Caption
VoLTA
SwissB
89.7
95.0
74.6/76.4
Table I: Uni-modal downstream: linear image classification. We benchmark learned representations on image classification tasks by training linear classifiers on fixed features. We report top-1 accuracy on ImageNet-Ik validation set, classification mAP on VOC07, and per-class (PC) and overall (O) Fl scores on COCO. Numbers with are re-implemented by Yuan et al. (2021), and the numbers with are re- implemented by us. methods trained with significantly larger datasets are colored gray. The best results are in bold.
Pre-training Supervision
VOCU/+12 des
COCO det
COCO instanee seg
Method
Pre-train
Arch.
APac
AFm
AFT.
A
APM
APM.
A.
APM
AP
Sup.
IN-IK
RN50
Label
51.5
81.3
58.8
38.2
58.2
41.2
33.3
54.7
36.2
MoCo-v2
IN-IK
HN30
31.1
56.5
MoCo
0OCO
HN50
NA
17.5
75.1
51.1
38.5
58.5
12.0
35.0
56.6
37.5
MoCo-v2
00C0
RN50
35.8
38.8
CAST
0000
RN60
Caption
54.2
80.1
50.9
39.4
60.0
42.8
35.8
57.1
38.6
VirTex
0000
RN50
Caption
36-9
58.4
LoeTox
00CO
RN50
Capeson
53.9
80.9
30.8
40.6
80.6
44.1
35.2
57.0
37.4
MCT
COCO
RN60
Capsion
56.1
62.4
366.9
58.2
VoLTA
COCO
RN50
Capcion
56.6
84.4
62.7
41.9
61.8
44.8
36.5
68.5
40.8
1
IN-IK
Label
13.7
MoBy!
IN-1K
Swin T
NA
50.2
68.8
51.7
$3.5
66. 1
16.9
VoLTA
0OCO
Swin-T
Caption
48.8
Sup. 4
IN-1K
Swin D
Label
51.9
70.9
56.5
45.0
68.4
48.7
VtTDet
IN-IK
VII-B
45.9
CLIP
LAION-20M
VIT-B
Clactonn
45.2
40.4
SLIP
LAION-20M
VII-B
Capsion
11.0
MaskCLIP
LAION-20M
VII-B
Capsion
46.6
11.7
VoLTA
COCO
Swin-B
Capcion
45.2
68.5
Table 2: Uni-modal downstream: object detection and instance segmentation with fine-tuning.
We benchmark learned representations on VOCO7 12 object detection task using faster R-CNN (Ren et al., 2015), and on COCO2017 object detection and instance sogmentation using mask R-CNN (He et al., 2017), both with C4 backbone variant (Wu al., 2019). The best results are bold. Methods marked with i and I are not direct comparison to other baselines as they use multiple MLP layers & significant data augmentations and cascade mask R-CNN (Cai & Vasconcelos, 2018) during fine-tuning, respectively.
4.2 Network Architectures
Following FIBER (Dou et al., 2022a), we adopt Swin-Base (Liu et al., 2021) and RoBERTa-Base (Liu et al., 2019) as our vision and text encoders, which are initialized with weights from uni-modal pre-training. We collect patch- and token features from the last transformer layers, feed them into the local projector network, and compute GOT loss. Furthermore, we apply AvgPool on patch and token features, feed them into the global projector network, and compute BT loss. Both local and global projector networks have three linear layers with dimensions 2048-2048-1024, with batch normalization and ReLU after the first two layers. Section 4.7 gives an ablation on projector dimension. We use the image and text features after the AvgPool layer during downstream tasks. For CMAF, we inscrt the cross-attention into the top 6 blocks of the vision and text encoders. Moreover, for direct comparison with existing uni-modal baselines, we re- or-irain VoLTA with Re (He et. al., 2016) and Swin-Ting image encoders.

---

Published in Transactions Machine Learning Research (09/2023)
#Prostrain Data
RefCOCO
RefCOCO+
ReiCOCOg
Method
1-T
I-T-B
vai
testA
test.B
val
tcst.A
test
vai
test
MARNNet
70.4
80.4
69.3
64.9
70.3
56.0
60.7
67.0
VLBERT
3M
71.6
77.7
61.0
VILBERT
3M
72.3
78.5
62.6
BrneeVI-I.
4M
75.9
82-4
06.9
Rosita
4M1
84.8
88.0
78.3
76.1
82.0
67.4
78.2
78.3
UNITER-L
TM
81 1
87.0
74:2
73.9
81.3
06.7
74.9
75.8
VILLA-L
70.2
66.9
70.7
Models trained Jon
Bos
MDETR-B
1.3M
87.5
901.
82.7
81.1
85.5
73.0
83.4
83.3
U
1.3M
86.3
88.8
80.6
78.7
83.2
69.5
80.0
80.0
X-VLM
4M
6.15M
80.2
86.4
71.0
OPA-L
16M
3M
90.1
92.9
85.3
84..5
90..
77.8
84.5
85.2
FIRER-B
0.9M
02.6
85.7
00.1
70.4
87.3
VoLTA-B
231k
86.1
88.6
81.8
77.0
82.7
67.8
78.3
78.3
Table 3: Multi-modal fine-grained downstream: referring expression comprehension. Methods pre-trained image-text-box (I-T-B) data are colored gray, Best comparable results are in bold. VoLTA-B
denotes Swin-B backbone.
COCO Val 2017
IVTS MinVaal
Method
AP
APr
A
API
AF
Modals goo-booiudd on Dis T2l- Hg--oe ond/on
unth
laryer wize
R-CNN
26.3
MDETR
20.9
24.9
24.3
24.2
GLIP-D
31.3
GLIP-L
60.8
FIBER-D
VoLTA-B
51.6
344
43.1
43.8
42.7
Table 4:
Multi-modal fine-grained downstream: language- object detection on COCO and LVIS. All available baselines are pre-trained on Im-Txt-Bo Txt-Box data and are colored gray. VoLTA- denotes Swin-B backbone.
4.3 Implementation Details
We perform pre-training for 20 epochs with 256 batch-size 64 V100 GPUs. Following Zbontar et al (2021), we use the LARS optimizer (You et al., 2017) with a learning rate of 0.2 for the weights and 0.0048 for the biases and batch normalization parameters. We use a learning rate warm-up period epochs, after which we reduce the learning rate by factor of 1000 using cosine decay schedule (Loshchilov & Hutter, 2016). We use le-6 weight decay, excluding the biases and batch normalization parameters. We conduct. a grid search for the GOT loss hyperparameter (wGor), and we empirically found the best value be 100. Appendix D explains other necessary pre-training and downstream hyper-parameters details.
4.4 Results on Vision-onky only tasks
We first experiment on three uni-modal tasks classification, object detection, and instance segmentation. For a direct comparison with existing ResNet50 and Swin-T baselines, we re-train identical encoders with VoLTA pipeline. Furthermore, since the uni-modal tasks do not utilize cross-attention parameters, we perform an ablation by dropping the MLM and ITM objectives from VoLTA.
Image Classification: Table 1 presents the linear probing results of uni-label classification on ImageNet and multi-label classification on VOCO7 and COCO. For all uni-modal tasks, we report results with COCO pre-training for a fair comparison with existing baselines. For ImageNet, we adopt COCO baselines from Yuan et al. (2021). Even without the MLM and ITM objectives, VoLTA achieves better performance than all baselines across three datasets with ResNet55 backbone. The Swin backbones and cross-aliention module further improve the performance. For VOCO7, we report the results for both SVM and MLP-based linear classifiers. VoLTA with ResNet50 backbone achieves state-of-tho-art results on VOCO7 SVM evaluation, beating the nearest baseline, SwAV, by 0.7 mAP scoro. These results indicate the ability of VoLTA to learn effective image-level visual features.

---

Published in Transactions Machine Learning Research (09/2023)
#Pre-train
COCO Captioning
Method
Method
Data
dev
std
dey
Leel-P
TR31
TRAI
Dota
Bitd
M
8
Models
6000
(1236)
VG
Models frsc-tencd without CIDW
optimization
SCAN
108k
18.0
67.4
Vir
18.T
SCG
J08k
29.3
71.8
VL-IS
180%
31.5
28.7
116.5
21.9
PPAN
108k
VI-BART
21.5
MaxtEnt
123k
54.1
54.8
VoLTA-B
231k
58.2
10.77
125.8
22.5
VieuaBERT
125k
70.5
11.0
67.4
67.0
VoLTA-GOLD-E
231k
58.9
30.5
128.5
23.1
LXMERT
231k
72.4
12.5
74.9
optomization
SOHO
231k
73.2
73.4
76.3
77.3
72.5
66.5
VoLTA-B
281k
39.7
30.5
133.0
23.7
VoLTA-B
GOLDA-GOLD-B
Table 5: Multi-modal coarsc-grained downstream: visual question answering, visual reasoning, retrieval, and captioning. We only compare with methods pre-trained on comparable amount of dataset. For captioning, 4 metrics are reported - B@4 BLEU@4, M: METEOR, C: CIDEr, S: SPICE. The best results in bold. VoI.TA- denotes Swin-B backbone.
VoLTA
RefCOCO
RefCOCO
RefCOCOg
Loot
Cer
Craiae
CrTM
val
test
test
val
testA
test
val
test
Lu
Lag
77.8
76.5
82.0
84.5
77.8
71.5
77.1
62.2
71.4
71.8
v
82.7
85.2
78.1
72.0
17.7
62.5
72.8
72.7
/
83.9
86.6
80.5
73.9
79.5
64.1
74.6
743
/
82.8
85.5
78.5
72.2
77.8
62.8
72.9
72.8
86.1
88.6
81.8
77.0
82.7
67.8
78.3
78.3
Table 6: Ablation study on different losses of the training objective of VoLTA for referring
expression comprehension tasks. Each model pre-trained on 231k samples from COCO2017 and VG.
Object Detection & Instance Segmentation: Next, we perform two uni-modal region-level tasks object detection on VOCO7 12 and COCO2017, and instance segmentation on COCO2017. As shown in Table 2, VoL.T yields the state-of-the-art performance in both tasks across the majority of metrics. The fine-grained region-level understanding helps VoLTA to perform well on detection and segmentation tasks.
4.5 Results Fine-grained Vision-Language tasks
Next, perform region-level multi-modal downstream tasks referring expression comprehension (REC) and langnage-guided object detection.
REC: This task aims to localize target objects in an image described by a referring expression phrased in natural language and, thus, perfectly evaluates the fine gra-neri feature representation capability of VoLTA. As depicted in Table 3, VoLTA beats larger-sized UNITER-L and VILLA-L models on the challenging testB split of both RefCOCO and RefCOCO+. Moreover, VoLTA performs comparably with MDETR and UniTAB, even without being trained on grounding data. These results indicate our model's efficacy in learning fino-grained local visual features.
We evaluate VoLTA on two challenging language-conditionee object detection bench- marks- COCO and LVIS. Note that, existing baselines for this lasks are pre-trained on fine-grained smage-tezt-bon data, whereas VoLTA only utilizes emage-caption pairs Table 4 shows that VoLTA performs comparatively with these strong baselines. Note that VoLTA beats Mask R-CNN, MDETR, and GLIP-B on LVIS APr, which denotes average prccision on rare objects. Thus, we conclude that VoLTA achieves impressive localization ability and robustness, even without utilizing any grounding annotations.
Object Detection:
4.6 Results on Coarse-grained Vision-Language tasks
Next, we perform image-level multi-modal downstream tasks visual question answering (VQA), visual reasoning, retrieval, and captioning.
VQA & Visual Reasoning: As reported in Table 5, VoLTA achioves the best performance on VQA and visual reasoning across the baselines pre-trained with comparable amount of data. Moreover, on VQA,
10

---

Published in Transactions Machine Learning Research (09/2023)
Can
VOCO7 MLP
MLP(PC70) (PC70)
COCO
MOO0T
VOC
MLP
MLPP (PC/O)
COCO
Projectur Config.
VOCIT
MLP
MCP [PC/O]
COCO
90.8
72.2/717
93.4
74.1/76.0
8192-8192-12 128
91.3
713/735.
74.0/73.7
100
94.0
74.5/78.4
8192-8192
722/T333
86.5
69.0/69.8
50.2
73.//755
2048-2048-512
91.4
73.9/76.1
91.0
74.5/78.1
500
93.1
728//733
2048-2048- 1024
94.0
74.5/76.4
(a) Ablation study on Intra- and Inter-modal Barlow Twins objec- tive for multi-label image classification on VOCO7 and COCO.
(b) Ablation study on the value of wGor, the weight of GOT loss in Liona in the objective VoLTA for multi- label image classiiiaatio on VOCO7 and COCO.
(c) Ablation study on the di- mension of local and global projector networks of VoLTA for multi-label image classification VOCO7 and COCO.
Table 7: Ablation on Intra- and Inter-modal Barlow Twins objective (a), the value of wooT (b), and the dimension of projector networks (c). We report classification mAP on VOCO7, and per-class (PC) and overall (O) Fl scores on COCO. Each model is pre-trained on 123k train-val samples from COCO2017.
VoLTA beats LXMERT, which is trained with 2x more data. These results demonstrate the efficacy of method even when utilizing mid-scale pre-training corpus.
Retrieval: Most existing VLP methods use a fusion encoder for image and text retrieval and feed every image-text pair into the model. Though such fine-tuning often results in higher performance, it introduces quadratic time cost and is not scalable. Following Dou et al. (2022a), we adopt a more efficient strategy. We drop the cross-attention parameters for this task and compute the dot product of image and text. features extracted separately in the produces superior performance among the baselines trained with a similar amount of data, beating all three baselines by a significant margin.
setting, As shown in Table 5, even with such an approach, VoLTA
We perform captioning on the COCO dataset to evaluate if VoLTA can adopt a generation task. We integrate GOLD (Pang & He, 2021) into VoLTA during fine-tuning as it produces significant improvements. As shown in Table 5, our approach maintains superior captioning performance across all baselines pre-trained with comparable data. Using CIDEr optimization further improves performance.
Captioning:
It is worth mentioning that besides achieving superior result than all baselines using a comparable amount of data on multi-modal coarse-graine gaaree-grained tasks, VoLTA also outperforms multiple methods pre-trained using magnitude ude more dala. These results, shown in Table F.1, indicate the effectiveness and generalizability of VoLTA across these tasks.
4.7 Ablation Study
We perform ablation studies on the pre-training objectives, GOT loss weight, and the dimension of projectors.
Pre-training Objectives: We ablate the effectiveness of different pre-training objectives and evaluate the pre-trained models on fine-grat fine-grained downstream tasks. First, we pre-train VoLTA only with the multi-modal BT loss. In this setup, VoLTA only acts as a dual encoder; thus, the atos-att parameters are not pre-trained. Next, we add MLM and ITM loss which helps the model to learn cross-modal information via attention fusion. Next, we add the GOT pre-training objective. Note that GOT adopts two types of OT distances WD for node matching and GWD for edge matching. As shown in table 6, applying WD and GWD) together improves the performance of reference expression comprehension across RefCOCO, RefCOCO+, and RefCOCOg datasets. Specifically on RefCOCOg, adding Lgw to E yields a significant. 4.0% boost in the challenging test set. Since this dataset contains intricate images with multiple similar objects with different shapes and colors, GWD is crucial in distinguishing between them. However, we see that adding GWD without WD is not helpful. This is because though GWD can capture the edge similarity between graphs, it cannot directly address graph alignment since it does not consider node information. For example, the word pair (boy, girl) has a similar cosine similarity as the pair (football, basketball). Still, the semantic meanings of the two pairs are different and should not be matched. But GWD) will treat these two
11

---

Published in Transactions Machine Learning Research (09/2023)
Caption: man standing next to his guitar case taliin on Nis cel phane.
Caption: A cat is on tatle next to soda bottle.
Caption: bag sitting
sticking
Caption:
siting phone.
Figure 4: Visualization of fine-grained word alignment, produced by VoLTA. We look at the optimal transport plan from GOT to represent the alignment between the words in red with the corresponding input image. All image-caption pairs are taken from the COC02017 train split. The visualizations are generated with 224p images, resulting in sequences of 196 tokens for 16 x 16 patches.
pairs the same since only considers the cosine similarity between nodes. Hence, when applied together, WD and GWD objectives only result in an effective region-word alignment.
We also verify the effectiveness of the multi-modal BT objective by ablating the intra- and inter-modal terms. The first row of Table 7a is identical to the original imagc-only BT objective. Next, we introduce the text branch and add same BT objective between the two views the caption. Afterward, we add the inter- modal BT objectives. As shown in Table 7a, each loss term improved the image classification performance, demonstrating the importance of intra- and inter-modal objectives. Overall, this set of experiments demon- strates that objectives are necessary for our model to perform well on different fino-grained multi-modal tasks.
GOT Loss Weight: our loss formulation, we introduce a GOT loss weight woor which regulates the alignment of local features through GOT loss. By conducting grid search uni-modal downstream classification tasks, we assessed the impact of #GoT as shown in Table 7b and experimentally found its best value be 100 our case. It be noted that very high value of wGoT considerably degrades the performance of downstream lasks.
Ablation on Projector Dimension: The design of the projector head plays a pivotal role in the down- stream performance of the model (Garrido et al., 2022). To investigate the impact of hidden and feature (projector output) dimensions, we have tested 4 different configurations on uni-modal downstream classifi- cation tasks. It can be observed (see Table 7c) that an increase the number of parameters the projector head does not necessarily lead to increase in performance. For example, projector configuration of 8192-8192-256 has roughly eight times more parameters than 2048-2048-102 However, the latter performs better downstream tasks (Table 7e), indicating that the output dimension of the projector plays crucial role in the final performance of the model.
4.8 Qualitative Results Error Analysis
Figure 4 shows the fine-grained alignment of image regions and caption words achieved by the pre-trained VoLTA system. The transport plan from GOT module outputs the similarity across every image patch and caption token. To obtain the visualizations in Figure we choose the similarity scores between the red words in the caption with every image patch. Next, we apply bilinear interpolation Lo these similarity scores convert them to the same dimension as the input image. Finally, we superimpose these interpolated similarity maps on the input images to obtain Figure 4 as the outcome. In most cases, the pre-trained model accurately learns to localize various objects using only global imago-caption data. However, objects in extremely cluttered scenarios occasionally not focused. We show such cases in Section E.
12

---

Published in Transactions Machine Learning Research (09/2023)
Conclusion
We present VoLTA, unified VLP paradign that utilizes image-caption data but achieves fine-grained region- level image understanding, climinating the use of expensive box annotations. VoLTA adopts graph optimal transport-based weakly supervised patch-token alignment and produces an explicit, self-normalized, and interpretable low-level matching criterion. Extensive experiments demonstrate the effectiveness of VoLTA on wide range of coarse- and fine-grained tasks.
Acknowledgement
The codebase for this work is built the Barlow Twins (Zbontar et al., 2021), GOT (Chen al., 2020a), and FIBER (Dou et al., 2022a) repository. We would like to thank the respective authors for their contribution, and the Meta AI team for discussions and leedback. Shraman Pramanick and Rama Chellappa were partially supported by ONR MURI Grant N00014-00-1-2887.
References
Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. Spice: Semantic propositional image caption evaluation. In European Conference on Computer Vision, pp. 382 398. Springer, 2016.
Stanislaw Antol, Aishwarya Agrawal, Jiasen Lu, Margaret Mitchell, Dhruv Batra, C Lawrence Zitnick, and Devi Parikh. Vqa: Visual question answering. In International Conference on Computer Vision, pp. 2425 2433, 2015.
Mahmoud Assran, Mathilde Caron, Ishan Misra, Piotr Bojanowski, Florian Bordes, Pascal Vincent, Armand Joulin, Michael G. Rabbat, and Nicolas Ballas. Masked siamese networks for label-efficient learning, arXiv, 0bs/2224.071411 2022.
Yogesh Balaji, Rama Chellappa, and Soheil Feizi. Normalized wasserstein for mixture distributions with applications in adversarial learning and domain adaptation. In International Conference Compuler Vision, pp. 6500-6560 6508, 2019.
Satanjeev Banerjee and Alon Lavie. Meteor: An automatic metric for mt evaluation with improved correla- tion with human judgments. In Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization, pp. 65 65-72, 2005.
Hangbo Bao, Li Dong, Soughao Piao, and Furu Wei. Beit: Bert pre-training of image transformers. In International Conference Learning Representations, 2021.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variane-invarrinne-coorraan regularization for self-supervised learning. International Conference on Learning Representations, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877-199 1901, 2020.
Zhaowei Cai and Nuno Vasconcelos. Cascade r-cmn: Delving into high quality object detection. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6154-6162, 2018.
Mathildo Caron, Hugo Touvron, Ishan Misra, Horv's J J'egon, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. International Conference on Computer Vision, pp. 9630-9640, 2021.
Chun-Fu (Richard) Chen, Quanfu Fan, and Rameswar Panda. Cross VIT: Cross-Attontinn Multi-Scale Vision Transformer for Image Classification. In International Conference on Computer Vision, 202 la.

---

Published in Transactions on Machine Learning Research (09/2023)
Liqun Chen, Yizhe Zhang, Ruiyi Zhang, Chenyang Tao, Zhe Gan, Haichao Zhang, Bai Li, Dinghan Shen, Changyou Chen, and Lawrence Carin. Improving sequence-to-sequeeee learning via optimal transport. In International Conference on Learning Representations, 2019. URL https://openrevaat S1xtAjR5tX.
net/forum?id
Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, and Jingjing Liu. Graph optimal transport for cross-domain alignment. In International Conference on Machine Learning, pp. 1542 1553. PMLR, 2020a.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International Conference on Machine Learning, pp. 1597-1607.
PMLR, 2020b.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. Proccedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 1575511773, 2021.
Xinlci Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.04997 2020c.
Xinlei Chen, Saining Xie, and Kaiming He. An empirical study of training self-supervised vision transformers. International Conference on Computer Vision, pp. 9620-9629, 2021b.
Con-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Lin. Uniters Universal image-lext representation learning. In European Conference Computer Vision, pp. (04-120. Springer, 2020d.
Jaomin Cho, Jie Lei, Hao Tan, and Mohit Bansal. Unifying dision-and-ll liinn-and-langguae tasks via text generation. In International Conference Machine Learning, pp. 1931 1942. PMLR, 2021.
Xiyang Dai, Yinpeng Chen, Bin Xiao, Dongdong Chen, Mengchen Liu, Lu Yuan, and Lei Zhang. Dynamic head: Unifying object detection heads with attentions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7373-7383 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenel: large-scale hierarchical image database. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 248 255. leee, 2009.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 [Long and Short Papers), pp. 4171-4188 2019.
Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- lerthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale. In International Conference on Learning Representations. 2021. URL https: /ttpr//////p/eeee/eeee http://parreeeee.ttttoomm////aaa wtt.://oromiici/.cccaaaa
Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie I., Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, et al. Coarse-to to-fine uision-lanuuae pre-training with fusion in the backbone. Advances in Neural Information Processing Systems, 2022a.
Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, La Yuan, Nanyun Peng, et al. An empirical study of training end-to-end vision-and- language transformers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18166-1888 18176, 2022b.
Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88(2):303 338, 2010.
14

---

Published in Transactions Machine Learning Research (09/2023)
Tianyu Gao, Xingcheng Yao, and Danqi Chen. Simese: Simple contrastive learning of sentence embeddings. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6894 6910, 2021.
Quentin Garrido, Yubei Chen, Adrien Bardes, Laurent Najman, and Yann Lecun On the duality between contrastive and non-contrastive self-supervieed learning. arXiv preprint arXiv:2206.0257. 2022.
Aude Genevay, Gabriel Peyré, and Marco Cuturi. Learning generative models with sinkhorn divergences. In International Conference Artificial Intelligence and Statistics, pp- 1608-1617. PMLR, 2018.
Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurams, Sergey Levine, and P. Abbeel. Multimodal masked autoencoders learn transferable representations. arXiv, abs/2205 14204, 2022.
Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming Hc. Accurate, large minibatch sgd: Training imagenct in hour. arXiv preprint 1rXiv:1766.0266: 706.02677, 2017.
Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Talloc, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a now approach to self-supervisod learning. Advances in Neural Information Procossing Systems, 33:21271-21284, 2020.
Agrim Gupla, Piotr Dollar, and Ross Girshick. Lvis: dataset for large vocabulary instance segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5356 5364, 2019.
Song Han, Jeff Pool, John Tran, and William Dally. Learning both weights and connections for efficient neural network. Advances in Neural Information Processing Systems, 28, 2015.
Xiao Han, Xiatian Zhu, Licheng Yu, Li Zhang, Yi-Zhe Song, and Tao Xiang. Fame-vil: Multi-tasking vision-language model for heterogeneous fashion lasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2669 2680, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 770-778, 2016.
Kaiming He, Georgis Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cnm. In International Conference on Computer Vision, pp. 2961-2969, 2017.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Procoedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoeness are scalable vision learners. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022.
Zhicheng Huang, Zhaoyang Zeng, Bei Liu, Dongmei Fu, and Jianlong Fu. Pixel-bert: Aligning image pixels with text by deep multi-modal transformers. arXiv preprint arXiv:2004.00849, 2020.
Zhicheng Huang, Zhaoyang Zeng, Yupan Huang, Bei Liu, Dongmei Fu, and Jianlong Fu. Seeing out of the box: End-to-end pre-training for vision-language representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 12976- 19985,, 2021.
Jiho Jang, Chacrin Kong, Donghyeon Jeon, Sconhoon Kim, and Nojun Kwak. Unifying vision-language representation space with single-tower transformer. In Procoedings of the AAAI Conference on Artificial Intelligence, 2023.

---

Published in Transactions on Machine Learning Research (09/2023)
Chao Jia, Yinfci Yang, Ye Xia, Yi-Ting Chen, Zarana Parckh, Hicu Pham, Quoc Le, Yun-Hsuan Sung, Zhen Lá, and Tom Duerig. Scaling up visual and vision-lagguage representation learning with noisy text supervision. In International Conference on Machine Learning, pp. 4904-4916. PMLR, 2021.
Aishwarya Kamath, Mannat Singh, Yann LeCun, Gabriel Synnaeve, Ishan Misra, and Nicolas Carion. Mdetr- understanding, In International Conference on Computer
modulated detection for mod-to-end Vision, pp. 1780 1790, 2021.
Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. Referitgame: Referring to objects in photographs of natural scenes. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, pp. 787 798, 2014.
Douwe Kiela, Suvrat Bhooshan, Hamed Firooz, and Davide Testuggine. Supervised multimodal bitrans- formers for classifying images and text. arXiv, abs/ /t//1999.02220 02950, 2019.
Wonjae Kim, Bokyung Son, and Iddoo Kim. Vilt: Vision-and- language transformer without convolution or region supervision. In International Conference on Machine Learning, pp. 5583-5594. PMLR, 2021.
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In International Journal of Computer Vision, volume 123, pp. 32-73. Springer, 2017.
Juncheng Li, Xin He, Longhui Wei, Long Qian, Linchao Zhu, Lingxi Xic, Yucting Zhuang, Qi Tian, and Siliang Tang. Fine-grained semantically aligned vision-language pre-training, In Advances in Neural Information Processing Systems, 2022a.
Jumnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. Aligu before fuse: Vision and language representation learning with momentam distillation. Advances in Neural Information Processing Systems, 34:9694 9705, 2021a.
Jumnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-iman pre-training for
unified vision- vision-language understanding and generation. In International Conference Machine Learning, 2022b.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Boolstrapping language-ing imnge pre-training with frozen image encoders and large language models. In International Conference on Machine Learning, 2023a.
Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. What does bert with vision look at? In Procoedings of the 58th Annual Meoting of the Association for Computational Linguistics, pp. 5265 5275, 2020a.
Liunian Harold Li, Pengchuan Zhang, Haotian Zhang, Jianwci Yang, Chunyuan Li, Yiwu Zhong, Lijuan Wang, Lu Yuan, Lei Zhang, Jenq-Neng Hwang, et al. Grounded language-image pre-training. In Proceedings of the ICEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10965- 10975, 2022c.
Manling Li, Ruochen Xu, Shuohang Wang, Luowei Zhou, Xudong Lin, Chenguang Zhu, Michael Zeng, Heng Ji, and Shih-Fu Chang. Clip-event: Connecting text and images with event structures. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 16420-16429, 2022d.
Xiujun Li, Xi Yin, Chunyuan Li, Pengchuan Zhang, Xiaowei Hu, Lei Zhang, Lijuan Wang, Houdong Hu, Li Dong, Furu Wei, et al. Oscar: Object-semanties aligned pre-training for vision-language tasks. In European Conference on Computer Vision, pp. 121-137. Springer, 2020b.
Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie Yan. Supervision exists everywhere: A data efficient contrastive language-image pre-training paradigm. In International Conference Learning Representations, 2021b.
16

---

Published in Transactions on Machine Learning Research (09/2023)
Yanghao Li, Haoqi Fan, Ronghang Hu, Christoph Feichtenhofer, and Kaiming Hc. Scaling language-image pre-training via masking. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 23390-2830 -3400, 2023b.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and Lawrence Zitnick. Microsoft. coco: Common objects conlext. European Conference on Computer Vision, pp. 740-755. Springer, 2014.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Dangi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. ar preprint 1xXiv:190.111622 2019.
Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin trans- former: Hicrarchical vision transformer using shifted windows. In International Conference on Computer 2021.
Vision, pp.
llya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent. with warm restarts. arXiv preprint. arXiv:160S 100.00003.3383 2016.
Jiasen Lu, Dhruv Batra, Devi Parikh, and Stefan Lee. Vilbert: Pretraining task-agnostic visiolinguistic representations for assionand-languaan tasks. Advances in Neural Information Processing Systems, 32, 2019.
Zhiyuan Ma, Jianjun Li, Guolui Li, and Kaiyan Huang. Cial: A novel cross-modal associative learning framework for vision-language pre-training. In Proceedings of the 30th ACM International Conference on Multimedia, pp. 4515-4524, 2022.
Youssef Mroueh, Chun-Liane Li, Tom Sercu, Anant Raj, and Yu Cheng, Sobolev gan. In International Conference on Learning Representations 2018.
Youssef Mrouch, Tom Sercu, and Anant Raj. Sobolev descent. In International Conference on Artificial Intelligence and Statistics, pp. 2976 2985. PMLR, 2019.
Norman Mu, Alexander Kirillov, David A. Wagner, and Saining Xie. Slip: Self-supervision meets language- image pre-training. arXiv, 0tss/2112.12750, 2021.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive predictive coding. arXiv preprint 1rXivx1807.007740 2018
Richard Yuanzhe Pang and He He. Text generation by learning from demonstrations. In International Conference on Learning Representations, 2021.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. Bleu: a method for automatic evaluation of machine translation. In Proccedings of the 40th Annual Meeting of the Association for Computational Linguistics, pp. 311 318, 2002.
Jaeyoo Park and Bohyung Han. Multi-modal representation learning with text-driven soft masks. In Proceedings of the IVEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2798-2807, 2023.
Gabriel Peyré, Marco Cuturi, and Justin Solomon. Gromov-wwsserstein averaging of kernel and distance matrices. In International Conference on Machine Learning, pp. 2664-2672. PMLR, 2016.
Gabriel Peyré, Marco Cuturi, et al. Computational optimal transport: With applications to data science. Foundations and Trends@ in Machine Learning, 11(5-)):355 607, 2019.
Bryan A Plummer, Liwci Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k ontities: Collecting region-to-phrase correspondences for richer image-to-sentenee models. In International Conference on Computer Vision, pp. 2641 2649, 2015.
17

---

Published in Transactions Machine Learning Research (09/2023)
Shraman Pramanick, Aniket Roy, and Vishal M Patel. Multimodal learning using optimal transport for sarcasm and humor detection. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 1880-3940, 2022.
Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengehuan Zhang. Egovipv2: Egocentric video-language pre-training with fusion in the backbone. Xiv preprint arXiv:2307.05463 2023.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. International Conference on Machine Learning, pp. 8748 8763. PMLR, 2021.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in Neural Information Processing Systems, 28, 2015.
Anshul Shah, Suvrit Sra, Rama Chellappa, and Anoop Cherian. Max-margin contrastive learning. In Procoedings the AAAI Conference on Artificial Intelligence, volume 36, pp. 8220-8230, 2022.
Amanpreet Singh, Ronghang Hu, Vedanuj Goswami, Guillaume Couairon, Wojeiech Galuba, Marcus Rohrbach, and Douwe Kiela. Flava: A foundational language and vision alignment model. In Proceedings of the LEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15638 11658,, 2022.
Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. Vi-bert: Pre-training of generic visual-linguistie representations. In International Conference on Learning Representations: 2019.
Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoar Artzi. A corpus for reasoning about natural language grounded in photographs. In Proccedings of the Annual Meeting of the Association for Computational Linguistics, 2019.
Hao Tan and Mohit Bansal. Lxmert: Learning cross-modality encoder representations from transformers.
In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Procossing, pp. 5100-5111, 2019.
Ramakrishna Vedanlam, C Lawrence Zitnick, and Devi Parikh. Cider: Consensus-basee image description evaluation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566-4575 2015.
Jianfeng Wang, Zhengyuan Yang, Xiaowei Hu, Linjie Li, Kevin Lin, Zhe Gan, Zicheng Liu, Ce Liu, and Lijuan Wang. Git: A generative image-to-text transformer for vision and language. Transactions Machine Learning Research, 2023a.
Jinpeng Wang, Pan Zhou, Mike Zheng Shou, and Shuicheng Yan. Position- guidded text prompt for vision- language pre-training. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition, pp. 23242-23251, 2023b.
Junke Wang, Dongdong Chen, Zaxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, and Lu Yuan. Omnivl: One foundation model for mmago-aaguuage and video-language tasks. In Advances in Neural Information Processing Systems, 2022a.
Peng Wang, An Yang, Rui Men, Jumyang Lin, Shuai Bai, Zhikang Li, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. Ofa: Unifying architectures, tasks, and modalities through simple sequence-to- sequence learning framework. In International Conference on Machine Learning, pp. 28318-23340 PMLR, 2022b.

---

Published in Transactions on Machine Learning Research (09/2023)
Tong Wang, Yixiao Ge, Feng Zheng, Ran Cheng, Ying Shan, Xiaohu Qie, and Ping Luo. Accelcrating vision-langnage pretraining with free language modeling. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23161-23170, 2023c.
Wenhui Wang, Hangbo Bao, Li Dong, and Furu Wei. Vlmo: Unified vision-language pre-training with mixture-of-mmodality-oooot.. arXiv preprint a1X1:2111.003588, 2021a.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Mohammed, Saksham Singhal, Subhojit Som, and Furu Wei. Image as a forcign language: Beit pretraining for all vision and lisionllaguanl tasks, arXiv, 0as//2008.1044 2022c.
Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, et al. Image as a forcign language: Beit pretraining for vision and vision-language tasks. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19175-19186 2023d.
Zirui Wang, Jiahui Yu, Adams Wei Yu, Zihang Dai, Yulia. Tsvetkov, and Yuan Cao. ple visual language model protraining with weak supervision. In International Conference on Learning Representations, 2021b.
Simvlm: Sim-
Jason Wei and Kai Zou. Eda: Easy data augmentation techniques for boosting performance text classifi- cation tasks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, pp. 6382 6388, 2019.
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Yen Lo, and Ross Girshick. Detectron2. https: 2019.
//github. hoaa///ccboooreserccd/eeeettttte
Hongwei Xuc, Yupan Huang, Bei Liu, Houwen Pong, Jianlong Fu, Houqiang Li, and Jiebo Luo. Probing inter-modality: Visual parsing with self-attention vision-and-langnage pre-training. Advances Neural Information Processing Systems, 34:4514-4525, 2021.
Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, and Jianfeng Gao. Unified contrastive learning in image-text-label space. In Proceedings of the IEFE/CVF Conference on Computer Vision and Pattern Recognition, pp. 19163-19173 2022a.
Jinyu Yang, Jiali Duan, Son Tran, Yi Xu, Sampath Chanda, Liqun Chen, Belinda Zeng, Trishul Chilimbi, and Junzhou Huang. Vision-langual pre-training with triple contrastive learning, In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15671-15680, 2022b.
Zhengyuan Yang, Zhe Gan, Jianfong Wang, Xiaowoi Hu, Faisal Ahmed, Zichong Liu, Yumao Lu, and Lijuan Wang. Unitab: Unifying text and box outputs for grounded vision-language modeling. In European Conference on Computer Vision,, 2022c.
Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Lí, Xin Jiang, and Chunjing Xu. Filip: Fine-grained interactive language-image pre-training. In International Conference on Learning Representations, 2022.
Haoxuan You, Luowei Zhou, Bin Xiao, Noel Codella, Yu Cheng, Ruochen Xu, Shih-Fu Chang, and Lu Yuan. Learning visual representation from modality-share contrastive language anngagg--mmage pre-training, In European Conference Computer Vision, 2022.
Yang You, Igor Gitman, and Boris Ginsburg. Large batch training of convolutional networks. arXiv preprint arXiv: 1xXiv:1708.08 008888888 2017.
Licheng Yu, Patrick Poirson, Shan Yang, Alexander C Berg, and Tamara Berg. Modeling context referring expressions. In European Conference Computer Vision, pp. 69-85. Springer, 2016.
Siyang Yuan, Ke Bai, Liqun Chen, Yiahe Zhang, Chenyang Tao, Chunyuan Li, Guoyin Wang, Ricardo Henao, and Lawrence Carin. Advancing weakly supervisod cross-domain alignment with optimal transport. In British Machine Vision Conference, 2020.
19

---

Published in Transactions Machine Learning Research (09/2023)
Xin Yuan, Zhe Lin, Jason Kuen, Jianming Zhang, Yilin Wang, Michael Maire, Ajinkya Kale, and Baldo Fai- eta. Multimodal contrastive training for visual representation learning. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pp. 6995-7004, 2021.
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning redundancy reduction. In International Conference on Machine Learning, pp. 12310-12320, PMLR, 2021.
Yan Zeng, Xinsong Zhang, and Hang Li. Multi-grained vision language pre-training: Aligning texts with visual concepts. In International Conference on Machine Learning, 2022.
Chi Zhang, Yujun Cai, Guosheng Lin, and Chunhua Shen. Deepemd: Pew-shot image classification with dif- ferentiable earth mover's distance and structured classifiers. Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pp. 12203-12222 12213, 2020.
Haotian Zhang, Pengchuan Zhang, Xiaowei Hu, Yen-Chun Chen, Liunian Harold Li, Xiyang Dai, Lijuan Wang, Lu Yuan, Jenq-Neng Hwang, and Jianfeng Gao. Glipv2: Unilying localization and vision-language understanding. Advances in Neural Information Processing Systems, 2022.
Pengchuan Zhang, Xiujun Li, Xiaowei Hu, Jianwei Yang, Lei Zhang, Lijuan Wang, Yejin Choi, and Jianfeng Gao. Vinvl: Revisiting visual representations in vision-language models. In Procoedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5579-5588, 2021.
Luowci Zhou, Hamid Palangi, Loi Zhang, Houdong Hu, Jason Corso, and Jianfeng Gao. Unified vision- language pre-training for image captioning and vqa. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 13041-13044 2020.
20

---

Published in Transactions Machine Learning Research (09/2023)
A Pesudo Code of VoLTA
The training pseudo code for VoL.T is as follows:
Algorithm 1 PyTorch-style pseudocode for VoI.TA.
# f_I: Image Encoder, f_T: Text Encoder
mask_names: string containing task names
Image input, Т: Text input, N: Batch size, D: Projector dim
a ET: Barlow Twins loss function
WD, GWD: Wasserstein and Gromov-kasserstein loss functions
MLM, ITK: NLM and ITM loss functions, respectively
gamma: coefficient of GWD loss in GDT
# w_GOT: weight of GOT loss
def GOT x_2, f f_1, f_2):
# compute embeddings
z_A, 2_B f_1(x_1), f_2(x_2) # X
normalize representation along batch dimension
2_A_norm (2_A A.m.mn (din=0)) / 2 st..st
2_ __normm (z_B 2_B. mean 2 s_B.std(di
cosine distance matrix
= cosine cosine_dist cett_matrix(z__. z_B)} Wasserstein distance losa_w Dcc, si)e s.A.e z s1))
# Gromov-Wassersterl distance
loss_g* GM GM_D(z_A. transpose 1), 2_B. transposs (2, (2, 1))
return gamma torch.mean (1oss_g4) (1 gamma)
norch.mean (lo8s_x)
VoLTA (I, T):
total loss torch. (O.)
for x in loader: load batch with Nl samples two augnented versions of I, T
I1, I2
gegnent__image (I); T1, augment_text
if "Bтсот" in task task_names: names:
BT loss
intra_loas ET(I1, f_I) BT(T1, T2, f_T)} inter loas ET(I, T1, I_I, f_T) BT(I2, T2, f_I, f_T)} BT [T_loss intra loss inter inter_loss
Lotal_loas BT_loss
GOT loss GOT loss CII T1, fII, f_T) + CIT T2, f.I, f_T)} total_loss w_GOT COT_loss
cross-attention is enabled if "MLK" in task_names. names:
# MLM loas MLM loss MLM(T1, I1, mask, T1, f_I, f_T)
total_ loss MLM
if " ITK" in task names:
ITM loss
ITM loss =ITM(T1, I1, false_ image_1, f_I, f_T)
total loss += loss
total loas
B Overview of Vision-Language Pre-training Models
Vision-Language Pre-trained (VLP) models have proven extromely beneficial for multi-modal tasks in recent years. Earlier works predominantly focused using pre-trained object detectors to extract patch

---

Published in Transactions Machine Learning Research (09/2023)
Model
Vision
Multimodality Fusion
Pre-train
Pre-training Objectives
FT
1-T-B
VILBERT
19
IIM-MMM
[Xformer
LXMERT
EMNLP'19
MLM+ITM +MMM-VQA
ViRwBBERT
ACI'20
MLM-ITM
VI-BERT
ICLR'20
MLM+MMMM
UNTTER
ECCV"20
MLM+IIM MA--II---MM--ERRR
OD
Emb.
Merged
OSCAR
EOCV
MLMH-IM
VinVI.
CVPR'21
MLM4ITM
VI-T5
(CMI,21
SONO
CVPR'21
MLM IIM-IM--- MIM
Eiub.
/
SinVLM
ICLR22
CNN
Merged stin
PretiXLM
MOETH
1CCV'21
Xformer
OD+TP+CA
VILT
ICML'21
Pasch Emab
MLM+ITM
Visud Parsing
N11
Eanb.
Mergod attm
MLM- -----TM---II MIM
ALBEF
N1
Xformer
MIM-IIM+IIC
Xformer
Co-atta
METER
CVPR222
MLM+ITM
CLIP
ICMLL'2 221
/NoeXfeener
ITC ITC ILL
DeCLIP
ICLR'11
None
ALIGN
ICMLL/21
CNN Oo4Xforeer
HTC
GLIP
CVPR'22
OD+CE+WRA
GLIPv2
NeurIP8222
OD-CE-WRA-MLAN
RLIP
JCMI,22
Cree-modality MHA
ITC- IIT--MM-MMMMMMMMMM-----M
OmLiVVI
NeurlP8
SWVVLC4VMM-I
X-VLM
ICML/22
Xlormer
BBP+ITC+ CAP-IIC+MP-AAA MPLM
CMAL
ACM MM'22
MLM ITM
LOUPE
Neu21P5/22
ITCH I TSA
None
UaiCL
CVPn'22
CAN/XEee CNN
Xformer
ITC CMA IMCH LMI -TM--MMMMM
UnITAB
ECCV'22
Merged atin
TCL
CVDD'22
MS-CLIP
ECCV
Shared Attention
FLM
CVPR'23
Xiormer
FLM + IIM +IC+IIM+IIG TTC ITC-ITM-IM ITC-ITM-MLM ITC: XMCI CMC
Croeemmodallly MHA
BLIP-2
Pama-Vil.
CVPR
Croe-nodality Arhstiise Attention
PTP
CVPR'23
Xformer
Crose-moddllty MHA
Softmack++
CVPR223
Merrjed simn
OucR.
2AAT223
Uiifted aftn
BEIT-3
CVPR
Xformer
Shiridd MBA
MDM ITC LM
PLIP
CVPR/23
Roine
GIT
TMLB'23
Linb."
FIBER
P2
Xloemer
Xformei
Chealin
MIMIITMATIC
VoLTA
T3
UNN/Xloemee
Xlormee
Mergee Coeait
-T MECOT+4MMME MMMM-ITM
Table B.I: Overview of VLP models. OD: objective detector. Xformer: transformer. Emb.: em- bedding, MLM/MIM: masked language/imann /image modeling. ITM: image-text matching. WRA: word-region aligmment. ITC: image-fext. contrastive learning. Grad: Grounding. Cap: Captioning. TP: Token Pre- diction. CA: Contrastive Alignment, NNS: Nearest Neighbour Supervision, MVS: Multiview Supervision, SL: Sim-siam Loss, MHA: Multi-head attn., LM: Language Modeling, UniVLC: Unified Vision Language Contrastive, VLM: Vision Language Matching, BBP: Bounding Box Prediction, MP: Matching Prediction, FSA: Fine-grained Semantic Alignment, TSA: Token-level Semantic Aligmment, AMC: Associative Mapping Classification, CMA: Cross-Modal Alignment, IMC: Intra-Modal Contrastive, LMI: Local Mutual Informa- Lion Maximization, I-T: Image-TTess Text, 1-T-B: Image- Tmage-Texxi-Box, ITG: Image grounded text generation, XMC: Cross-Modal Mixup Contrastive, CIC: Contextual Invariance Contrastive, CMC: Contextual Mixup Con- trast, MDM: Masked Data Modeling.
(region) level information from corresponding images (Lu et al., 2019; Li et al., 2020a; Tan & Bansal, 2019; Chen et al., 2020d; Su et al., 2019). In of these models, such ViLBERT (Lu et al., 2019), and LXMERT (Tan & Bansal, 2019), multi-modality fusion has been achieved via co-attention using a third transformer which contains fused information independently obtained from respective vision and language encoders. On the contrary, VisualBERT (Li et al., 2020a), VL-BERT (Su et al., 2019), and UNITER (Chen et al., 2020d) employ a merged at attention strategy to fuse both image patches and text features Loget her into a unified transformer through corresponding image and text embedders. In addition to these, OSCAR (Li al., 2020b) uses object tags inputs. VinVL (Zhang et al., 2021) follows a similar strategy to that of OSCAR, the only difference being their novel 3-way contrastive loss which optimizes the training objectives used for VQA and text-image matching. VI.-T5 (Cho et al., 2021) exploits bounding-box coordinate information,

---

Published in Transactions on Machine Learning Research (09/2023)
image IDs, and region IDs along with ROI features for visual embodding. Here, encoded visual and textual features are fed into bi-directional multi-modal encoder and an auto-regressive text decoder framework, respectively, for pre-training.
In all the above methods, pre-trained object detectors are kept frozen during the training. Furthermore, extracting region-level features from images can be ledious. To address these shortcomings, end-to-end pre- training methods have been developed. PixelBERT (Huang et al., 2020) uses a and sentence encoder to obtain image and text representations, respectively. These representations are subsoquently fod into a transformer via a cross-modality alignment. SOHO (Huang et al., 2021) uses grid features- hased discretization via learned vision dictionary which is then fed into cross-modal module. SimVLM (Wang et al., 2021b) uses CNN and text token embedding for image and text feature representation extraction with a unified encoder-decoder transformer trained on a PrefixI. objentive. Finally, MDETR (Kamath et al., 2021) uses CNN and RoBERTa (along with corresponding projection layers) for image and text feature extraction. These extracted features are concatenated before passing through a unified transformer trained on 1.3M Image-Texte Text-Box (1-T-B) annotated data.
vision encoder
In recent years, the rise of Vision Transformers (VTT) (Dosovitskiy et al., 2021) has motivated the rescarch community to have an all-transformer framework by incorporating ViTs (instead of CNN backbones) in VLP models. Image patch features and text token embeddings
fed directly into ViT model for pre-training in ViLT (Kim el al., 2021). Visual Parsing (Xue et al., 2021), ALBEF (Li et al., 2021a), and METER (Dou et al., 2022b) use VITs as vision encoders for image feature generation. ALBEF and METER use co-attention in their pre-training frameworks for multinodality fusion.
Another class of VLP models in the form of CLIP (Radford et al., 2021), DeCLIP (Li et al., 2021b), and ALIGN (Jia et al., 2021) has been introduced lately. Although known for their impressive zero-shot recognition ability and excellent. transferability to downstream tasks, these models typically rely on huge amounts of image-text pairs for pre-training, Contrastive loss forms the core component of the pre-traiming objectives in these VLP models. In such models (e.g., CLIP (Radford et al., 2021). DeCLIP (Li et al., 2021b)), separate encoders have been used for each modality. On the contrary, modality-sharod contrastive language-image pre-training (MS-CLIP) (You et al., 2022) leverages knowledge distribution across multiple modalities (image and text) through paramcter sharing. In their unified framework, the paramcters which are being shared between two modalities include the attention and forwrd modules and the layerNorm layers.
GLIP (Li et al., 2022c) and GLIPv2 (Zhang et al., 2022) use a localization loss along with a word-reph vord-region aligmment loss for pre-training corresponding encoders using image-text-box annotations. BLIP (Li et al., 2022b) employs image and text encoders connected through cross-modality multi-head attention which are pre-trained on image-text pairs using contrastive and language modeling objectives. OmniVL (Wang et al., 2022a) utilizes unified image (and video) encoder and text encoder pre-trained on imago-text, imbge-label, video-text, and video-label pairs using unified vision-language contrastive, vision-language matching and language modeling losses. Furthermore, a visual-grounded alignment decoder is also present for facilitating better learning and alignment between various modalities. X-VLM (Zeng et al., 2022) employs vision transformer to extract features from the subset of patches reprosenting moages/regionss /ooggiss//oocess.. These patch features are then paired with associated text features for contrastive learning, matching, and masked language modeling. Additionally, image and text pairings are also done for bounding-box prediction which is usod to locate visual concepts in the image. CMAL (Ma et al., 2022) proposes interactions between features (obtained from respective image and text. encoders) via cross-modal associative mappings which help in fine-grained semantic aligument between the learned representations. LOUPE (Li et al., 2022a) implements token-level and semantics-level Shapley interaction modeling with global image-text contrastive loss (in a dual-encoder setting) for explicit learning of fine- grained semantic aligument between visual regions and textual phrases without using expensive bounding-box annotations. FILIP (Yao et al., 2022) removes the need for cross- modality attention fusion by modeling the fino- grained semantic alignment between visual and textual tokens via. a novel cross-modal late interaction mechanism in contrastive loss. TCL (Yang et al., 2022h) uses global cross- modal alignment, intra-modal aligument, and local mutual information maximization losses along with masked language modeling and image-lext matching learn robust image-text representations during pre- training. UniCL (Yang et al., 2022a) utilizes a unified learning method with a two-way contrastive loss
23

---

Published in Transactions on Machine Learning Research (09/2023)
(imago-to-text and text-to-image) the imago-text-label space which can learn representations from either of the image-label and image-text data both. UniTAB (Yang et al., 2022c) employs a transformer-based eucoder-decodee framework that can jointly output open-ended text and box, encouraging alignment between words and boxes.
In order to accelerate the convergence of VL pretraining, Wang et al. (2023c) proposed free language modeling (FLM) which addresses the issues inherent to masked language modeling (MLM) and autoregressive mod- cling. Using FLM as pro-training objective, the authors have achieved impressive performance on several downstream tasks. Fame- VaL (Han et al., 2023) introduces a paramcter-cffiiint VLP approach employing a task-versatile architecture with cross-attention and task-specific adapters. Fame-Vill Vin applies single model for various heterogencous fashion tasks achieving performance gains over previous SOTA benchmarks. Wang et al. (2023b) have introduced a simple yet effective position- text prompt. (PTP) paradigm to improve the visual grounding capability of existing cross moss-modal VL architectures and help them better handle various downstream tasks. Park & Han (2023) have proposed a VL framework based on the explainable soft feature masking and regularization via diversification strategics for improving the performance of VL models several downstream tasks. Li et al. (2023a) have devised BLIP-2, where lightweight querying transformer is pro-trained using a two-stage stratogy to bridge the modality gap. A frozen encoder is used in the first stage to bootstrap VL representation learning, and rision-to-languee generative learning is bootstrapped in the second stage employing frozen LLM, allowing vero-shot generation capabilitiess Jang el al. (2023) have developed a simple and unified VL model (as a single tower) in a modality- multimodal foundation model to pro-train multiway
manner. BEiT-3
(Wang et al., 2023d) introduccs general-purpert transformer by performing masked data modeling on inputs irrespective of modalities (i.e., images, texts, and image-text pairs). FLIP (Li et al., 2023b) extends CLIP (Radford et al., 2021) by performing contrastive learning pairs of masked image patches and corresponding texts without reconstructing the masked image content. GIT (Wang et al., 2023a) unifies the Vl. architecture (an image encoder and a text decoder) under a single language modeling task while also scaling up the pre- pre-training data and the model size to gain superior performance captioning and quession-answering downstream lasks.
FIBER (Dou ot al., 2022a) fuses vision and language encoder backbones through merged co-attention which are then pre-trained on 4M data with two-stage pre-training (coarse- and fine fininnddd Image-text pairs are used in the coarse-grained pre-training stage which is then followed by fine-grained pre-training stage with amage-text-box annotations. However, these bounding box annotations come with extra overheads. There- fore, in our model, VoLTA, we propose an alternate solution for optimal-transport based local feature-level alignment using global image-caption annotations which performs well not only on coarse-grained tasks (such as VQA and Image Captioning), but also fine-grained lasks (such as Referring Expression Comprehen- sion and Object Detection). Table B.1 encapsulates an overview of the details of all these aforementioned methods.
C Downstream Datasets
Our downstream tasks can be categorized into three groups: uni-modal, multi-modal coarse-grained, and multi-modal fine-grained.
Uni-modal: For uni-modal tasks, we fine-tune (and validate) our pre-trained model on ImageNet-1k (Deng et al., 2009) for image classification, VOC07+12 (Everingham et al., 2010) for image classification and object detection, and COCO (Lin et al., 2014) for image classification, object detection, and instance segmentation.
Multi-modal Coarse-grained: Here, we fine-tune (and validate) our pre-trained model on VQAv2 (Antol al., 2015) for visual question answering, NI.RR2 (Suhr et al., 2019) for visual reasoning. Flickr30k (Plummer et al., 2015) for image and text retrieval, and COCO (Lin et al., 2014) for image captioning.
Multi-modal Fine-grained: For these tasks, we fine-tune (and validate) our pre-trained model on Re- ICOCO, ReICOCO+, and RefCOCOg (Kazemzadeh et al., 2014; Yu et al., 2016) for referring expression comprehension, and COCO (Lin et al., 2014) and LVIS Mini (Gupta et al., 2019) for language-condtionee object detection.
24

---

Published in Transactions Machine Learning Research (09/2023)
Modality
Task
Dataset
Image Src
# #Images
#Tex
Metric
IN-IK
IN-IK
1.3M
Accuracy
Image Classhn
COCO
COCO
123K
F1
VOCU7-12
VOC07+12
16K
mAP
Uni-modal
Object Det.
COCO
COCO
123K
APM
VOCUT-12
VOCU7+12
16K
Instanoe Sog.
COCO
COCO
123K
A
VQA NLVR"
VQA
COCD
201K
LIM
VQA-Soore
Multi-modal
NLVR
Web Crawled
214K
107K
Accuracy
coorse-grained
IR-RR
Flickr30K
Flick33KK
32K
100K
Recal
Capsioning
COCO
COCO
123K
615K
B.L
ReCOOC
20R
142K
Ref. Exp. Comp.
ReCCOCO+
COCO
20K
142K
Accuracy
Mult-moodal fine-grained
ReiCOCOg
20K
95K
"COCO"
C000
123K
6T5K
Obj,
AP
IVIS Mini
COCO
123K
615K
Table C.I: Dataset statistics for uni-modal and multi- downstream tasks.
Several multi- downstream tasks are built based on the COCO dataset, where the validation and test splits of these downstream lasks are scattered across the raw COCO splits. Therefore, during pre-training, carefully selected the portion of the COCO dataset which does not overlap with the validation of these multi-modal downstream tasks.
/tttt /test splits
D Implementation Details & Hyper-parameter Values
D.1 Data Augmentation
We use ResNNet50/SviinT/Srrii (He et al., 2016; Liu al., 2021) as image encoder and RoBERTa (Liu al., 2019) text encoder. Each encoder is followed by projector network which 3-layer MLPP with the configuration 1-200482048-100 1024). Here, represents the embedding dimension of the encoder's output.
Data Type
View #
Augmentation
Probability
LandonRenizeddran
1.0
Bandeafiori2ontalF..p
0.5
ColorJitter
0.8
RandonGrayscale
0.2
Gaussianšlur
Solarization
0.0
Image
RandonRenizeddrrp
1.0
LandonflorizantaPP..
0.5
ColorJitter
0.8
RandonGrayscale
0.2
LanBur
0.1
Solarization
0.2
Synonym Replacement
0.1
Random Insertion
0.1
Random Swop
0.1
Random Deletion
0.1
Text
"Syñonýı Replačeinent
0.1
Random Insertion
0.2
Random Swap
0.1
Random Deletion
0.2
Table D.1: Image and text augmentation details.
Image Augmentations: Two sets of random transformations sampled from an augmentation pool applied on each input image to generate two disparate distorted views. The auganentation policy is com- posed of Retdcmmeeiizedrrrg Randomfoorizanttlllii, ColorJitter, RandomGrayscale, GaussianBlur, and Solarization augmentations. RandomResizedCroop is applied with probability of 1.0, whereas the remaining ones are applied randomly with varying probabilitics following Zbontar et al. (2021) (sce Table D.

---

Published in Transactions on Machine Learning Research (09/2023)
Text Augmentations: Two sets of random transformations are applicd on imput text using EDA (Wei & Zou, 2019) including synonym replacement, random insertion, random swap, and random deletion with different probabilities as outlined in Table D.1.
D.2 Pre-training Setup
Table D.2 shows the details of hyper-parameters used during training.
VoLTA comprises vision encoder and a language encoder with a merged co-attention for cross-modality fusion. In our experiments, we have considered two types of vision encoder backbones ResNet-50 (He et al., 2016) and Swin Transformer (Liu et al., 2021). For fair comparisons with related works (Dou al., 2022b;a), the input image resolution for ResNet-50 encoder backbone is kept as 224 x 224, whereas for Swin-B, it is 384 x 384. The output embedding dimension of the image encoder in both cases is 1024. Similarly, to be consistent with Dou et al. (2022b;a), we have selected RoBERTa as the language encoder with vocabulary size of 50265, a lokenizer as 'roberta-hase', maximum input text length of 30, and output embedding dimension of 768 (please refer to Table D.2 for more details).
Separate projector heads follow vision and language encoders. A projector head consists of 3 linear layers, each with 2048 output units (except for the last one, which has 1024 output units), followed by a Batch Normalization layer and ReLU activation (for exact configuration, please refer to Table D.2). The final projected output denotes the input (image//ext)) feature representation used in downstream tasks. The embeddings (i.e., output from respective encoders) are fed into the loss function of VoLTA to learn these representations.
The loss function of VoLTA includes four different loss components, namely, multi-modal Barlow Twins for intra- and inter-modality redundancy reduction, GOT for alignment of local features, and MLM and ITM together for encouraging cross-modal attention fusion. For MIM,, we randomly mask 15%² (MLM probability in Table D.2) of the input tokens, and the model is trained to reconstruct the original tokens. For ITM, the model predicts whether a given image-text pair is matched.
For optimization, we follow the same protocol as described in Zbontar et al. (2021), where we use the LARS (You et al., 2017) optimizer to train our model for 20 epochs with batch size of 256. A base LR of 0.1 used for the weights and 0.0048 for the biases and batch normalization parameters which are then multiplied by a factor of 2. We employ a learning rate warm-up (linear) up to a period of 2 epochs followed by a cosine decay schedule to reduce the LR by factor of 1000. A weight decay parameter 1e-6 is used, excluding the biases and batch normalization parameters. We conduct a grid search for the GOT loss hyperparameter (WGOT), and we empirically found the best value to be 100.
Pre-training Cost: Our Swin-B backbone lakes 6 hours per epoch to train on 64 V100 GPUs, with per GPU batch-size 4.
D.3
Downstream Setup
D.3.1 Uni-modal downstream tasks
Linear Evaluation: For ImageNet, the linear classifier has been trained for 100 epochs with a batch size of 256, an LR of 0.3, and a cosine LR schedule. Cross-cntropy loss is minimized with SGDM optimizer (momentum of 0.9), and a weight decay of le-6. For both COCO and VOC, the linear classifier has been trained for 100 epochs with AdamW optimizer with batch size of 256 an LR of 5e-2, and a weight decay of 1e-6.
Object Detection: For training the detection model, the detectron2 library (Wu et al., 2019) has been used. The backbone networks for Faster R-CNN (Ren al., 2015) and Mask R-CNN (He al., 2017) has been initialized using our pre-trained model.
For V0 we used the trainval set comprising 16K images for training a Faster R-CNN (Ren et al., 2015) C-4 backbone for 24K iterations using batch size of 16 across 8 GPUs (using SyneBatchNorm). The
Pollowing BERT, we decompose this 15% 10% randorn words, 10% unchanged, and 80% with a special token [MASK].
26

---

Published in Transactions Machine Learning Research (09/2023)
Hyper-parameters
Notation
Value
Model
Tmg. proj- layer config.
(14
2048 2048 1024
Ing. embed. dim
d.
1024
Img. reso, (RN-50 & Swin-T)
224
224
Img. reso. (Swin-B)
384 384
Txt. proj. layer config.
[d
2048
2048 1024
Txt. embed. dim
de
768
Tokenizer
'roberta-hase"
Vocab size
50265
MLM prob.
0.15
Max. longth of text
30
# Heads of Xformer
12
# Layers of Xformer
12
# Fusion block
6
Dropout rate
0.1
Task names
BBGGTT, MLM, ITM,
Training
Batch size
256
Epochs
20
Lambda BT
X
0.005
WD and GWD loss weight.
0.1
GOT loss weight.
MGOT
100.0
Optimizer
LARS
Base LR for weights
0.1
Base LR for biases
0.0048
Momentum
0.9
LR scheduler
Cosine LR decay (with linear warm-up)
Warm-up steps
x Epochs
Weight decay
1e-6
End LRR factor
0.001
Cosine LR amplitude factor
0.5
Table D.2: Pre-training hyper-parameter details for VoLTA.
initial learning rale for the model is 0.15, which is reduced by a factor of 10 after 18K and 22K iterations. Linear warmup (Goyal et al., 2017) is used with a slope of 0.333 for 1000 iterations.
For COCO, Mask R-CNN (He et al., 2017) with C-4 backbone the COCO 2017 train split is used for training, and the results are reported on the val split. A learning rate of 0.03 is used, and other parameters are kept the same as in the 1x schodule detectron2 (Wu et al., 2019).
D.3.2 Coarse- Coarse-grained multi-modal downstream tasks
Vision- LanguugL Classification (VQAv2 and NLVR*): Vision-Language Classification task encom- passes VQAv2 and NLVR?? whose hyper-paraneter setup has been taken from METER (Dou et al., 2022b) and FIBER (Dou et al., 2022a). Model finetuning done with peak learning rates 26-5 for the backbones, 1e-4 for the cross-most rross-modal parameters, and 1e-3 for the head layer for 10 epochs with a batch size of 512. The image resolutions are set to 576 for VQA22 and 384 for NLVR² and the models are evaluated with the VQA-Scores for VQAv2 and accuracy for NLVR? (Table C.1).
Image- Text Retrieval (IRTR): We follow Dou et al. (2022a) for IR-TR setup for the Flickr30k dataset, where the cross-attention layers in the backbones are removed during IR-TR fine-tuning and evaluation. The peak learning rates set 2e-5 for the backbones, and le-4 for the head layer. Parthermore, a batch size of 1024 is considered, and each image resolution is set to 576. We evaluate the Recall@1 metric for both the text and image retrieval tasks as outlined in Table C.1.
Image Captioning: For image captioning, only the image-to-text attentions are kept for the cross-modality attention fusion, and the model is converted into a standard seq2seq model (Dou et al., 2022a). We used causal mask the decoding side, and the outputs are predicted auto-regressively (Dou et al., 2022a).
27

---

Published in Transactions Machine Learning Research (09/2023)
KIIL HILL
Caption: Red peppers the table.
Caption: microwave oven kept on the countertop.
Caption: collection apples crates.
Caption: person taking photo.
Figure E.1: Limitations of our method: tiny and hindered objects in cluttered environments are not distinctly attended by VoLTA. We look at the optimal transport plan from GOT to represent the aligument between the words in red with the corresponding input image. The visualizations are generated with 224p images, resulting in sequences of 196 tokens for 16 16 patches. All imago-caption pairs are taken from the COCO2017 train split.
Models are trained with the cross-entropy loss for 5 epochs with the peak learning rates of 5e-5 for the backbones, and 2.5e-4 for the rest of the parameters, followed by a two-stage finetuning. In the first stage, finetuning with GOLD (Pang He, 2021) is done for epochs with peak learning rate of 1e-5 for the backbones, since it is efficient and has been proven to be effective when the model inpul can correspond to different outputs. The sccond stage of fine-tuning involves CIDEr optimization where the learning rate is further reduced 1e-6, and the model is trained for 3 epochs. batch size of 512 considered in both these cases, and a beam size of 5 is used during inference. Evaluation metrics include BLEU (Papineni et al., 2002), METEOR (Banerjee Lavie, 2005), CIDEr (Vedantam et al., 2015), and SPICE (Anderson et al., 2016) scores (shown in Table C.1).
D.3.3 Fine-grained molti-moddl downstream tasks
We follow Dou el. al. (2022a) for training and evaluation on 3 different datasets (RefCOCO, RefCOCO+, and RefCOCO) where the models are finetuned with a batch
Referring Expression Comprehension (REC):
size 16 for 20 epochs. warmupp 2000 steps with a peak LR of lc-5 used for the OD head as well as the rest of the model's parameters. LR drops twice, once at 67% and the other at 89% of the total number of steps. Horizontal flip augmentation has boen turned off during REC training because it was observed by Dou al. (2022a) that horizontal flip adversely affected the performance, particularly on the RefCOCO dataset. Accuracy is used as the evaluation metric in this case (Table C.I).
We follow the training and evaluation setup of Dou el al. (2022a) for text-conditioned (multi-modal) object detection. For both COCO and LVIS datasets, the model has been finetuned for 24 epochs with a batch size of 32, an LR of 1e-5, and two learning rate drops, once at 67% and the other at 89% of the total number of steps. AP scores are used this case for model evaluation (Table C.1).
Object Detection:
E Error Analysis
Although VoLTA Iearns impressive fine-grained region-level understauding during pro-training, there are still some cases where the model fails to identify tiny and hindcred objects, especially in cluttered environments. We show four such examples in Figure E.I. In the first image, the object 'red peppers' is barely visible
28

---

Published in Transactions Machine Learning Research (09/2023)
# #Pre-train
VQÁv2
NLVR
Fa0k IRTR
# Pre-trrii Data
COCO Captioning
Method
Method
Data
dev
std
dev
test-P
TR&I
TR
Ba1
M
C
S
Models pre-trained on COCO
(1235) and/or
VG
(108A)
Models finc-huned without CTDEr
optaniiatiinn
SCAN
48.6
67.4
VI-T5
180k
34.5
28.7
116.5
21.9
SCG
108k
49.3
71.8
VI-BART
180k
35.1
26.7
116.6
21.5
PEAN
108k
50.4
70.0
Unifted VLP
3M
36.5
28.4
117.7
21.3
MaxEnt
123k
54.1
54.8
OSCAR
4M
36.5
30.3
123.7
23.1
VisudBERT
71.0
67.4
UPO-B
1M
36.0
28.9
122.8
LXMERT
231k
72.1
72.5
71.9
71.5
VITCAP
1M
36.3
29.3
125.2
22.6
SOHO
231k
73.2
73.4
76.3
77.3
72.5
86.5
METER-CLIP-B
1M
38.8
30.0
128.2
23.0
Modala pre-snusseed on COCO,
VG, SIBU
(1.M)
CC (3M))
VIVV-B
4M
38.2
30.3
129.3
23.6
VILBERT
3M
70.5
70.9
38.2
XGPT
3.1M
37.2
28.6
120.1
21.8
UNITER
4M
72.7
72.9
77.2
77.9
72.5
85.9
FIBER B
4M
39.1
30.4
128.4
23.1
VILLA-B
4M
73.6
73.7
78.4
79.3
74.7
86.6
FIBER GOLD-B
4M
40.3
30.7
133.6
23.0
UNIMO-B
74.0
VOLTA-GOLD-B
30.5
128.5
ViLT-B
4M
71.3
75.7
76.1
61.1
83.5
Model Invell with CTDEr oplionization
ALBEF-B
4M
715
74.7
80.2
80.5
82.8
91.3
VITCAP
1M1
11:2
30.1
138.1
21.1
VLMo-B
76.9
$2.8
63.3
79.3
92.3
ViaVI-B
4M
40.9
30.9
140.4
METER-Swin.EB
4M
76.4
76.4
62.2
83.5
79.0
92.4
PIBERBB
4M
42.8
31.0
142.8
24.3
FIBER-B B
4M
78.6
78.4
84.6
85.5
81.4
92.9
PIBER GOLD
4M
43.4
31.3
144.4
24.6
VoLTA-B
231k
74.6
74.6
76.7
78.1
72.7
83.6
VoLTA-GOLD-B
231k
40.2
30.9
137.5
23.7
Table F.I: Multi-modal coarse-grained downstream: visual question answering, visual reason- ing, retrieval, and captioning. Methods pre-training with a significantly larger dataset are colored gray. For captioning, 4 metrics are reported B&4: BLEU@4, M: METEOR, C: CIDEr, S: SPICE. The best. comparable results are in bold. VoLTA-B denotes Swin-B backbone.
even in human eyes, and thus, VoLTA can not precisely identify these objects. However, it can identify the coarse region (the fruit basket) where 'red peppers' can be prcsent. In the second image, VoLTA confuses a 'dishwasher' with a 'microwave oven,' probably because the 'microwave oven' is present in a cluttered environment, and both objects have similar appearances in low-resolution frames. VoLTA can correctly identify 'red apples', but fails to spot 'green apples', probably because VoLTA has not seen enough such samples. In the last. image, the face of the 'person' is hindered by the camera, and VoLTA fails to locate it. Since we pre-train VoLTA with 224 224 images, such tiny objects are often hard to be distinguished. However, righer-resolution images will be more helpful in addressing such intricate scenarios, which we plan to explore in future works.
the third image,
Additional Quantitative Results on Coarse granse-grained Vision-Language Tasks: Comparison with Methods using More Pre-training Data
Table F.1 presents a comparison of VoLTA on the multi-modal coarse-grained tasks with state-of-the-art. methods pre-trained using magnitude more data. On VQA, VoLTA beats VILBERT, UNITER-B, VILLA- B, UNIMO-B, and ViL.T-, each pre-trained on 3 - 4M datasets. Please note that VoLTA is trained only on COCO and VG, whereas the other methods use a combination COCO, VG, CC, and SBU datasets. Such strong performance proves the generalizability of VoLTA. On captioning, VoLTA beats Unified VLP, OSCAR, UFO-B, ViTCAP, VinVL-B, METER-CLIP-B, and XGPT. However, for IRTR and NLVR VoLTA can not yield better performance over these baselines. We assume that the large domain difference between pre-training and downstream datasets is the reason behind the limited performance on IRTR and NLVR.
G Additional Qualitative Results
Visual Question Answering and Visual Reasoning: Visual question answering (VQA) is widely recoguized multi-modal lask that infers an answer in response to a text-based question about an image. In Figure G.1, we demonstrated several examples image-question pairs and corresponding answers predicted by VoLTA on the VQAv2 validation set. The primary aim of the visual reasoning task is to ascortain the veracity of a natural language statement against an associated image pair. Figure G.2 displays examples responses (True/ False) predicted by Vol.A on the NIVR? validation set.
29

---

Published in Transactions Machine Learning Research (09/2023)
Examples VQAv2 validation
Q: What is moving the photo?
the helr tult?
What is he about do?
yes
O: What the desk made out of?
Are people on the balcony?
Is this a natural enviromment?
A: wood
A:
a: How many lights green?
What is the holding in his hands? scissors
What color the water? blue
A:
Figure G.1: Examples Visual Question Answering from VQAv2 validation dataset. We display a variety of examples (e.g,, number items, the color of objects, type of objects, events, and actions) with respective answers predicted by VoLTA.
Languagg-condditioned Object Detection: Object detection forms indispensable constituent of several multi-modal understanding systems. However, the conventional object detection pipeline is employed as a black-box tool and predicts all possible objects the image. On the other hand, for better apprehension combinations of these objects in free-form texts, language-conditiones object detection task is considered (Kamath et al., 2021; Dou et al., 2022a). We use pre-trained VoLTA and fine-tuned and evaluated COCO and LVIS datasots for the text-conditioned object dotoction task. As illustrated in Figure G.3, VoLTA predicts bounding boxes relevant to the text prompts (captions) and labels them with the corresponding spans from the text. For example, the top-middle image has 4 objects. However, based on the text prompt, our model predicts boxes only for person and cup.
Referring Expression Comprehension (REC): The objective of REC to aligu the entire referring expression (text) with the corresponding box by disambiguating among the several occurrences of an object belonging to the same category and therefore, one box expression is to be predicted. For example, the bottom-left image in Figure G.4 depicts VoLTA's box prediction for the corresponding reforring expression: the slice of cake on the left.

---

Published in Transactions Machine Learning Research (09/2023)
Examples from NLVR validatioo set
Dcy
ROW
spoon dough
image right
Image contuine teAz! three dags atting in grazn
Patuu
right image contrita wractly colorful parrotx
Troee
person holding a cup the right imagee
Figure G.2: Examples on Visual Reasoning from NLVR validation dataset. For each statement
(text prompt), images shown alongside each other and VoLTA predicts whother the given statement is True (green box) or False (red box).
lext-conditonee
detection COCO validation
person, cup.
handbag.
person, umbrelia.
laptop. mouse.
Figure G.3: Examples of Object Detection from COCO validation dataset with various text prompts. Our model predicts boxes relevant to the text (caption) and labels thei with the corresponding spans.
Comparison of CLIP vs. VoLTA on Referring Expression Comprehension (REC): Figure G.5 shows a comparative qualitative evaluation between frozen CLIP + dynamic head (Dai al., 2021), and VoLTA. We concatenate the vision and text features from the CLIP encoders and train dynamic head on top of frozen features for the REC task. Since CLIP a dual-encoder system pre-trained with imago-level features, it not learn superior finegraiied features. Hence, CLIP fails harder REC samples. For example, if multiple similar-looking objects (benchos or bowls) cxist in image, CLIP fails to distinguish between them. However, VoLTA succeeds on such complex samples, which can be attributed to the fine- grained alignment achieved by the GOT objective.

---

Published in Transactions Machine Learning Research (09/2023)
Examples from RefCOCo validation
bird on for right
the red container in the blue cart
left evephant
RefCOCO+ validation
mifee
red laptop
Examples from RefCOCOg validation set
The ced chose top 0.85
case St
the mode Con tiver
group
cell phone on top
Figure G.4: Examples of Referring Expression Comprehension from RefCOCO (top), Ref- COCO+ (middle) and RefCOCOg (bottom) validation datasets. The expressions in RefCOCOg typically have florid and longer constructions compared to RefCOCO and RefCOCO+ The model has access to the entire text and uses to disambiguate amongst different objects in the image.
RefcoCOg
CLUP DyHeaa
2002
VOLTA
right
red umbreite opezed
animal bebindt the man on bench with three men bosel with spoon pointing
(a) Simpler samples, both CLIP and VoLTA succeed.
(b) Harder samples, CLIP fails, but VoLTA succeeds.
Figure G.5: Comparative qualitative evaluation between frozen CLIP Dynamic Head (Dai et al., 2021),
and VoLTA on different RefCOCO, RefCOCOg, RefCOCO+ validation samples.
32

---

Published in Transactions Machine Learning Research (09/2023)
Caption: TAo poar bears in water hoding smail ned objects
Captice: Thos small dogs are shown staring at
Caption:: A red mectrryyci is paraed on top of
pizza
parking lot.
Caption: Bunch
Captions
Captions Three luggage bags on the g ground near anothes
Caption: Twoy pretty sitting on branch.
Caption: A pince pia piece
umbrellas flynn sky above dit.
Severl
gras..
flai..
Caption: nat azzen doughnut pata Deside two glasses.
Caption: divvr fre hydrant troc of siler car.
Caption:
Caption: apple long
Caption: man putting king on of chocolate cookdes.
Caption: A hand holding minizture frocted doughaut with ronpariess.
standing bayy elephant top
Caption:
doughnut phone
Figure G.6: This figure shows how different words in captions attend relevant image regions, produced by the GOT module of VoLTA pre-trained on COCO. Extension of Figure 4. All image- caption pairs are taken from the COCO2017 train split. The visualizations are generated with 224p images, resulting in sequences 196 tokens for 16 patches.


Reviews:
[{"review_id": "8I7XfaYYtE", "reviewer": "Reviewer_NA62", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n\n1. The experimental evaluations are particularly strong. Both vision and vision-language downstream tasks are benchmarked with extensive comparisons to state-of-the-art pre-training methods.\n\n2. The proposed method is easy to understand. The details of the implementations are provided in the appendix, thus this work should be easy to reproduce.\n\nWeakness:\n\n1. It is recommended to better explain the insight of BT loss. The traditional constrative loss aligns the embeddings of two augmentations of the sample example. In contrast, BT aligns the two augmentations at the same embedding channel. The correlation between different channels could be negative but the $\\sum_i\\sum{j\\neq i} C_{ij}^2$ encourages the i-th and j-th channels to be non-correlated. Does this rule out the negative correlation between i-th and j-th channels?\n\n2. In Section 3.2, the notation indicating the edges is confusing. Specifically, please check $e\\in{E_x, E_y}$ and $G_x(V_x,E_x)$.\n\n3. The Gromov-Wasserstein distance requires examining quadruples of tokens/patches. I am wondering the computation cost on long text descriptions. It is not clear how much additional computation is introduced by solving the assignment problem.\n\n4. From the ablation study presented in Tab. 6, the most effective component is $L_w$, while other components are almost not contributing at all. Similar observations are made from Tab.1 where MLM and ITM losses are not making any contribution at all.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This work proposed a method, VoLTA, to pretrain vision language model with image-caption pairs only. The contributions can be summarized as the following points. \n- The BT loss is first adopted as the intra and inter modal as the main training objective to learn embedding both within and across modalities.\n- The proposed method exploits optimal transport algorithm to discover the optimal match between image patch features and text token features to improve the downstream fine-grained visual tasks. \n- Cross-modal attention fusion is further introduced to better fuse the two modalities. MLM and ITM are adopted to pre-train the cross attention parameters.\n- VoLTA is pretrained on the combination of VG dataset and COCO dataset, and is demonstrated on both vision and vision-language downstream tasks, achieving very competitive performance. \n", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. Please provide more insight into the adopted BT loss.\n\n2. Fix the notations $e\\in{E_x, E_y}$ and $G_x(V_x,E_x)$., if it is a typo.\n\n3. Please provide an analysis of the additional computation cost introduced by the assignment problem.\n\n4. Observing from the ablation study, the effectiveness of loss terms are not very useful except for $L_w$. Please clarify why components of the introduced losses, other than $L_w$, are not making much contribution to the downstream task results.\n\n5. The visualizations of patch-word alignment in Fig. 4 reveals the effectiveness of the method and provide interesting insights. But, instead of showing the successful cases, I would recommend the authors to provide some failure cases, if any. This would provide more insight into the weakness of the method. \n\n6. The performances of VoLTA on Swin-T/B are very close to ImageNet-1K + Label pretraining, according to Tab. 2. Therefore, one potential question is the annotation cost, it would be more interesting to analyze the annotation cost for ImageNet-1K + Label v.s. COCO + Caption.\n\n7. The comparisons on vision downstream task in Tab. 1 & 2 are not totally fair. For example, I am wondering why only VoLTA is evaluated with Swin-T/B backbone in Tab. 1? The same issue persists in Tab. 2, why VoLTA is not evaluated with VIT-B while numerical comparison to existing methods are based on ViT-B? I would suggest to evaluate VoLTA on LAION-20M with ViT-B for fair comparison here.", "broader_impact_concerns": "There are no outstanding ethical issues. Some potential societal concerns could be the breach of copyright in the training data or incorporating bias due to the image-caption correlations.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "60184USok1", "reviewer": "Reviewer_8rJP", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. The main subject of the paper is currently a very important research direction. Image-text pretraining has revolutionzed vision, however more work is required to establish strong methods for learning spatial represenations.\n2. The proposed model is evaluated on a wide range of benchmarks. Qualitative results are also present.\n\nWeaknesses:\n1. Throughout the paper it is often emphasized that VoLTA relies on ~100k COCO images for pretraining, while competing methods rely on much larger ~1M ImageNet source of images. Thus, VoLTA is superior. I believe this reasoning is very misleading on many levels and should be corrected:\n    - While COCO dataset is 10x smaller than ImageNet, it uses much stronger and more expensive labels: image captions, not semantic labels.\n    - COCO images were carefully selected to be good for detection like tasks and thus can be more appropriate for learning spatial representations.\n    - It is doubly unfair not to directly compare against _unlabeled_ imagenet pretraining. Table 1 omits moco-v2/simclr baselines which were pretrained in such fashion. Moco-v2/SimCLR are much simpler conceptually and use much _weaker_ supervision for pretraining (in fact, almost none). In its current form, Table 1 is very misleading.\n    - Minimizing the amount of images in the training data is not a useful objective by itself. A much more important metric is the cost of collecting and annotating the data. It is unlikely 100k captioned COCO images are easier to acquire than 1M (weakly) labeled ImageNet images. \n\n2. Table 2 shows a very marginal improvement of VoLTA over the original SWIN baseline. SWIN relies on much easier to collect image-level labeled dataset (not captions). Also SWIN is much simpler conceptually. If anything, I think Table 2 speaks against VoLTA being a practically useful technique for detection.\n\nNote that I appreciate that VoLTA aims to improve spatial represenations of image-text pretraining, I just do not find the current evaluation protocol convincing. Showing good scaling on real large data, like LAION, and surpassing simple imagenet pretraining can be one option. Another interesting option  would be to evaluate _frozen_ image models pretrained with CLIP-like objective only vs VoLTA. This may more clearly confirm that VoLTA improves the quality of spatial representations when doing image-text pretraining.   \n\n3. The paper experiments would make more sense if the pretraining data would be something like LAION dataset. Unlike COCO data, the LAION-like data scales in quantity (just crawl the internet). Limiting the evaluation to models pretrained on a small, human-labelled dataset has limited value. \n\n4. Qualitative results only show results of the final VoLTA model and does not ablate its components. In particular, I wonder about 10-20 non cherry-picked images that show alignment with and without optimal transfer objective. This objective is quite involved and I would appreciate a visual confirmation that it works as intended.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "No", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper proposes a new composite objective for image-text pretraining, with the emphasis on learning spatial visual representations. The proposed objective has 4 components:\n1. \"Barlow twins\" (BT) objective for image-image, image-text and text-text alingment.\n2. Graph-optimal transfer (GOT) objective that strives to align local image patch representations and text token representations.\n3. Masked language modeling (MTM) objective.\n4. Image text matching (ITM) objective.\nIn conjunction with the latter two objectives the authors propose a specialized cross-modality interaction mechanism, that futher boost the performance.\n\nThe proposed model is evaluated on a wide range of vision-language tasks, including classification, detection, captioning, VQA,  and retrieval.", "claims_and_evidence_yn": "No", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "My main critism is that the paper uses strong COCO caption supervision for pretraining, and then is worse (Table 1) and only matching (Table 2) than much simpler ImageNet pretraining. In Table 1, even a simple *unsupervised* ImageNet pretraining would surpass VoLTA (MOCO, SimCLR), however these baselines are omitted.\n\nSimilarly, Table 2 is also not very compelling: a VoLTA model, that uses strong COCO captions and 5 non-trivial objectives, just matches a basic ImageNet pretrained baseline.\n\nWith these, and other weaknesses mentioned above, I find it hard to recommend the acceptance of this paper. I think the paper should be more clear about its limitation (as, for example, rather weak results when compared to ImageNet pretraining). On top of this, I will appreciate a focused analysis of how the proposed objective terms improve spatial quality of visual representations compared to the standard CLIP training. For example, training a detection model on top for the frozen image encoder, and detailed qualitative results (not just results for the final method with no reference point).\n\nCurrent ablations are also very sparse. E.g. GOT loss seems to be only ablated in Table 6. I would also appreciate detailed ablations on more classical benchmarks, e.g. COCO / VOC / captioning.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "O34YQnh6kH", "reviewer": "Reviewer_Y9gh", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "- The main contribution of this paper lies in the construction of a comprehensive pipeline. Although the integrated techniques are borrowed from other tasks, the motivation behind their incorporation is intuitive and clear. Previous works have traditionally addressed image-level and region-level applications separately, often requiring bounding box annotations for region-level understanding. In contrast, this work proposes a unified pre-training framework that eliminates the need for bbox annotations, making it more practical and applicable in real-world scenarios.\n\n- The utilization of the graph optimal transport (GOT) algorithm for local feature alignment is a commendable approach, effectively addressing the problem at hand.\n\n- The gating formulation of the Cross-Modal Attention Fusion (CMAF) module is a sound strategy for integrating the Barlow Twins (BT) and GOT losses during pre-training, enabling a unified backbone for different downstream tasks.\n\n- The experimental results strongly support the claims made for both image-level and region-level downstream tasks, clearly demonstrating the superiority of the proposed method within the same data setting. The performance on uni-modal, multi-modal fine-grained, and multi-modal coarse-grained downstream tasks validates the effectiveness and robustness of the proposed approach. The ablation study table provides valuable insights into the effectiveness of each component and design choice.\n\n- The inclusion of Figure 4, showcasing the learned fine-grained patch-word alignment, is a valuable addition that enhances the paper's visualization and aids in understanding the achieved results.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper presents a novel vision-language pre-training method called VoLTA, which achieves unified pre-training for image-level and region-level downstream tasks without requiring expensive region bounding box annotations. The proposed framework integrates various techniques and designs to accomplish this objective.\n\nFirstly, in contrast to previous contrastive supervision methods, the paper introduces a non-contrastive covariance regularization called Barlow Twins (BT) (Zbontar et al., 2021). BT is computed on global image and text features and serves as the foundational image-level objective for VoLTA.\n\nSecondly, to enable region-level supervision without relying on bounding box annotations, the paper introduces a local feature-level alignment approach that utilizes global image-caption annotations. This alignment is formulated as a patch-word graph-matching problem, and the graph optimal transport (GOT) algorithm (Chen et al., 2020a) is employed for implementation.\n\nLastly, a Cross-Modal Attention Fusion (CMAF) module is designed to facilitate cross-modal interactions and enhance the representation of complex multi-modal features.\n\nExperimental results demonstrate that VoLTA performs well across both fine-grained and coarse-grained applications.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "- My main concern regarding this paper pertains to the performance evaluations. While the proposed method demonstrates superior performance within the same data setting, I am curious about its potential for significant improvements when provided with sufficient data and annotations. In Appendix E, it is suggested that the limited performance on IRTR and NLVR may be attributed to the large domain differences between the pre-training and downstream datasets. Considering the high cost associated with training on large-scale data, I am interested in any indicative evidence that showcases the potential of the proposed method surpassing other approaches given ample data, especially considering its nature as a pre-training task.\n\n- My second concern relates to the absence of recent related works in the discussion. All referenced works are from before 2023, but given the rapidly evolving field, I anticipate the existence of new works, such as \"Scaling language-image pre-training via masking,\" CVPR 2023. I believe it is crucial to include some discussions or even comparisons with recent works to establish the paper's relevance within the current research landscape.\n\n- Furthermore, I feel that the limitations and failures of the proposed method have not been adequately discussed and analyzed. It would be beneficial to have a comprehensive discussion on the specific constraints or scenarios in which the proposed method may not perform optimally and provide insights into potential areas for future improvement.\n\n- Regarding minor issues, based on my understanding of the Inter-modal Objective, there should be two additional loss terms: (I, T') and (I', T).\n\n- Lastly, the third-row result in Table 7(a) is missing, which should be rectified for clarity and completeness.", "broader_impact_concerns": "I do not have concerns about the ethical implications of this work.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper proposes a vision-language pre-training paradigm that avoids the need for high-resolution data. This is achieved with a weakly supervised alignment objective on local image patches and text tokens. Following the author response, three reviewers unanimously recommend acceptance, calling out the comprehensive evaluations and novel technique.
Decision: Accept (poster)